/
docs.go
251 lines (177 loc) · 9.61 KB
/
docs.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
// Copyright (c) 2011 CZ.NIC z.s.p.o. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// blame: jnml, labs.nic.cz
/*
WIP: Package falloc provides allocation/deallocation of space within a
file/store (WIP, unstable API).
Overall structure:
File == n blocks.
Block == n atoms.
Atom == 16 bytes.
x6..x0 == least significant 7 bytes of a 64 bit integer, highest (7th) byte is
0 and is not stored in the file.
Block first byte
Aka block type tag.
------------------------------------------------------------------------------
0xFF: Free atom (free block of size 1).
+------++---------++---------++------+
| 0 || 1...7 || 8...14 || 15 |
+------++---------++---------++------+
| 0xFF || p6...p0 || n6...n0 || 0xFF |
+------++---------++---------++------+
Link to the previous free block (atom addressed) is p6...p0, next dtto in
n6...n0. Doubly linked lists of "compatible" free blocks allows for free space
reclaiming and merging. "Compatible" == of size at least some K. Heads of all
such lists are organized per K or intervals of Ks elsewhere.
------------------------------------------------------------------------------
0xFE: Free block, size == s6...s0 atoms.
+------++---------++---------++---------++--
| +0 || 1...7 || 8...14 || 15...21 || 22...16*size-1
+------++---------++---------++---------++--
| 0xFE || p6...p0 || n6...n0 || s6...s0 || ...
+------++---------++---------++---------++--
Prev and next links as in the 0xFF first byte case. End of this block - see
"Block last byte": 0xFE bellow. Data between == undefined.
------------------------------------------------------------------------------
0xFD: Relocated block.
+------++---------++-----------++------+
| 0 || 1...7 || 8...14 || 15 |
+------++---------++-----------++------+
| 0xFD || r6...r0 || undefined || 0x00 | // == used block
+------++---------++-----------++------+
Relocation link is r6..r0 == atom address. Relocations MUST NOT chain and MUST
point to a "content" block, i.e. one with the first byte in 0x00...0xFC.
Relocated block allows to permanently assign a handle/file pointer ("atom"
address) to some content and resize the content anytime afterwards w/o having
to update all the possible existing references to the original handle.
------------------------------------------------------------------------------
0xFC: Used long block.
+------++---------++--------------------++---------+---+
| 0 || 1...2 || 3...N+2 || | |
+------++---------++--------------------++---------+---+
| 0xFC || n1...n0 || N bytes of content || padding | Z |
+------++---------++--------------------++---------+---+
This block type is used for content of length in N == 238...61680 bytes. N is
encoded as a 2 byte unsigned integer n1..n0 in network byte order. Values
bellow 238 are reserved, those content lengths are to be carried by the
0x00..0xFB block types.
1. n in 0x00EE...0xF0F0 is used for content under the same rules
as in the 0x01..0xED type.
2. If the last byte of the content is not the last byte of an atom then
the last byte of the block is 0x00.
3. If the last byte of the content IS the last byte of an atom:
3.1 If the last byte of content is in 0x00..0xFD then everything is OK.
3.2 If the last byte of content is 0xFE or 0xFF then the escape
via n > 0xF0F0 MUST be used AND the block's last byte is 0x00 or 0x01,
meaning value 0xFE and 0xFF respectively.
4. n in 0xF0F1...0xFFFF is like the escaped 0xEE..0xFB block.
N == 13 + 16(n - 0xF0F1).
Discussion of the padding and Z fields - see the 0x01..0xED block type.
------------------------------------------------------------------------------
0xEE...0xFB: Used escaped short block.
+---++----------------------++---+
| 0 || 1...N-1 || |
+---++----------------------++---+
| X || N-1 bytes of content || Z |
+---++----------------------++---+
N == 15 + 16(X - 0xEE). Z is the content last byte encoded as follows.
case Z == 0x00: The last byte of content is 0xFE
case Z == 0x01: The last byte of content is 0xFF
------------------------------------------------------------------------------
0x01...0xED: Used short block.
+---++--------------------++---------+---+
| 0 || 1...N || | |
+---++--------------------++---------+---+
| N || N bytes of content || padding | Z |
+---++--------------------++---------+---+
This block type is used for content of length in 1...237 bytes. The value of
the "padding" field, if of non zero length, is undefined.
If the last byte of content is the last byte of an atom (== its file byte
offset & 0xF == 0xF) then such last byte MUST be in 0x00...0xFD.
If the last byte of content is the last byte of an atom AND the last byte of
content is 0xFE or 0xFF then the short escape block type (0xEE...0xFB) MUST be
used.
If the last byte of content is not the last byte of an atom, then the last byte
of such block, i.e. the Z field, which is also a last byte of some atom, MUST
be 0x00 (i.e. the used block marker). Other "tail" values are reserved.
------------------------------------------------------------------------------
0x00: Used empty block.
+------++-----------++------+
| 0 || 1...14 || 15 |
+------++-----------++------+
| 0x00 || undefined || 0x00 | // == used block, other "tail" values reserved.
+------++-----------++------+
All of the rules for 0x01..0xED applies. Depicted only for its different
semantics (e.g. an allocated [existing] string but with length of zero).
==============================================================================
Block last byte
------------------------------------------------------------------------------
0xFF: Free atom. Layout - see "Block first byte": FF.
------------------------------------------------------------------------------
0xFE: Free block, size n atoms. Preceding 7 bytes == size (s6...s0) of the free
block in atoms, network byte order
--++---------++------+
|| -8...-2 || -1 |
--++---------++------+
... || s6...s0 || 0xFE | <- block's last byte
--++---------++------+
Layout at start of this block - see "Block first byte": FE.
------------------------------------------------------------------------------
0x00...0xFD: Used (non free) block.
==============================================================================
Free lists table
The free lists table content is stored in the standard layout of a used block.
A table item is a 7 byte size field followed by a 7 byte atom address field
(both in network byte order), thus every item is 14 contiguous bytes. The
item's address field is pointing to a free block. The size field determines
the minimal size (in atoms) of free blocks on that list.
The free list table is n above items, thus the content has 14n bytes. Note that
the largest block content is 61680 bytes and as there are 14 bytes per table
item, so the table is limited to at most 4405 entries.
Items in the table do not have to be sorted according to their size field values.
No two items can have the same value of the size field.
When freeing blocks, the block MUST be linked into an item list with the
highest possible size field, which is less or equal to the number of atoms in
the new free block.
When freeing a block, the block MUST be first merged with any adjacent free
blocks (thus possibly creating a bigger free block) using information derived
from the adjacent blocks first and last bytes. Such merged free blocks MUST be
removed from their original doubly linked lists. Afterwards the new bigger free
block is put to the free list table in the appropriate item.
Items with address field == 0 are legal. Such item is a placeholder for a empty
list of free blocks of the item's size.
Items with size field == 0 are legal. Such item is a placeholder, used e.g. to
avoid further reallocations/redirecting of the free lists table.
The largest possible allocation request (for content length 61680 bytes) is
0xF10 (3856) atoms. All free blocks of this or bigger size are presumably put
into a single table item with the size 3856. It may be useful to additionally
have a free lists table item which links free blocks of some bigger size (say
1M+) and then use the OS sparse file support (if present) to save the physical
space used by such free blocks.
Smaller (<3856 atoms) free blocks can be organized exactly (every distinct size
has its table item) or the sizes can run using other schema like e.g. "1, 2,
4, 8, ..." (powers of 2) or "1, 2, 3, 5, 8, 13, ..." (the Fibonacci sequence)
or they may be fine tuned to a specific usage pattern.
==============================================================================
Header
The first block of a file (atom address == file offset == 0) is the file header.
The header block has the standard layout of a used short non escaped block.
Special conditions apply: The header block and its content MUST be like this:
+------+---------+---------+------+
| 0 | 1...7 | 8...14 | 15 |
+------+---------+---------+------+
| 0x0F | m6...m0 | f6...f0 | FLTT |
+------+---------+---------+------+
m6..m0 is a "magic" value 0xF1C1A1FE51B1E.
f6...f0 is the atom address of the free lists table (discussed elsewhere).
If f6...f0 == 0x00 the there is no free lists table (yet).
FLTT describes the type of the Free List Table. Currently defined values:
------------------------------------------------------------------------------
FLTT == 0: Free List Table is fixed at atom address 2. It has a fixed size for 3856 entries
for free list of size 1..3855 atoms and the last is for the list of free block >= 3856 atoms.
*/
package falloc
const (
INVALID_HANDLE = Handle(-1)
)