Repository navigation
File Format
An .mlb file is a fixed 256-byte header followed by tensor arrays. Every array
starts at a 64-byte boundary, so the runtime can mmap the file and point
straight into it without copying. Everything is little-endian.
tools/mlb.py is the reference writer and reader. model/mlb.c is the loader.
| Offset | Type | Field | Notes |
|---|---|---|---|
| 0 | char[4] |
magic | "MLB1" |
| 4 | u32 |
version | 1 |
| 8 | u32 |
dim |
model width; multiple of 32 |
| 12 | u32 |
n_layers |
|
| 16 | u32 |
n_heads |
must divide dim; head_dim = dim / n_heads must be even |
| 20 | u32 |
n_kv_heads |
must divide n_heads (GQA) |
| 24 | u32 |
hidden_dim |
FFN width; multiple of 32 |
| 28 | u32 |
vocab |
at least 3 (ids 1 and 2 are BOS and EOS) |
| 32 | u32 |
max_seq |
longest context the model allows; the KV cache holds --ctx positions, at most this |
| 36 | f32 |
rope_theta |
> 0 |
| 40 | f32 |
norm_eps |
≥ 0 |
| 44 | u32 |
wtype |
1 = Q8_0, 2 = Q4_0 |
| 48 | u32 |
shared_classifier |
non-zero: wcls is omitted and tok_emb is used |
| 52 | u32 |
rope_type |
0 = plain RoPE, 1 = llama3, 2 = linear, 3 = yarn scaling |
| 56 | f32 |
rope_factor |
llama3, linear, yarn: > 0 |
| 60 | f32 |
rope_low_freq_factor |
llama3: > 0; yarn: beta_fast
|
| 64 | f32 |
rope_high_freq_factor |
llama3: > rope_low_freq_factor; yarn: beta_slow, > 0 and < beta_fast
|
| 68 | u32 |
rope_orig_max_seq |
llama3, yarn: > 0 |
| 72 | u32 |
q8_mask |
wtype Q4_0 only: matrix classes stored as Q8_0 instead (below); 0 for Q8_0 models |
| 76 | u32 |
qkv_bias |
0 or 1: the q, k and v projections have biases (Qwen2) |
| 80 | u32 |
head_dim |
0: dim / n_heads; else the head size (even; Qwen3) |
| 84 | u32 |
qk_norm |
0 or 1: per-head RMSNorm of q and k before RoPE (Qwen3) |
| 88 | f32 |
rope_attn_factor |
yarn only: > 0, multiplies RoPE's cos and sin; 0 otherwise |
| 92 | u32 |
sliding_window |
0: full attention; else each position attends to the last this many |
| 96 | u32 |
layer_flags |
0 or 1: a byte of flags per layer follows the header (below) |
| 100 | f32 |
rope_local_theta |
≥ 0: RoPE base of the layers flagged for it, used unscaled (Gemma 3) |
| 104 | u32 |
act |
FFN activation: 0 = SiLU, 1 = GELU with the tanh approximation (Gemma) |
| 108 | u32 |
post_norms |
0 or 1: the attention and FFN outputs are RMS-normalized before the residual (Gemma 3) |
| 112 | f32 |
embed_scale |
≥ 0: factor on the embedding rows; 0 means 1 (Gemma: sqrt(dim)) |
| 116 | f32 |
attn_scale |
≥ 0: attention score factor; 0 means 1/sqrt(head_dim)
|
| 120 | u32 |
scale_f16 |
0 or 1: every quantized matrix stores its scales d as IEEE f16 instead of f32
|
| 124 | f32 |
attn_softcap |
≥ 0: attention scores become cap · tanh(score / cap) before the softmax; 0 means none (Gemma 2) |
| 128 | f32 |
final_softcap |
≥ 0: the same on the logits; 0 means none (Gemma 2) |
| 132 | u32 |
rot_dim |
even, at most head_dim: RoPE rotates the first rot_dim dimensions of each head; 0 means all (Qwen3.5: a quarter) |
| 136 | u32 |
attn_gate |
0 or 1: attention layers store wg; their output is multiplied by sigmoid(wg · x) before wo (Qwen3.5) |
| 140 | u32 |
lin_k_heads |
linear attention (below): key heads; 0 when the model has none |
| 144 | u32 |
lin_v_heads |
value heads: a multiple of lin_k_heads and of 4 |
| 148 | u32 |
lin_k_dim |
size of a key head |
| 152 | u32 |
lin_v_dim |
size of a value head; lin_v_heads × lin_v_dim is a multiple of 32 |
| 156 | u32 |
lin_conv |
kernel size of the causal convolution |
| 160 | padding | zero up to offset 256 |
dim, n_layers, n_heads, n_kv_heads and hidden_dim are limited to
65536. vocab and max_seq are limited to 2^24.
Files written before a field existed have zeros there, and zero keeps the
earlier behavior. Zeros at offsets 52–71 (RoPE scaling) mean plain RoPE. A zero
q8_mask (offset 72, mixed models) stores no matrix as Q8_0. A zero at offset
76 means no biases. Zeros at offsets 80–95 mean head_dim = dim / n_heads, no
q/k norm and full attention. Zeros at offsets 132–159 mean RoPE on every
dimension, no attention gate and no linear attention.
Derived values: head_dim (offset 80, or dim / n_heads when 0),
q_dim = head_dim × n_heads (a multiple of 32) and
kv_dim = head_dim × n_kv_heads.
q8_mask bits, one per matrix class: 1 tok_emb, 2 wcls, 4 wq and
wk (also wg and lqkv), 8 wv (also lz), 16 wo (also lout), 32 w1
and w3, 64 w2. With a shared classifier,
tok_emb is Q8_0 if bit 1 or 2 is set.
With qkv_bias set, each layer stores bq, bk and bv after wo; they
are added to the projections before RoPE. With qk_norm set, each layer then
stores q_norm and k_norm (head_dim floats each), applied to every query
and key head after the biases and before RoPE, with norm_eps.
With layer_flags set, n_layers bytes (padded to 64 like every array) come
first in the body, one per layer. Bit 0 puts the layer behind the sliding
window; bit 1 makes it use rope_local_theta instead of rope_theta and its
scaling; bit 2 leaves the layer's queries and keys unrotated (no RoPE, as in
SmolLM3), and cannot be combined with bit 1; bit 3 makes it a linear-attention
layer (below), alone; other bits must be 0. Without layer flags, the window (if
any) applies to every layer. Bit 2 needs Krafla 0.8.0 or later and bit 3,
like offsets 132–159, 0.9.0 or later; older versions refuse such files.
A layer with bit 3 replaces attention with the gated delta rule of Qwen3.5.
With K = lin_k_heads, V = lin_v_heads, dk = lin_k_dim,
dv = lin_v_dim, the layer projects its normalized input into
2·K·dk + V·dv channels (queries, keys, values), V·dv gate values z, and
V values each of b and a. For each position, in order:
- Each channel passes through a causal convolution of
lin_convtaps over the channel's last inputs (lconv), then SiLU. - Each query and key head is divided by its L2 norm (
ε = 1e-6); queries are also divided bysqrt(dk). Value headhuses query and key headh / (V / K). -
β = sigmoid(b),g = −exp(A_log) · softplus(a + dt_bias). - The value head's state
S(dk × dv, zero at position 0) becomesS·exp(g), thenS + k (β (v − Sᵀk))ᵀ; the head's output isSᵀq. - The output is RMS-normalized per head with
lnormandnorm_eps, timessilu(z), and goes throughlout.
The state replaces the KV cache: such a layer has no KV cache, and its memory does not grow with the context.
The arrays start at offset 256 and come in this order:
layer flags n_layers (u8; only if layer_flags)
tok_emb vocab × dim (wtype, or Q8_0 per q8_mask; same below)
for each layer:
attn_norm dim (F32)
wq q_dim × dim (wtype)
wk kv_dim × dim (wtype)
wv kv_dim × dim (wtype)
wo dim × q_dim (wtype)
wg q_dim × dim (wtype, class of wq; only if attn_gate)
bq, bk, bv q_dim, kv_dim, kv_dim (F32; only if qkv_bias)
q_norm, k_norm head_dim, head_dim (F32; only if qk_norm)
ffn_norm dim (F32)
w1 (gate) hidden_dim × dim (wtype)
w2 (down) dim × hidden_dim (wtype)
w3 (up) hidden_dim × dim (wtype)
post_attn_norm, post_ffn_norm dim, dim (F32; only if post_norms)
final_norm dim (F32)
wcls vocab × dim (wtype; omitted if shared_classifier)
A linear-attention layer (bit 3) stores, in place of wq to q_norm, k_norm:
attn_norm dim (F32)
lqkv (2·K·dk + V·dv) × dim (wtype, class of wq)
lz V·dv × dim (wtype, class of wv)
lb, la V × dim, V × dim (always Q8_0)
lconv (2·K·dk + V·dv) × lin_conv (F32)
dt_bias, A_log V, V (F32)
lnorm dv (F32)
lout dim × V·dv (wtype, class of wo)
followed by ffn_norm and the FFN as in any layer.
Matrices are stored row by row, one row per output: y[r] = Σ_c W[r][c] · x[c].
An F32 tensor is one array of floats. A quantized matrix is two arrays, written one after the other:
-
q: the quants.- Q8_0:
rows × colsbytes ofint8. - Q4_0:
rows × cols / 2bytes.
- Q8_0:
-
d:rows × cols / 32scales, one per block of 32 consecutive weights in a row:f32, orf16whenscale_f16is 1 (Q4_0 then takes 4.5 bits per weight instead of 5, Q8_0 8.5 instead of 9). The runtime convertsf16scales tof32exactly when it loads the file.
Each array is zero-padded to a multiple of 64 bytes, so the next array starts aligned. The padding after the last array is optional.
Weight w = q × d. Quants must be in [-127, 127]. The loader rejects any
byte equal to -128, because the x86 kernels negate weights by the
activation's sign and -128 has no int8 negation.
Each block of 32 weights takes 16 bytes. Byte j of a block holds element j
in its low nibble and element j + 16 in its high nibble. Weight
w = (nibble - 8) × d, which gives values in [-8, 7].
Both the writer and the runtime's activation quantizer use these exact float32 operations:
amax = max |x| over the block
if amax < 1e-30: d = 0, q = 0 # 127/amax would overflow
else: d = amax / 127, q = trunc(x * (127/amax) + copysign(0.5, x)), clamp to ±127
Every SIMD implementation is bit-identical to the scalar one.
With f16 scales, a Q8_0 weight block's d = amax / 127 is first rounded to
the nearest f16, and the quants are trunc(x × (1/d) + copysign(0.5, x))
against that rounded d, clamped to ±127. A scale above 65504 cannot be
stored, and the writer refuses it.
Because Q4_0 weights are written only once, tools/mlb.py searches for the
best fit per block. It maps the block's largest-magnitude element to each level
from -6 to -8.4 in steps of 0.2 (clamped to [-8, 7]). For each candidate, d
is the least-squares fit to those levels with each element weighted by x², and
the candidate with the smallest weighted error wins. A block whose largest
magnitude is below 1e-6 is scaled up to 1 before fitting and its d scaled
back, so that float32 squares and cubes do not underflow. d can be negative.
With f16 scales the fitted d is rounded to the nearest f16. Any d and
nibbles are valid input to the runtime.
With tools/convert.py --q4-rule round (the default for Gemma 3), Q4_0 blocks
are rounded as llama.cpp does instead: d is the element of largest magnitude
divided by -8, and each level is trunc(x × (1/d) + 8.5), at most 15, minus 8.
Quantization-aware checkpoints trained for Q4_0 expect exactly this rounding.
With tools/convert.py --imatrix, each element is instead weighted as in
llama.cpp, by imp × sqrt(s² + x²). Here s² is the mean x² of the element's
row, and imp is the importance of its input channel: the channel's mean
squared input (from ml_run --imatrix) divided by the mean over all channels
of that matrix, with a floor of 0.001.
ml_run --imatrix FILE writes a 40-byte header followed by the sums, all
little-endian:
| Offset | Type | Meaning |
|---|---|---|
| 0 | char[4] |
magic "MLI1"
|
| 4 | u32 |
version, 1
|
| 8 | u32 |
n_layers |
| 12 | u32 |
dim |
| 16 | u32 |
hidden_dim |
| 20 | u32 |
q_dim if it differs from dim, else 0 |
| 24 | u64 |
tokens summed into the layers |
| 32 | u64 |
tokens summed into the classifier (prompt tokens prefilled without logits skip it) |
| 40 | float64[] |
sums of squares per channel: for each layer, the input of wq/wk/wv (dim), of wo (q_dim), of w1/w3 (dim) and of w2 (hidden_dim); then the classifier input (dim) |
Rotary embeddings act on interleaved pairs (2i, 2i+1) of each head. Pair i
uses the inverse frequency inv_i = rope_theta^(-2i / head_dim), as in
llama2.c. Hugging Face checkpoints use the rotate-half layout instead; the
converter permutes the rows of wq and wk to match.
With rope_type = 1 (llama3), each frequency is rescaled the same way
transformers does it. Here wavelen = 2π / inv_i, f is rope_factor,
lo and hi are the low and high frequency factors, and orig is
rope_orig_max_seq:
wavelen < orig / hi → inv_i (high frequency: unchanged)
wavelen > orig / lo → inv_i / f (low frequency: scaled)
otherwise → (1 - s) · inv_i / f + s · inv_i, s = (orig / wavelen - lo) / (hi - lo)
With rope_type = 2 (linear), every frequency is divided by f.
With rope_type = 3 (yarn), the frequencies are computed as transformers
computes them. Here d(r) is the dimension at which a wavelength fits r
rotations into orig:
d(r) = head_dim · ln(orig / (2π r)) / (2 ln rope_theta)
lo = max(floor(d(beta_fast)), 0), hi = min(ceil(d(beta_slow)), head_dim - 1)
ramp = clamp((i - lo) / (hi - lo), 0, 1)
inv_i → inv_i / f · ramp + inv_i · (1 - ramp)
The cos and sin tables are then multiplied by rope_attn_factor. The converter
takes attention_factor from the config, or computes 0.1 ln f + 1 or the
mscale / mscale_all_dim ratio, as transformers does.
| Problem | Status |
|---|---|
| file cannot be opened | ML_E_IO |
empty file, file shorter than the header, bad magic or version, unknown wtype or rope_type, invalid llama3, linear or yarn parameters, a rope_attn_factor without yarn, qkv_bias, qk_norm, layer_flags, post_norms or scale_f16 other than 0 or 1, an unknown act, a negative or non-finite embed_scale, attn_scale, attn_softcap or final_softcap, unknown layer flag bits, a layer with both the local RoPE base and no RoPE, a negative rope_local_theta or a local-RoPE layer without one, sliding_window above 2^24, file shorter than the arrays the header implies, a Q8 quant equal to -128 |
ML_E_FORMAT |
zero or over-limit dimension, dim or hidden_dim not a multiple of 32, dim % n_heads ≠ 0 without head_dim, n_heads % n_kv_heads ≠ 0, odd head_dim, q_dim not a multiple of 32 or above 65536, vocab < 3
|
ML_E_SHAPE |
All size arithmetic is overflow-checked, which matters on 32-bit targets.
tools/convert_tokenizer.py writes this from a Hugging Face tokenizer.json;
model/bpe.c reads it. All integers are little-endian.
| Offset | Type | Field | Meaning |
|---|---|---|---|
| 0 | char[4] |
magic | MLT1 |
| 4 | u32 |
version |
1 |
| 8 | u32 |
kind |
1 = SentencePiece-style BPE, 2 = byte-level BPE with the Llama 3 pre-tokenizer |
| 12 | u32 |
n_vocab |
tokens, at most the model's vocab
|
| 16 | u32 |
n_merges |
merge rules |
| 20 | i32 |
bos |
BOS id, or -1 |
| 24 | i32 |
eos |
EOS id, or -1 |
| 28 | u32 |
flags |
1: prepend BOS when encoding; 2: ignore_merges (a whole pre-token found in the vocabulary is one token); 4: SentencePiece prefix ▁ only at the start of the text, not after each added token; 8: byte-level, NFC-normalize the text between added tokens; 16: byte-level, numbers are split into single digits (\p{N} instead of \p{N}{1,3}); 32: byte-level, the GPT-2 regex instead of the Llama 3 one; 64: byte-level, every \p{N} is split off before the regex (a Digits(individual_digits) pre-tokenizer); other bits must be 0 |
Then n_vocab tokens, each u8 attr, u32 len and len bytes:
-
attr 1 (added): matched literally in the text before anything else
(
<s>,<|eot_id|>); the longest match wins. -
attr 2 (byte): a SentencePiece byte-fallback token
<0xXX>, stored as its single byte. -
Bytes: byte-level pieces are stored as the raw bytes they stand for (not
the GPT-2 printable characters of
tokenizer.json); SentencePiece pieces as UTF-8 with▁(U+2581) for spaces.
Then n_merges rules of three u32: left, right and merged token, in rank
order. Encoding applies the lowest-ranked applicable merge first, the
leftmost among equal ranks, which is what the tokenizers library does.
Generated from docs/file-format.md; edit it there.