Fixed
waste infoandwaste runcrashed on K3 on every x86 build
(#10). The tensors the
loader skips — the vision tower, and anything outsidetensor_prefix—
keptgroupat 0, and the row-scratch sizing divided by it. The
architecture decided what that meant: arm64'ssdivanswers 0 and the
run continues, x86'sidivraises#DE.waste planwas unaffected
because it does not load. §37.WASTE_Q8=0could not load a 4-bit trunk
(#6) — that is, any
container a defaulttools/convert.pyrun produces. The dequantizer
read one byte per weight, true of Q8G alone, while catching every
quantized format. It now decodes throughwaste_deq_row, the one place
that knows all three widths. The same lines also predatedwaste_f16's
subnormal fix and flushed group scales below 6.1e-05 to zero.embed_tokensstays on disk underWASTE_Q8=0, as it does
otherwise: 7.93 → 6.52 GiB of peak RSS on Kimi-Linear, identical logits.
The f32-equivalence check now differs from the default path in the
storage width alone, which is what it claims to compare.
Added
- Router lookahead in the decode path. At the end of a MoE layer, once
its reads are consumed and the disk is about to idle through the next
layer's attention, layer L+1's router runs on layer L's hidden state and
issues speculative reads for its top 6. Demand hit rate 14–19% → 38–40%
with total bytes read unchanged (254.2 → 254.5 GB): the records were
going to be read anyway, and only when changes. Nine paired runs,
median 1.17x.WASTE_LOOKAHEAD=0disables. §34, §35. WASTE_MLOCKwires the trunk and the expert cache;WASTE_MLOCK=cache
wires the cache alone. Off by default — Linux'sRLIMIT_MEMLOCKis
commonly 8 MB. Wiring the trunk is worth 3x in the transition zone around
52 GiB and nothing below it; it does not move the knee. §30, §31, §32.
Changed
tests/run.shgenerates its own PyTorch oracle from the container
under test (16.9 s) instead of diffing against a shipped fixture, which
can only ever be valid for the container that produced it: expert
codebooks are k-means, and the same seed on a different--device
trains different books — one layer of 26 moves the logits by 1.24
against a 1e-3 threshold. The fixture remains as the fallback whereuv
is absent, with its provenance recorded beside it.
(#7), §33.tools/make_test_container.pyemits what a real conversion does: a
Q4G/Q8G/F32trunk rather than Q8G throughout, and--prefixfor a
container whose tensors are not all under itstensor_prefix. Both are
shapes the suite could not previously reach, and both had a live bug
behind them.- Checks that cannot run now say why instead of reporting a refusal as a
divergence:WASTE_Q8=0on K3 wants 211 GB of f32 trunk on a 64 GB
machine, the oracle prompt is Kimi-Linear's, and a cold hotlist run that
already missed nothing demonstrates neither outcome
(#5).
Measured and not adopted
- Cross-layer prefetch from
next_layer_top— gated before building
and refused: 29.0% recall against a 60% break-even. §29, revisited and
superseded by the lookahead above in §34. - The lookahead in the prefill path. Built, measured, removed: a chunk
layer's disk is busy continuously, so a prefetch there does not move a
read into idle time, it moves it in front of another read and pays an
eviction for it — 7% more bytes. Decode keeps it. §36.
Section numbers refer to docs/LEARNED.md, which carries the reasoning and the full measurements behind each entry.
Full Changelog: v0.6.1...v0.6.2