v1.24.0 — three defects that were live, and a disk tier we measured and missed

Three defects were live in 1.23.0. One of them could only ever make the tool look cautious.

pip install --upgrade quantprobe

The one worth upgrading for

upgrade_advisor built its counterfactual by copying the baseline and overriding one resource — but it dropped n_layer and true_size_gb on the way in. evaluate() re-derived both from presets, so the thing being priced was not your model with more RAM. It was a different model with more RAM.

The error is one-directional: it could only ever suppress good upgrades, never invent bad ones. So every symptom looked like the tool being conservative. If it told you "more RAM won't help," it may simply have been mis-pricing the question. 31 of 340 grid cells.

Also fixed:

• The speculation block printed a multiplier the row could not reach — 17 of 127 cells, true bound 1.025–1.229× against a printed 2.10×.
• A +73% prefill figure was quoted flat on every row, including all-in-VRAM rows where the same flag measured −39%. And it ignored which ubatch was actually emitted, so a 3 GB card sized to -ub 1024 (measured +38.7%) was told +73%.
• Partial calibration printed the same "calibration applied" line as a complete one — while measuring 12.5% median error against 8.8% for the fully-preset baseline. Worse than no calibration, presented identically.
• The disk probe took one sample and asked you to spot the bad ones. Its docstring said "treat a number above ~1.5 GB/s as evidence of a warm cache." Measured: with 73% of a file warmed, 6 of 8 draws returned RAM speed (max 2.854 GB/s — a 6.3× error), and 1 in 8 even after deliberate eviction. It now takes the minimum over N disjoint regions.

The disk tier: measured for the first time, and we missed

Every previous release validated against 14 ladder rows, all VRAM- or RAM-resident. Zero touched the disk tier. A component no validation row reads can stay wrong indefinitely.

So we staked a prediction and measured it:

predicted	0.332 tok/s
measured	0.476 tok/s
error	−30.4% — outside our own band

The law is pessimistic about streaming from disk. We have not repriced it, because one datapoint is not a coefficient.

What the miss cost to chase, in order: the access-pattern explanation was refuted (scattered 0.452 vs contiguous 0.459 — and the same experiment independently confirmed the corrected disk calibration). The byte deficit is real but over-explains the speed-up by 1.84×, so two errors are partially cancelling. A per-fault latency mechanism was staked, appeared to confirm at r²=0.9999, and was withdrawn — fault counts were derived from byte counts, making "per-fault latency" and "lower throughput" the same equation in different units.

The disk tier ships disclosed as 30% pessimistic rather than quietly fitted.

What is new

• Layer-by-layer streaming placement — one layer resident, the rest streaming, so VRAM scales with layer size instead of model size. Shown but never selected as the winner: its number is an upper bound with two named unpriced terms.
• KL divergence as a quality delta. Perplexity called a Q2_K regression "23% worse." The model picks a different token on 27.4% of positions. llama.cpp was already computing this on the passes we ran; we were discarding it.
• Every cited evidence file is now in the repository, and a gate enforces it. The register cited 109 paths; 37 were missing. For a project claiming every number is re-derivable, an unverifiable citation is the worst defect available — it looks like rigour and isn't.

Refutations this cycle

Eight, five of them our own claims, including two sub-noise-floor "improvements" I announced and retracted, a per-fault mechanism I staked and withdrew, and a "damage hides in the tail" framing the data corrected — the tail wasn't special; KL beats perplexity at every percentile.

Full ledger: FINDINGS.md · preregistrations · CHANGELOG

Still not validated

The disk tier's 30% pessimism, the 1.82× streaming-efficiency gap between llama.cpp and raw reads, expert-usage skew (inferred by elimination, never observed), and one ladder row — gemma4-12B — that returned 13.23 / 12.25 / 15.62 across three passes and should not be cited until diagnosed.

All stated in the changelog at the same size as the wins.