Found while putting M5 into the README for 0.7.0.
What is stale
bench/report.ts's README_SOURCES declares one file:
bench/results/t702-m4-final.jsonl 112 rows model: not recorded
The block it generates is what scripts/check-readme-numbers.mjs regenerates and byte-compares in CI. It is the only place the README is allowed to carry a measured statistic — and it publishes M4, while the README's behaviour claim now rests on M5.
Why this was not fixed in the release
README_SOURCES is deliberate machinery, not a config value. Its own comment explains why a glob would be dangerous, why M1/M1-b/M2 are not pooled, and why the ablation log is excluded. Repointing it at M5 means declaring seven shards with a status note covering supersession (80 cells superseded by re-run), the excluded error row, and the per-arm truncation split — the same care the M4 note took.
Doing that between a CI failure and a tag would have been the wrong moment. The release ships with the statistics living in bench/VERDICT-M5.md and the README linking there.
What the gate did right
It rejected the first attempt, which had p = 0.0000, −16.1pp and Fisher exact typed into the prose. That is exactly the failure it exists to prevent: a statistic retyped into a paragraph drifts from the log that produced it, and the generated block below is worth nothing if the paragraph above it carries a hand-written one.
Scope
- Declare the seven M5 shards in
README_SOURCES with a status note in the M4 note's register: supersession rule, exclusions, the single harness and dist digest, and what the number does not cover ([claim]-tier delivery, one model, ten constructed fixtures).
- Decide whether M4 stays declared beside M5 or is retired to historical — they are different matrices with different arms, and the existing note already says pooling them would be wrong.
- Regenerate all four READMEs with
--write.
Acceptance
Found while putting M5 into the README for 0.7.0.
What is stale
bench/report.ts'sREADME_SOURCESdeclares one file:The block it generates is what
scripts/check-readme-numbers.mjsregenerates and byte-compares in CI. It is the only place the README is allowed to carry a measured statistic — and it publishes M4, while the README's behaviour claim now rests on M5.Why this was not fixed in the release
README_SOURCESis deliberate machinery, not a config value. Its own comment explains why a glob would be dangerous, why M1/M1-b/M2 are not pooled, and why the ablation log is excluded. Repointing it at M5 means declaring seven shards with a status note covering supersession (80 cells superseded by re-run), the excluded error row, and the per-arm truncation split — the same care the M4 note took.Doing that between a CI failure and a tag would have been the wrong moment. The release ships with the statistics living in
bench/VERDICT-M5.mdand the README linking there.What the gate did right
It rejected the first attempt, which had
p = 0.0000,−16.1ppandFisher exacttyped into the prose. That is exactly the failure it exists to prevent: a statistic retyped into a paragraph drifts from the log that produced it, and the generated block below is worth nothing if the paragraph above it carries a hand-written one.Scope
README_SOURCESwith a status note in the M4 note's register: supersession rule, exclusions, the single harness and dist digest, and what the number does not cover ([claim]-tier delivery, one model, ten constructed fixtures).--write.Acceptance
node scripts/check-readme-numbers.mjspasses with no--write