Skip to content

The README's generated numbers block still publishes M4; M5 measured the thing the README leads with #480

Description

@MongLong0214

Found while putting M5 into the README for 0.7.0.

What is stale

bench/report.ts's README_SOURCES declares one file:

bench/results/t702-m4-final.jsonl   112 rows   model: not recorded

The block it generates is what scripts/check-readme-numbers.mjs regenerates and byte-compares in CI. It is the only place the README is allowed to carry a measured statistic — and it publishes M4, while the README's behaviour claim now rests on M5.

Why this was not fixed in the release

README_SOURCES is deliberate machinery, not a config value. Its own comment explains why a glob would be dangerous, why M1/M1-b/M2 are not pooled, and why the ablation log is excluded. Repointing it at M5 means declaring seven shards with a status note covering supersession (80 cells superseded by re-run), the excluded error row, and the per-arm truncation split — the same care the M4 note took.

Doing that between a CI failure and a tag would have been the wrong moment. The release ships with the statistics living in bench/VERDICT-M5.md and the README linking there.

What the gate did right

It rejected the first attempt, which had p = 0.0000, −16.1pp and Fisher exact typed into the prose. That is exactly the failure it exists to prevent: a statistic retyped into a paragraph drifts from the log that produced it, and the generated block below is worth nothing if the paragraph above it carries a hand-written one.

Scope

  • Declare the seven M5 shards in README_SOURCES with a status note in the M4 note's register: supersession rule, exclusions, the single harness and dist digest, and what the number does not cover ([claim]-tier delivery, one model, ten constructed fixtures).
  • Decide whether M4 stays declared beside M5 or is retired to historical — they are different matrices with different arms, and the existing note already says pooling them would be wrong.
  • Regenerate all four READMEs with --write.

Acceptance

  • The generated block reports M5's rows, run ids, driver, model and matrix
  • node scripts/check-readme-numbers.mjs passes with no --write
  • The status note names the supersession rule and the excluded row
  • Whatever happens to M4 is stated in the note rather than implied by its absence

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions