The defect
readRows in bench/m5-analysis.ts:
if (/^m5-seeds-.*\.jsonl$/.test(name) || /\.jsonl$/.test(name)) files.push(join(target, name));
The second clause matches any .jsonl, so the first is decoration. npm run bench:m5 defaults to bench/results, which currently holds 22 files and 1,835 rows:
t702-m1-final.jsonl M1 — a different run
t702-m3-invalidated.jsonl invalidated
t702-m4-final.jsonl M4 — withdrawn as guard evidence
t702-pilot-haiku45-superseded.jsonl
m5-off-design-20-tasks.jsonl deviation 1: "not part of the analysis set and not citable"
t703-ablation.jsonl different arms
token-ledger-…jsonl metric rows, not runs
deterministic-…, external-corpus-…, decision-delivery-…, m4-qualification
Why it is worse than a wrong number
REGISTERED_ROWS = 1160 and the §8 guard is:
if (rows.length < REGISTERED_ROWS) { … refuse … }1,835 > 1,160. So the stopping rule passes on the strength of contamination, and the script computes a 2×2 mixing M1, M4, an invalidated run and a file the pre-registration names as non-citable into the registered table — while printing rows read: 1835 of the registered 1160, which reads like a surplus rather than an error.
The one guard written to enforce the pre-registration is the thing that would break it.
Found before it mattered
The re-run is at 880 of 1,160 registered measurements. Nothing has been computed — bench:m5 has only ever been run with --validate, which takes an explicit file (t702-m1-final.jsonl) and does not use this path.
Fix
The analysis set is not "whatever is in the directory". It is the six shards deviations 3 and 4 name:
- surviving, error-free:
m5-seeds-21-30, 31-40, 41-50, 51-58 (seeds 55–58 superseded by the re-run)
- re-run:
m5-seeds-1-10-rerun, m5-seeds-11-20-rerun, m5-seeds-55-58-rerun
An explicit list, with the script failing loudly on a file it was told to read but cannot find — the opposite of silently accepting one it was not.
While there: deviations 3 and 4 oblige the verdict to report the production window per shard and state that seeds 1–20 and 55–58 are re-runs. The script does not do that yet, and adding it after the numbers are visible would be the wrong order.
The defect
readRowsinbench/m5-analysis.ts:The second clause matches any
.jsonl, so the first is decoration.npm run bench:m5defaults tobench/results, which currently holds 22 files and 1,835 rows:Why it is worse than a wrong number
REGISTERED_ROWS = 1160and the §8 guard is:1,835 > 1,160. So the stopping rule passes on the strength of contamination, and the script computes a 2×2 mixing M1, M4, an invalidated run and a file the pre-registration names as non-citable into the registered table — while printing
rows read: 1835 of the registered 1160, which reads like a surplus rather than an error.The one guard written to enforce the pre-registration is the thing that would break it.
Found before it mattered
The re-run is at 880 of 1,160 registered measurements. Nothing has been computed —
bench:m5has only ever been run with--validate, which takes an explicit file (t702-m1-final.jsonl) and does not use this path.Fix
The analysis set is not "whatever is in the directory". It is the six shards deviations 3 and 4 name:
m5-seeds-21-30,31-40,41-50,51-58(seeds 55–58 superseded by the re-run)m5-seeds-1-10-rerun,m5-seeds-11-20-rerun,m5-seeds-55-58-rerunAn explicit list, with the script failing loudly on a file it was told to read but cannot find — the opposite of silently accepting one it was not.
While there: deviations 3 and 4 oblige the verdict to report the production window per shard and state that seeds 1–20 and 55–58 are re-runs. The script does not do that yet, and adding it after the numbers are visible would be the wrong order.