Found while reviewing #390, which fixed the drift but not the reason it went unnoticed.
Measured
bench/verify.mjs validates benchmark result rows against bench/schema/result.schema.json. Nothing runs it:
$ grep -rn "verify.mjs" .github/workflows/ package.json
(no matches)
It has no npm script and no CI step. .github/workflows/ci.yml runs check-engines, both tsc projects, the build, the dist/ staleness check, vitest, check-test-files-ran, spec/verify.sh, the perf test, check-readme-numbers.mjs, validate --range, doctor, and a fresh-clone smoke test — but never the schema gate over bench/results/.
What that cost
bench/results/m5-off-design-20-tasks.jsonl was committed on 2026-08-01 and sat in the tree failing the gate for two days:
$ git stash && git checkout origin/dev~N && node bench/verify.mjs bench/results/m5-off-design-20-tasks.jsonl
FAIL …:78 / must NOT have additional properties; …
FAIL …:79 …
FAIL …:80 …
verify: 80 problem(s) across 80 rows
Five fields — reproposal_matches and four rejected_path_* — reached the runner's rows without reaching the schema. #390 fixed the schema; the file now validates. But a gate nobody runs will drift again the next time a field is added, and the next person to add one has no reason to know it exists.
This is the same failure shape docs/RELEASE-GATE.md was written about: "A gate nobody can check is a slogan."
Shape of the fix
An npm script and a CI step that runs bench/verify.mjs over the result files whose shape the current schema is supposed to describe.
The one real decision is which files. bench/results/ holds rows from several eras — matrices withdrawn, superseded, and recorded under earlier schemas — and ADR-0018's provenance rules mean an old file is not automatically wrong for failing a newer schema. Validating everything would either fail permanently or force the schema to describe every historical shape at once. The scope needs to be stated deliberately and the reason recorded, not inferred from whatever happens to pass today.
Found while reviewing #390, which fixed the drift but not the reason it went unnoticed.
Measured
bench/verify.mjsvalidates benchmark result rows againstbench/schema/result.schema.json. Nothing runs it:It has no npm script and no CI step.
.github/workflows/ci.ymlrunscheck-engines, bothtscprojects, the build, thedist/staleness check,vitest,check-test-files-ran,spec/verify.sh, the perf test,check-readme-numbers.mjs,validate --range,doctor, and a fresh-clone smoke test — but never the schema gate overbench/results/.What that cost
bench/results/m5-off-design-20-tasks.jsonlwas committed on 2026-08-01 and sat in the tree failing the gate for two days:Five fields —
reproposal_matchesand fourrejected_path_*— reached the runner's rows without reaching the schema. #390 fixed the schema; the file now validates. But a gate nobody runs will drift again the next time a field is added, and the next person to add one has no reason to know it exists.This is the same failure shape
docs/RELEASE-GATE.mdwas written about: "A gate nobody can check is a slogan."Shape of the fix
An npm script and a CI step that runs
bench/verify.mjsover the result files whose shape the current schema is supposed to describe.The one real decision is which files.
bench/results/holds rows from several eras — matrices withdrawn, superseded, and recorded under earlier schemas — and ADR-0018's provenance rules mean an old file is not automatically wrong for failing a newer schema. Validating everything would either fail permanently or force the schema to describe every historical shape at once. The scope needs to be stated deliberately and the reason recorded, not inferred from whatever happens to pass today.