chore(gate): walk the ADR-0003 pins, delete the placeholder versions.json

[myselfsiddharth]

Closes #26.

There were two version matrices and only one was real. run-matrix.ts iterated
experiments/gate-v1/versions.json, which held a single pending-b1@placeholder entry — so
npm run gate:matrix walked a fake list and any report generated from it was a report about
nothing.

scripts/testbed/matrix.json (ADR-0003) is now the single source of truth, read through
src/testbed/matrix.ts rather than a second ad-hoc JSON import.

versions.json is deleted, not kept in sync

Two lists for one thing drift. Its only non-duplicated fields were per-version site_key /
task_key overrides, and those were actively misleading: they relabelled the hand-written
2-step local-demo program per version as though a Grafana run had happened.

So only testbed_version varies per run now. site_key and task_key stay whatever the
compiled program actually is. That is the honest split — the run really is on the local-demo
program; only which matrix slot it belongs to changes.

What's new

--versions <a,b,c> / --versions all (default all). Unknown ids are rejected with the
valid list and exit 2 — never a fall-through to a default, which is exactly how a
placeholder run gets mistaken for a measurement:

$ npm run gate:matrix -- --dry-run --versions 99.0.0
gate:matrix: unknown version id(s): 99.0.0
valid ids (scripts/testbed/matrix.json): 9.5.21, 10.0.13, 10.4.19, 11.0.0, 11.5.2, 12.0.0, 12.2.1, 13.0.3
$ echo $?
2

"status": "unavailable" versions are skipped, logged, and recorded — never silently
dropped. New out/matrix-run.json ledger carries selection, versions_in_matrix,
versions_walked, and versions_skipped[] {id, reason}. Without it a later reader cannot tell
"8 pins, 3 unavailable" from "5 pins" and the denominator shrinks silently.

No pin carries the flag today, so I proved it by temporarily marking 9.5.21 unavailable
(reverted — matrix.json is unchanged in this diff):

gate:matrix dry-run — grafana-oss matrix, 7 version(s) to walk, program=local-demo-click
  skipped 9.5.21: TEMPORARY probe for issue #26 — not a real finding
...
"versions_in_matrix": 8,
"versions_walked": ["10.0.13","10.4.19","11.0.0","11.5.2","12.0.0","12.2.1","13.0.3"],
"versions_skipped": [{"id":"9.5.21","reason":"TEMPORARY probe for issue #26 — not a real finding"}]

isUnavailable / availableVersions are unit-tested against a synthetic matrix, plus a test
asserting no pin is currently marked unavailable.

Exit 1 when the selection walks zero versions. Not in the issue, but an empty NDJSON is a
missing denominator, not a successful run, and a CI caller should notice:

$ npm run gate:matrix -- --dry-run --versions 9.5.21   # with 9.5.21 unavailable
gate:matrix: walked 0 versions — nothing was measured. 1 of 1 selected version(s) are unavailable.
$ echo $?
1

Still not a measurement

Unchanged on purpose: the --dry-run gate (live execution is #62), hard-coded PASS outcomes,
and zero token counts. Every emitted row still carries
dry-run — tokens remain 0; not a gate measurement, and out/matrix-run.json repeats it in a
not_a_measurement field. Walking eight versions instead of one makes the row count honest,
not the numbers meaningful.

Verified output:

$ npm run gate:matrix -- --dry-run
gate:matrix dry-run — grafana-oss matrix, 8 version(s) to walk, program=local-demo-click
… 8 rows, one per version …

total rows: 24   (16 step + 8 run)
testbed_version per run row: 9.5.21, 10.0.13, 10.4.19, 11.0.0, 11.5.2, 12.0.0, 12.2.1, 13.0.3
all token values seen: [0]

$ npm run gate:report
{"rows":24,"status":"computed_from_ndjson", …}
  step-level replay-validity                value=1     status=computed
  task-level success (with ≤2 repairs/run)  value=1     status=computed
  repair cost vs fresh (tokens)             value=None  status=no_data
  repair cost vs fresh (wall-clock)         value=None  status=no_data
  self-heal success rate                    value=None  status=no_data
  time-to-repair (mean ms)                  value=None  status=no_data
  amortized tokens/task over N runs         value=0     status=computed

No invented rates; the empty denominators still report no_data.

Docs

experiments/gate-v1/README.md (version-list section, commands, exit codes, skip ledger, open
questions), docs/gate/runner.md open questions, docs/architecture.md (package table,
artifact table, and stub-table row 4 — which was about this placeholder), and the
dependabot.yml comment that pointed at versions.json.

fixtures/compiled-program.json still declares testbed_version: "pending-b1@placeholder". Left
alone deliberately — it is accurate, that program was compiled against no testbed at all, and
the runner overrides it per version. Noted in the README.

Merge note

Test-merged against my three other open branches: #80 and #79 merge clean. #78 conflicts
in docs/architecture.md — one hunk in the package table, because #78 inserts a src/shared/
row directly above the experiments/gate-v1/ row whose "Contract read" cell I changed.
Resolution is to keep both: #78's new src/shared/ row, then this branch's edited
experiments/gate-v1/ row.

Tests

npm run ci            # green — 48 unit (3 new), 1 integration
npm run test:canary   # 6 pass

🤖 Generated with Claude Code

[OM152002]

Approve. I ran it rather than read it, and every behavioural claim checked out.

unknown id (--versions 99.0.0)   exit=2   + valid-id list printed
full dry-run                     exit=0   walks all 8 pins
live attempt (no --dry-run)      exit=2
out/metrics.ndjson               24 rows = 8 versions x (2 steps + 1 run)
out/matrix-run.json              selection / versions_in_matrix: 8 / versions_walked[8]

Deleting versions.json instead of syncing it is the right call, and the reason given is the real one: two lists for one thing drift, and its only non-duplicated fields were the actively misleading part — per-version site_key / task_key relabelling the local-demo program as though a Grafana run had happened. Letting only testbed_version vary is the honest split.

Rejecting unknown ids with exit 2 rather than falling through to a default is the detail I'd have asked for if it were missing. A silent default is exactly how a placeholder run gets mistaken for a measurement, which is the bug this PR is closing.

The skip ledger earns its place — without versions_skipped[] {id, reason} a later reader cannot distinguish "8 pins, 3 unavailable" from "5 pins", and the denominator shrinks with nobody noticing. Proving it by temporarily marking 9.5.21 unavailable and reverting is the right way to test a branch no pin currently exercises.

Merges clean against main and #80, despite touching src/testbed/matrix.ts and tests/unit/testbed.test.ts, which #80 also edits. Worth re-checking after whichever lands first, since that overlap is real even though git resolves it today.

The thing I appreciate most is what the PR refuses to claim: walking eight versions over the same hand-written 2-step program is called "a better-shaped denominator, not a measurement", and stub 4 in architecture.md is narrowed rather than deleted — the dry-run PASS with zero tokens is still flagged. That is the distinction that keeps this row honest when someone reads the report later.

[myselfsiddharth]

Rebased onto main @ 19f0ff6 (which now includes #80) — 23b73a6. Conflicting → mergeable.

One real conflict, in docs/architecture.md: #74 added the src/shared/ row to the package table immediately above the experiments/gate-v1/ row this PR edits. Kept both — src/shared/ untouched, and experiments/gate-v1/'s "contract read" still becomes scripts/testbed/matrix.json (via src/testbed/matrix.ts).

src/testbed/matrix.ts and tests/unit/testbed.test.ts auto-merged, and the merge is coherent rather than merely clean: #80 rewrote testdataTypeFor and its boundary tests, this PR adds isUnavailable / availableVersions and the availability tests below them. No overlap in either file.

One line added on top of the rebase: MatrixVersion.status's comment now records that no pin carries unavailable today — #23 booted and seeded all eight — so the flag reads as a guard for the day a tag stops resolving rather than as an unfinished verification. The test asserting no pin is currently unavailable was already here and still passes.

Re-verified after the rebase: npm run ci green (93 unit + 1 integration), gate:matrix --dry-run walks all 8 pins and writes out/matrix-run.json, and --versions 99.0.0 still exits 2 with the valid-id list.

🤖 Generated with Claude Code