DevToolie · myselfsiddharth · Jul 28, 2026 · Jul 28, 2026
@@ -45,7 +45,8 @@ updates:
     ignore:
       # Playwright ships the browser build that gate measurements run against, so a
       # major bump changes churn results. Bump it deliberately in its own PR
-      # alongside experiments/gate-v1/versions.json. Security updates still land.
+      # alongside scripts/testbed/matrix.json (the single version matrix since
+      # #26 deleted experiments/gate-v1/versions.json). Security updates still land.
       - dependency-name: playwright
         update-types:
           - version-update:semver-major

@@ -111,7 +111,7 @@ record → compile → cache-write → replay) is the issue that closes both.
 | `src/runner/` | Replay a compiled program in Playwright; repair actions only on failure; emit measured metrics | `src/runner/index.ts` (library only — driven by `experiments/gate-v1/run-matrix.ts`) | `cache-row.schema.json`, `assertion.schema.json` shapes (via `CompiledProgram`) | none directly — emits through `src/metrics/` | [gate/runner.md](./gate/runner.md) |
 | `src/metrics/` | Cost arithmetic, NDJSON emitter, PRD §9 aggregates that report `no_data` on an empty denominator | `src/metrics/index.ts` (library only) | `metrics.schema.json` (`readMetricNdjson`) | `metrics.schema.json` | §9 sections in [prd/PRD-trajectory-cache-v0.2.md](./prd/PRD-trajectory-cache-v0.2.md) |
 | `src/shared/` | **Not a pipeline stage.** In-page JS source strings two capture sites must run identically — today the `visible_landmarks` enumeration | `src/shared/index.ts` (library only) | none | none — feeds the `trajectory.schema.json` `visible_landmarks` field written by the recorder | see below |
-| `experiments/gate-v1/` | Throwaway harness: walk the version list, emit rows, render the report. **Not a product API** | `npm run gate:matrix`, `npm run gate:report` | `metrics.schema.json`, local `versions.json` | `metrics.schema.json` | [experiments/gate-v1/README.md](../experiments/gate-v1/README.md) |
+| `experiments/gate-v1/` | Throwaway harness: walk the version list, emit rows, render the report. **Not a product API** | `npm run gate:matrix`, `npm run gate:report` | `metrics.schema.json`, `scripts/testbed/matrix.json` (via `src/testbed/matrix.ts`) | `metrics.schema.json` | [experiments/gate-v1/README.md](../experiments/gate-v1/README.md) |
 
 `src/cache/` has no spec doc under `gate/`; its contract is
 [privacy/boundary-spec.md](./privacy/boundary-spec.md) and the merge-blocking canary
@@ -163,7 +163,7 @@ assertion against `assertion.schema.json`); the wrapper around them is not.
 | `experiments/gate-v1/out/report/report.{json,csv,html}`, `amortized.svg` | `npm run gate:report` | gitignored |
 | `scripts/testbed/.runtime/<version>/provisioning/` | `prepareProvisioningOverlay` on every testbed invocation, including `--dry-run` | gitignored (`scripts/testbed/.gitignore`) |
 | `experiments/gate-v1/fixtures/compiled-program.json` | hand-written | **committed** — stands in for a real compiled program, see break 2 |
-| `experiments/gate-v1/versions.json` | hand-written | **committed** — one-element placeholder, see below |
+| `experiments/gate-v1/out/matrix-run.json` | `npm run gate:matrix -- --dry-run` | gitignored — selection, versions walked, versions skipped with reason |
 
 Generated output that is gitignored **must stay that way**. The recorder's own artifacts are
 committed on purpose: they are fixtures, and `assertNoLiteralSecrets` runs on every write
@@ -180,7 +180,7 @@ The section a new agent needs most. Blunt, and current as of `605c384` + ADR-000
 | 1 | **Repair proposes nothing.** `StubRepairModelClient` always returns `corrected_action: null`, `tokens_in: 0`, `tokens_out: 0` | `src/runner/repair.ts:17-27` | Every repair attempt lands on `REPAIR_EXHAUSTED`. Self-heal rate is structurally 0; `cost_repair` tokens are structurally 0 | [#27](https://github.com/DevToolie/Paragent/issues/27) |
 | 2 | **`cost_fresh` is always zeros.** `ReplayRunner` defaults it to `zeroCost()` and no caller ever passes it | `src/runner/replay.ts:79`; no `costFresh` argument anywhere in `src/` or `experiments/` | The §9 kill line "repair cost ≥ 70% of fresh" has no denominator. `repairCostVsFresh` correctly returns `status: no_data` rather than a ratio | [#39](https://github.com/DevToolie/Paragent/issues/39) |
 | 3 | **The matrix runner refuses to run live.** `--dry-run` is mandatory; anything else exits 2 | `experiments/gate-v1/run-matrix.ts:49-54` | No browser is ever driven by the harness. Dry-run outcomes are hard-coded `PASS` with zero tokens — explicitly *not* a gate measurement | [#62](https://github.com/DevToolie/Paragent/issues/62) |
-| 4 | **`versions.json` is a placeholder, not the matrix.** One entry, `id: "pending-b1@placeholder"` | `experiments/gate-v1/versions.json` | The gate harness walks one fake version while `scripts/testbed/matrix.json` holds the real eight ADR-0003 pins. The two files are unconnected | [#26](https://github.com/DevToolie/Paragent/issues/26) |
+| 4 | ~~`versions.json` is a placeholder, not the matrix.~~ **Fixed by [#26](https://github.com/DevToolie/Paragent/issues/26)** — deleted; `run-matrix.ts` reads `scripts/testbed/matrix.json` through `src/testbed/matrix.ts` | `experiments/gate-v1/run-matrix.ts` | The harness now walks the real eight ADR-0003 pins, one run row each, and records skipped versions in `out/matrix-run.json`. **Still not a measurement** — dry-run outcomes remain hard-coded `PASS` with zero tokens (stub 3) | — |
 | 5 | **The cache has a write path only.** No read path, no persistence — the only `CacheStore` in the tree is `{ write(_row) { /* sink */ } }`; no `writeFile`/`appendFile` anywhere in `src/cache/` | `src/cache/pipeline.ts:125`; `src/cache/write.ts` | Nothing can be replayed *from* cache. There is no cache hit, so there is no cache hit-rate | [#63](https://github.com/DevToolie/Paragent/issues/63) |
 | 6 | **Confidence never moves.** `confidence`, `success_count`, `failure_count` are written as `0` and never updated by any code path | `src/compiler/compile.ts:85-87`; `src/cache/pipeline.ts:91` | PRD §5.3's self-invalidating, self-healing cache does not exist. The fields are shape, not behaviour | [#64](https://github.com/DevToolie/Paragent/issues/64) |
 | 7 | **Bundle → cache and bundle → runner are unwired *in the runtime*** (the two dashed edges above) | no import of `src/cache/` outside `src/cache/` and `tests/`; the `CompiledTrajectoryBundle` → `CompiledProgram` adapter lives in `tests/integration/pipeline.test.ts`, deliberately not in `src/` | Narrower than it was: since [#52](https://github.com/DevToolie/Paragent/issues/52) the seam **is** exercised end to end by the integration test, which caught a real compiler bug on its first run. What is still missing is a *product* path — nothing outside a test walks bundle → cache → replay | [#62](https://github.com/DevToolie/Paragent/issues/62), [#63](https://github.com/DevToolie/Paragent/issues/63) |

@@ -70,3 +70,11 @@ npm run gate:report
 - Whether `compiled_trajectory` bundle `$id` becomes a first-class contract (B3 packaging convention today).
 - Fresh-reasoning cost capture for `cost_fresh` — measured separately; defaults to zeros when unwired.
 - Live `page` injection API for matrix vs caller-owned browser lifecycle — not locked yet.
+- ~~Which version list the gate matrix walks.~~ **Settled (#26)** — `scripts/testbed/matrix.json`
+  (ADR-0003 pins), read through `src/testbed/matrix.ts`. The placeholder
+  `experiments/gate-v1/versions.json` is deleted, so `npm run gate:matrix -- --dry-run` now
+  emits one run row per pinned version instead of one row for `pending-b1@placeholder`. Still a
+  dry run: outcomes are hard-coded `PASS`, every token count is 0, and the rows say so.
+- Whether walking eight versions changes anything the report can *conclude*. It does not — more
+  rows over the same hand-written 2-step program is a better-shaped denominator, not a
+  measurement. That waits on live execution ([#62](https://github.com/DevToolie/Paragent/issues/62)).
@@ -4,7 +4,7 @@ doc_type: runbook
 status: draft
 owner: B4
 created: 2026-07-24
-updated: 2026-07-25
+updated: 2026-07-27
 confidence: MED
 supersedes: null
 sources_verified: true
@@ -18,24 +18,59 @@ sources_verified: true
 
 | Path | Role |
 | --- | --- |
-| `versions.json` | Pluggable version matrix (prefer B1 `scripts/testbed/matrix.json` when wired) |
 | `fixtures/compiled-program.json` | 2-step local demo program |
 | `fixtures/local-demo.html` | Static page for future live runs |
 | `trajectories/` | B2 recorded trajectories (fixture gate task) |
 | `run-matrix.ts` | Matrix CLI (`--dry-run` required for now) |
 | `report/generate-amortized.ts` | SVG/CSV/HTML/JSON from NDJSON (`writeReport`) |
 | `out/` | Generated metrics + reports (gitignored) |
 
+## The version list
+
+[`scripts/testbed/matrix.json`](../../scripts/testbed/matrix.json) — the ADR-0003 pins — is the
+**only** version list, read through `src/testbed/matrix.ts`. This directory used to carry its
+own `versions.json` holding a single `pending-b1@placeholder` entry, so every report generated
+here was a report about nothing. It was deleted in
+[#26](https://github.com/DevToolie/Paragent/issues/26) rather than kept in sync: two lists for
+one thing drift, and its only non-duplicated fields (`site_key` / `task_key` overrides) were
+actively misleading — they relabelled the local-demo program per version as if a Grafana run had
+happened.
+
+Consequently **only `testbed_version` varies per run.** `site_key` and `task_key` stay whatever
+the compiled program actually is.
+
+A version marked `"status": "unavailable"` in `matrix.json` is **skipped and recorded**, never
+silently dropped — see `out/matrix-run.json`.
+
 ## Commands
 
 ```bash
-npm run gate:matrix -- --dry-run
+npm run gate:matrix -- --dry-run                          # every available pin
+npm run gate:matrix -- --dry-run --versions 11.0.0,12.0.0 # a subset
+npm run gate:matrix -- --dry-run --versions all           # explicit default
 npm run gate:report
 npm run recorder -- --fixture
 ```
 
-Dry-run emits zero-token step/run rows under `experiments/gate-v1/out/metrics.ndjson`.
-Live runs exit **2** until B1 pins and Playwright live wiring land.
+Unknown ids are rejected with the valid list, never defaulted:
+
+```text
+gate:matrix: unknown version id(s): 99.0.0
+valid ids (scripts/testbed/matrix.json): 9.5.21, 10.0.13, 10.4.19, 11.0.0, 11.5.2, 12.0.0, 12.2.1, 13.0.3
+```
+
+Dry-run emits zero-token step/run rows under `experiments/gate-v1/out/metrics.ndjson`, one run
+row per version walked, each labelled `dry-run — tokens remain 0; not a gate measurement`.
+Live runs exit **2** until Playwright live wiring lands ([#62](https://github.com/DevToolie/Paragent/issues/62)).
+
+Exit codes: **2** for a usage error or a live-run attempt; **1** if the selection walked zero
+versions — an empty NDJSON is a missing denominator, not a successful run.
+
+### `out/matrix-run.json` — the skip ledger
+
+Written every run beside the NDJSON. Records the selection, `versions_in_matrix`,
+`versions_walked`, and `versions_skipped[] {id, reason}`. Without it a later reader cannot tell
+"8 pins, 3 unavailable" from "5 pins", and the denominator shrinks silently.
 
 Report paths (after `gate:report`):
 
@@ -56,6 +91,12 @@ Empty NDJSON → scaffold with `status: no_data` and null values (never invented
 
 ## Open questions / what I could not verify
 
-- Wire `versions.json` to ADR-0003 / `scripts/testbed/matrix.json` pins.
+- ~~Wire `versions.json` to ADR-0003 / `scripts/testbed/matrix.json` pins.~~ **Done (#26)** —
+  `versions.json` is deleted and the harness reads the pins directly. What this does **not**
+  change: the run is still a dry run over a hand-written 2-step local-demo program, so walking
+  eight versions instead of one makes the row count honest, not the numbers meaningful.
+- The program under test is still `fixtures/compiled-program.json`, whose own
+  `testbed_version` reads `pending-b1@placeholder` because it was compiled against no testbed at
+  all. That is accurate and left alone; the runner overrides it per version.
 - Fresh-reasoning baseline cost measurement path — not wired; `cost_fresh` stays zeros until measured.
-- Live matrix against Grafana OSS — needs Docker + `--base-url` recording.
+- Live matrix against Grafana OSS — needs Docker + `--base-url` recording ([#62](https://github.com/DevToolie/Paragent/issues/62)).