feat(gate): repeat runs per version, per-version variance, safe interruption

[OM152002]

Closes #66. Stacked on #91 (the live matrix runner it repeats) — base is track1/b4-live-matrix; retarget to main once #91 lands.

One run per version is not a sample. §9 specifies 3×/day for 14 days — ≥42 runs and ≥400 step-executions — and swapping the calendar for the version matrix does not change the statistics. Eight pins at one run each is 8 runs, and it cannot tell "this locator broke on v12" from "that run flaked".

--runs, default 3, and the arithmetic

--runs	Runs across 8 pins	Clears ≥42?
1	8	no
3 (default)	24	no
5	40	no — two short
6	48	yes

The issue suggests "8 versions × 5 gets there". It doesn't: 40 < 42. runsToClearSection9(8) returns 6, and there's a test pinning that off-by-one, because it is exactly the shape of thing that ships as "we cleared §9".

Default 3 is a deliberate compromise, not the floor: enough to see disagreement between repeats, cheap enough that people run it. 6 × 8 versions is 48 container boots. The floor is a property of the published measurement, not of the loop you run while developing — so the shortfall is reported, never enforced:

gate:matrix live — grafana-oss matrix, 2 version(s) × 3 run(s) = 6 run(s), …
  note: 6 runs / 12 step-executions is below the PRD §9 floor (>=42 / >=400).
        Use --runs 21 over 2 version(s) to clear it. The report records the shortfall either way.

and in report.json:

"sample": {"runs": 6, "step_executions": 12, "meets_floor": false,
           "shortfall": "below PRD §9 floor: 6/42 runs, 12/400 step-executions"}

Per-version variance

perVersionBreakdown() — a pooled ratio makes 3/3 and 2/3 the same number:

{"testbed_version": "11.0.0", "runs_attempted": 3, "runs_succeeded": 2,
 "step_validity_per_run": [1, 0.5, 1], "step_validity_spread": 0.5, "status": "computed"}

status: "no_data" rather than 0 when a run emitted no step rows — zero would be indistinguishable from every step having failed.

Building the interrupt path found a real bug

Playwright installs its own SIGINT handler and closes the browser on Ctrl-C. That fires during the in-flight run, so the next capturePageState threw and the matrix died with a stack trace instead of finishing the run it had been told to finish:

page.title: Target page, context or browser has been closed
    at capturePageState (src/runner/page-state.ts:24:28)
    at ReplayRunner.run (src/runner/replay.ts:148:21)

chromium.launch now passes handleSIGINT/SIGTERM/SIGHUP: false — the driver already closes the browser in its own finally. After the fix:

9.5.21 run 1/3: SUCCESS steps_valid=2/2 repairs=0 wall=0.4s
gate:matrix: interrupt received — finishing the current run, then tearing down.
             Press Ctrl-C again to abort immediately (may orphan a container).
9.5.21 run 2/3: SUCCESS steps_valid=2/2 repairs=0 wall=0.3s
9.5.21: SKIPPED (interrupted) — interrupted after 2 of 3 run(s); the completed runs
        are in the NDJSON and are real measurements

Verified after that interrupt: exit=0, 6/6 NDJSON lines parse, 2 run rows with 2 distinct run_ids, interrupted: true with runs_planned: 6 / runs_completed: 2, gate:report handled the partial data, and zero leftover containers or networks.

Rows are now appended after every run rather than written once at the end — MetricsEmitter.flush() overwrites, so the previous behaviour lost every row on interrupt.

Independence is real, not asserted

A fresh browser context and a fresh login per run, not per version. Reusing either would let run 1's cookies and cache decide run 2's outcome; the repeats would correlate and the spread would understate the variance it exists to measure.

State-mutating tasks are handled explicitly rather than magically. The gate task creates a dashboard, so run 2 against one container collides and fails for a reason that is not churn:

npm run gate:matrix -- --runs 3 --param 'dashboard_title=Paragent Gate {run}'

{run} in a --param value is substituted per run. Auto-suffixing every param would silently change values a recording captured while the assertion templates still compared against the recorded hole.

The measurement #66 asked for

Three live repeats of 9.5.21 against one unchanged container:

9.5.21 run 1/3: SUCCESS steps_valid=2/2 repairs=0 wall=0.5s
9.5.21 run 2/3: SUCCESS steps_valid=2/2 repairs=0 wall=0.3s
9.5.21 run 3/3: SUCCESS steps_valid=2/2 repairs=0 wall=0.3s

{"testbed_version": "9.5.21", "runs_attempted": 3, "runs_succeeded": 3,
 "step_validity_per_run": [1, 1, 1], "step_validity_spread": 0}

No harness flakiness observed. Stated with its caveat, in the PR and in gate/runner.md: this is a weak probe. The program was the 2-step example bundle (navigate + fill on a login page). The 12-step gate task with drawer and picker interactions is where flakiness would surface, and that arrives with #25.

No run is ever discarded

Including a failed one, and one cut short by an interrupt. A skip can now arrive with completed runs (interrupted partway, or login broke on repeat 2) and those runs are kept. Dropping an outlier is the single easiest way to manufacture a passing gate and would be undetectable in the report.

Skips stay distinct from failures: versions_skipped[] gains an interrupted stage, and skipped versions still never reach the NDJSON.

Not done

Nothing reseeds between runs. {run} substitution covers a task whose mutation is parameterisable; a task that mutates state it cannot parameterise away would need a container recreated per run, which is not implemented and is called out in the README.

Tests

11 new unit tests (27 in the file): the 3/3-vs-2/3 distinction, no_data vs 0, both §9 floors independently, the 8×5=40 off-by-one, and {run} substitution including that it returns a new object. Guard proven — forcing step_validity_spread to 0 fails separates 3/3 from 2/3.

npm run ci            # green — 136 unit, 1 integration, secret-scan clean, lint-docs clean (43 docs)
npm run test:canary   # 6 pass

🤖 Generated with Claude Code

[myselfsiddharth]

Code review

Read the PR description, the full diff (stacked on #91), and cross-checked the runtime behavior against src/metrics/aggregate.ts, src/runner/replay.ts, and the actual experiments/gate-v1/live-run.ts / run-matrix.ts on the PR branch.

Overview

Closes #66. Adds --runs <n> (default 3) to repeat the program per version with a fresh browser context and fresh login each time, reports per-version step-validity spread (perVersionBreakdown) so a pooled ratio can't hide "3/3 vs 2/3", adds an honestly-reported (not enforced) §9 sampling-floor check, and fixes a real bug found while building this — Playwright's own SIGINT handler was killing the browser mid-run, so chromium.launch now passes handleSIGINT/SIGTERM/SIGHUP: false and the driver owns graceful interrupt handling itself.

This PR fixes the data-loss issue I flagged in my #91 review. I verified the mechanism directly: MetricsEmitter.appendFlush() is now called via onRunComplete after every individual run, not once at the end via the overwriting flush(). So an uncaught exception mid-matrix now only loses the in-flight run — everything already completed is durably on disk. The interrupt handling built on top of it (finish current run → teardown → record SKIPPED (interrupted) with completed runs kept) is correct: shouldStop() is checked between runs, never mid-run, matching the stated design.

New correctness issue: two disagreeing implementations of the §9 floor

section9SampleFloor() in src/metrics/aggregate.ts computes step_executions from the actual deduped step rows in the NDJSON (dedupeLatestSteps(filterSteps(rows)).length) — the real count of steps attempted. But run-matrix.ts:480 computes the section9_floor.meets_floor written into matrix-run.json differently:

meets_floor: runs.length * program.steps.length >= 400

This assumes every completed run executed every step in the program. That's not always true: ReplayRunner.run() (replay.ts:297) sets steps_total = program.steps.length unconditionally, but stepResults — the actual emitted step rows — can be shorter, because the loop breaks after a hard failure (if (!SUCCESS_OUTCOMES.has(final.outcome)) break;, replay.ts:270). So the moment any run fails partway through (which is the entire scenario this harness exists to observe), matrix-run.json's meets_floor can read true while gate:report's report.json — built from section9SampleFloor() on the same NDJSON — reads false for the same run. Two artifacts from one measurement disagreeing about whether the §9 floor was cleared is exactly the kind of thing this PR is otherwise very careful to prevent.

Fix: have run-matrix.ts call section9SampleFloor() on the actual rows (e.g. readMetricNdjson(ndjsonPath) after the final persist()) instead of re-deriving the same concept by hand.

Residual / minor items

• browser.newContext() / context.newPage() are still unguarded (live-run.ts:359-361), same gap as feat(gate): drive a real browser per matrix version, remove the exit-2 guard #91, now inside the per-run loop so it's exercised --runs times per version instead of once. The blast radius is smaller now (only the in-flight run's NDJSON rows are at risk, not prior ones), but matrix-run.json itself (skip ledger, section9_floor, interrupted flag) is still written exactly once at the very end of main(), so an uncaught throw here still means that summary file — including why an earlier version was skipped — never gets written at all. Worth wrapping in the same try/catch pattern used for chromium.launch two lines above, attributed to "browser".
• Interrupt responsiveness during bring-up. shouldStop() is only checked inside the per-run loop, after compose-up → readiness → seed → fingerprint → browser-launch all complete for that version. A Ctrl-C pressed while a fresh version is mid-pull/mid-boot has to wait out that entire sequence before "finishing the current run" even applies (there is no current run yet). Not a correctness bug — no run is ever torn down mid-flight — but the console message doesn't quite describe what the user is waiting through in that case.
• Carried over from feat(gate): drive a real browser per matrix version, remove the exit-2 guard #91, still unaddressed: versions pre-marked "unavailable" in scripts/testbed/matrix.json are still force-labeled stage: "compose-up" in the skip ledger even though they were never attempted. Cosmetic, but the ledger can't distinguish "we tried and compose failed" from "config said skip it."
• Cosmetic: the shared onSignal handler calls process.exit(130) on a second signal for both SIGINT and SIGTERM; 130 is the SIGINT convention (128+2), not SIGTERM's (128+15). Harmless.

Code quality, tests, security

Same strong fit with repo conventions as #91 — rationale-heavy comments, honest docs updates, no contract changes smuggled in. substituteRunIndex is a clean, well-scoped answer to state-mutation-between-repeats, explicit rather than magic, and pins its own edge case (leaves non-{run} params untouched, returns a new object). 11 new unit tests (27 in the file total) directly pin the off-by-one in runsToClearSection9(8) === 6 (the PR explicitly calls out that the issue's own suggestion of --runs 5 doesn't clear the floor — good catch, well tested), the 3/3-vs-2/3 spread distinction, and no_data vs 0 semantics. No security concerns — no new credential handling, no injection surface in --param/{run} substitution.

Summary

Meaningfully improves on #91: the crash/data-loss issue is genuinely fixed via per-run incremental persistence, and the SIGINT/Playwright interaction bug is a legitimate find caught by actually building the interrupt path. The one thing worth fixing before merge is the meets_floor divergence between matrix-run.json and report.json — reuse section9SampleFloor() in both places rather than hand-rolling it twice. Everything else here is minor.

---

🤖 Reviewed with Claude Code

[myselfsiddharth]

Approving. Merge conflicts against `main` (from #91 + #89/ADR-0006 landing) resolved by merge commit — every conflicted file (`docs/gate/runner.md`, `experiments/gate-v1/README.md`, `experiments/gate-v1/live-run.ts`, `experiments/gate-v1/run-matrix.ts`, `tests/unit/gate-matrix.test.ts`) verified byte-identical to this PR's original intended content after resolution — nothing from #92 was lost or altered.

`npm run ci` and `npm run test:canary` both green locally (152 + 6 tests), and all required GitHub checks pass: `lint-typecheck-test-secrets`, `privacy-canary`, `testbed-smoke`, CodeQL, PR title.

Per my earlier review, one known non-blocking issue (`matrix-run.json`'s `section9_floor.meets_floor` can diverge from `report.json`'s once a run fails partway through a step sequence) is tracked as a follow-up in #95 rather than blocking this merge.

---

🤖 Reviewed with Claude Code