Skip to content

test(runner): integration tests for repair invariants - #105

Merged
myselfsiddharth merged 2 commits into
mainfrom
track1/b4-repair-invariant-tests
Aug 3, 2026
Merged

test(runner): integration tests for repair invariants#105
myselfsiddharth merged 2 commits into
mainfrom
track1/b4-repair-invariant-tests

Conversation

@myselfsiddharth

Copy link
Copy Markdown
Contributor

Closes #65.

Summary

tests/integration/repair-invariants.test.ts drives ReplayRunner end to end against a real headless Chromium page and fully scripted RepairModelClient fakes — no network, no API key, deterministic. tests/unit/runner.test.ts only ever exercised StubRepairModelClient (always corrected_action: null), which structurally cannot test what happens when a proposal is hostile or when the budget must be exhausted across several real, failing proposals.

All seven required cases, 9 tests:

  1. Assertion tamper rejected — both paths, as separate tests: a client that reassigns ctx.assertion to a new object, and a client that mutates the live (non-frozen) step.assertion in place.
  2. Strength downgrade rejected — the specific strong→weak attack, as its own test.
  3. Budget honoured — a client that always proposes a real (but still-failing) action hits exactly maxRepairsPerRun attempts, ends REPAIR_EXHAUSTED, repair_count === 2.
  4. Budget is per run, not per step — a two-step program where step 0 consumes both repair slots (succeeding on its 2nd attempt); step 1 gets REPAIR_EXHAUSTED having made zero calls to the client, and repair_count never exceeds the cap.
  5. success_with_le_2_repairs accuracytrue for a run needing exactly 2 repairs and succeeding; false for a run needing 3 (with the cap explicitly raised to 3) even though it still succeeds — pins that this field is hard-coded to a threshold of 2, independent of whatever maxRepairsPerRun was configured to.
  6. Cost accounted on failure — a client that burns tokens and then proposes null still shows those tokens in cost_repair.
  7. A thrown client error does not silently pass the step — see the runner fix below.

Runner hole found and fixed here

Case 7 exposed a real gap: ReplayRunner catches a failing browser action and a failing assertion evaluation into typed StepOutcomes (src/runner/actions.ts, src/runner/assertions.ts), but had no equivalent catch around this.repairClient.propose(ctx) — the one other external call in the loop. A repair client that threw (a model API erroring, timing out, whatever) propagated straight out of run(), rejecting the whole run instead of failing one step. Fixed in src/runner/replay.ts: the propose() call is now wrapped in try/catch, and a throw is recorded as REPAIR_EXHAUSTED with the error message preserved — the same shape already used for a null proposal.

Guard verification (required by the issue)

Confirmed each assertAssertionUnchanged call is load-bearing, not vacuous, by disabling it and watching the corresponding attack succeed, then restoring it:

  • Disabled only the ctx.assertion check (line 197). Result: the "reassigns ctx.assertion" test failed — the run silently returned task_success: true / REPAIRED_PASS instead of throwing. The other two tamper tests still passed, confirming they're caught by a different, still-active check.
  • Disabled both step.assertion checks together (the one immediately after propose() and the one after the retry — line 196 and line 241; either alone still gets caught by the other, since a mutation made during propose() persists through the retry). Result: both the "mutate step.assertion" test and the "strength downgrade" test failed — the run reported REPAIRED_PASS with "assertion_strength": "weak" sitting right there in the result, completely undetected.
  • Restored all three checks; full suite green again (verified below).

This is more precise than "comment out one line" because two of the three checks have partial redundancy for some attack shapes (a mutation made during propose() is still caught by the later check even with the earlier one disabled) — so I disabled combinations until each attack actually got through, to avoid claiming a check is load-bearing when it's actually just redundant with a neighbor.

Test plan

  • All 7 cases implemented and passing (9 tests, tests/integration/repair-invariants.test.ts)
  • Each guard verified to fail when its protection is removed (described above); guards restored before committing
  • No network, no API key — all clients are local classes; only a local headless Chromium page is involved
  • Runner hole found (case 7) fixed in this PR with the test that caught it
  • npm run ci green, npm run test:canary green
  • docs/gate/runner.md invariants section updated to reference these tests by path (invariants 1, 3, and a new invariant 6 for the throw-containment fix)

Checklist (from the issue)

  • All seven cases implemented and passing
  • Each guard verified to fail when its protection is removed; described in the PR body
  • No network, no API key, no flakiness
  • Any runner hole found is fixed in this PR with the test that caught it
  • npm run ci, npm run test:canary green
  • docs/gate/runner.md invariants section references these tests by path
  • Branch track1/b4-repair-invariant-tests

🤖 Generated with Claude Code

…lient-throw hole (#65)

tests/integration/repair-invariants.test.ts drives ReplayRunner end to end
against a real headless page and fully scripted RepairModelClient fakes —
no network, no API key. tests/unit/runner.test.ts only ever exercised
StubRepairModelClient (always corrected_action: null), which can't test a
hostile or budget-exhausting proposal.

Covers all seven required cases: assertion tamper via ctx.assertion
reassignment, tamper via in-place step.assertion mutation, the specific
strong->weak downgrade the contract forbids, the repair budget honoured at
exactly maxRepairsPerRun, the budget shared across steps rather than reset
per step, success_with_le_2_repairs computed correctly (true at exactly 2,
false at 3 even under a raised cap), and repair tokens billed even when a
proposal is rejected.

Verified each assertion-immutability guard is load-bearing, not vacuous, by
disabling it and confirming the attack goes undetected before restoring it
(described in the PR body) — disabling the ctx.assertion check alone lets a
reassignment through as a silent REPAIRED_PASS; disabling both
step.assertion checks together lets a strong->weak downgrade through the
same way, with the corrupted strength visible in the result.

Runner hole found and fixed here, with the test that caught it: a
repair-model client that throws was the one external call in the loop
without the same catch-into-StepOutcome treatment already given to browser
actions and assertion evaluation, so it aborted run() entirely instead of
failing one step. Now caught and recorded as REPAIR_EXHAUSTED.

docs/gate/runner.md invariants section updated to reference these tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@myselfsiddharth
myselfsiddharth requested a review from a team as a code owner July 31, 2026 02:29
@github-actions
github-actions Bot requested a review from OM152002 July 31, 2026 02:29
@github-actions github-actions Bot added size/L <= 600 changed lines documentation Improvements or additions to documentation gate PRD section 9 gate measurement area: runner Touches runner and removed size/L <= 600 changed lines labels Jul 31, 2026
@github-actions github-actions Bot added the size/L <= 600 changed lines label Aug 3, 2026
@myselfsiddharth

Copy link
Copy Markdown
Contributor Author

@copilot Review this PR after reading the docs.

Copilot AI commented Aug 3, 2026

Copy link
Copy Markdown

@copilot Review this PR after reading the docs.

Reviewed after reading docs/gate/runner.md and the PR changes. No additional code changes are needed from this comment; invariants coverage and runner behavior look consistent with the documented requirements.

@myselfsiddharth
myselfsiddharth merged commit c08469c into main Aug 3, 2026
13 checks passed
@myselfsiddharth
myselfsiddharth deleted the track1/b4-repair-invariant-tests branch August 3, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: runner Touches runner documentation Improvements or additions to documentation gate PRD section 9 gate measurement size/L <= 600 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration tests for repair invariants: assertion immutability and budget caps

2 participants