test(runner): integration tests for repair invariants - #105
Merged
Conversation
…lient-throw hole (#65) tests/integration/repair-invariants.test.ts drives ReplayRunner end to end against a real headless page and fully scripted RepairModelClient fakes — no network, no API key. tests/unit/runner.test.ts only ever exercised StubRepairModelClient (always corrected_action: null), which can't test a hostile or budget-exhausting proposal. Covers all seven required cases: assertion tamper via ctx.assertion reassignment, tamper via in-place step.assertion mutation, the specific strong->weak downgrade the contract forbids, the repair budget honoured at exactly maxRepairsPerRun, the budget shared across steps rather than reset per step, success_with_le_2_repairs computed correctly (true at exactly 2, false at 3 even under a raised cap), and repair tokens billed even when a proposal is rejected. Verified each assertion-immutability guard is load-bearing, not vacuous, by disabling it and confirming the attack goes undetected before restoring it (described in the PR body) — disabling the ctx.assertion check alone lets a reassignment through as a silent REPAIRED_PASS; disabling both step.assertion checks together lets a strong->weak downgrade through the same way, with the corrupted strength visible in the result. Runner hole found and fixed here, with the test that caught it: a repair-model client that throws was the one external call in the loop without the same catch-into-StepOutcome treatment already given to browser actions and assertion evaluation, so it aborted run() entirely instead of failing one step. Now caught and recorded as REPAIR_EXHAUSTED. docs/gate/runner.md invariants section updated to reference these tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
@copilot Review this PR after reading the docs. |
Reviewed after reading |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #65.
Summary
tests/integration/repair-invariants.test.tsdrivesReplayRunnerend to end against a real headless Chromium page and fully scriptedRepairModelClientfakes — no network, no API key, deterministic.tests/unit/runner.test.tsonly ever exercisedStubRepairModelClient(alwayscorrected_action: null), which structurally cannot test what happens when a proposal is hostile or when the budget must be exhausted across several real, failing proposals.All seven required cases, 9 tests:
ctx.assertionto a new object, and a client that mutates the live (non-frozen)step.assertionin place.maxRepairsPerRunattempts, endsREPAIR_EXHAUSTED,repair_count === 2.REPAIR_EXHAUSTEDhaving made zero calls to the client, andrepair_countnever exceeds the cap.success_with_le_2_repairsaccuracy —truefor a run needing exactly 2 repairs and succeeding;falsefor a run needing 3 (with the cap explicitly raised to 3) even though it still succeeds — pins that this field is hard-coded to a threshold of 2, independent of whatevermaxRepairsPerRunwas configured to.nullstill shows those tokens incost_repair.Runner hole found and fixed here
Case 7 exposed a real gap:
ReplayRunnercatches a failing browser action and a failing assertion evaluation into typedStepOutcomes (src/runner/actions.ts,src/runner/assertions.ts), but had no equivalent catch aroundthis.repairClient.propose(ctx)— the one other external call in the loop. A repair client that threw (a model API erroring, timing out, whatever) propagated straight out ofrun(), rejecting the whole run instead of failing one step. Fixed insrc/runner/replay.ts: thepropose()call is now wrapped in try/catch, and a throw is recorded asREPAIR_EXHAUSTEDwith the error message preserved — the same shape already used for anullproposal.Guard verification (required by the issue)
Confirmed each
assertAssertionUnchangedcall is load-bearing, not vacuous, by disabling it and watching the corresponding attack succeed, then restoring it:ctx.assertioncheck (line 197). Result: the "reassignsctx.assertion" test failed — the run silently returnedtask_success: true/REPAIRED_PASSinstead of throwing. The other two tamper tests still passed, confirming they're caught by a different, still-active check.step.assertionchecks together (the one immediately afterpropose()and the one after the retry — line 196 and line 241; either alone still gets caught by the other, since a mutation made duringpropose()persists through the retry). Result: both the "mutatestep.assertion" test and the "strength downgrade" test failed — the run reportedREPAIRED_PASSwith"assertion_strength": "weak"sitting right there in the result, completely undetected.This is more precise than "comment out one line" because two of the three checks have partial redundancy for some attack shapes (a mutation made during
propose()is still caught by the later check even with the earlier one disabled) — so I disabled combinations until each attack actually got through, to avoid claiming a check is load-bearing when it's actually just redundant with a neighbor.Test plan
tests/integration/repair-invariants.test.ts)npm run cigreen,npm run test:canarygreendocs/gate/runner.mdinvariants section updated to reference these tests by path (invariants 1, 3, and a new invariant 6 for the throw-containment fix)Checklist (from the issue)
npm run ci,npm run test:canarygreendocs/gate/runner.mdinvariants section references these tests by pathtrack1/b4-repair-invariant-tests🤖 Generated with Claude Code