fix(verifier): adapter-raise errored verify checkpoints retry on resume - #377
Merged
Conversation
…e IS the error signal
The resume classifier _cp_is_error returned True only when the checkpoint
had no verification dict or when verification['correct_finding'] ==
'error' — a value the normal verify error path never writes. That path
writes verification = {'incomplete': True} (the adapter-raise fail-safe,
finding_verifier.py:802) and puts the error string on the RESULT dict,
which the checkpoint writer does not copy. Every errored unit therefore
looked like finished work on resume and was adopted, never retried (#286:
the reporter's 223/223 adoption with 48 hard errors).
The classifier now also treats verification['incomplete'] as errored —
the exact signal the error path writes. The legacy
correct_finding == 'error' path (written by the Stage-1 consistency
path) is retained.
Counts (commands run at commit time):
- pytest tests/test_issue286_errored_checkpoint_retry.py -q → 4 passed
(the classifier is extracted from the LIVE source via inspect — no
copy-paste predicate; the incomplete shape, the completed shape, the
missing-verification shapes, and the legacy correct_finding shape)
- hermeticity oracle: env -i HOME=/tmp/fakehome-noconfig → 4 passed
- mutation smoke: removing the incomplete branch → the core test fails
(restored → green)
- pytest tests/ -q → 3123 passed, 2 failed (pre-existing zig local-env,
stash-verified; CI green with them), 32 skipped
- ruff check . → clean
Fixes #286
…me — the error string IS the retry signal The verify resume classifier _cp_is_error returned True only when the checkpoint had no verification dict or when verification['correct_finding'] == 'error' — a value the normal verify error path never writes. That path (an adapter raise — the retryable class) writes the error string on the RESULT dict (finding_verifier.py:800), which the checkpoint writer did not copy, and sets verification['incomplete'] = True. Every errored unit therefore looked like finished work on resume and was adopted, never retried (#286: the reporter's 223/223 adoption with 48 hard errors). The fix is SURGICAL (per the wave's catch): the checkpoint writer — both sequential and parallel — now copies the adapter-raise error string into cp_data['error'], and the classifier retries ONLY checkpoints carrying it (plus the legacy correct_finding == 'error' and the missing-verification shapes). verification['incomplete'] alone is deliberately NOT a retry signal: five non-error fail-safe writers set it (deterministic outcomes — max iterations, truncated finish, no tool calls, degenerate finish, no-agree finish); retrying those on every resume is unbounded waste, and #212 established that refusals are DETERMINISTIC — an incomplete-based classifier would retry them forever. Counts (commands run at commit time): - pytest tests/test_issue286_errored_checkpoint_retry.py -q → 5 passed - hermeticity oracle: env -i HOME=/tmp/fakehome-noconfig → 5 passed - mutation smoke: reverting the error-check → the core test fails (restored → green) - pytest tests/ -q → 3124 passed, 2 failed (pre-existing zig local-env, stash-verified; CI green with them), 32 skipped - ruff check . → clean Fixes #286
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
August 28, 2026 11:56
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The verify resume classifier
_cp_is_errorreturned True only when the checkpoint had noverificationdict or whenverification["correct_finding"] == "error"— a value the normal verify error path never writes. That path (an adapter raise — the retryable class) writes the error string on the result dict (finding_verifier.py:800), which the checkpoint writer did not copy, and setsverification["incomplete"] = True. Every errored unit therefore looked like finished work on resume and was adopted, never retried (#286: the reporter's 223/223 adoption with 48 hard errors).The fix is surgical (per adversarial review): the checkpoint writer — both sequential and parallel — now copies the adapter-raise error string into
cp_data["error"], and the classifier retries only checkpoints carrying it (plus the legacycorrect_finding == "error"and the missing-verification shapes).verification["incomplete"]alone is deliberately not a retry signal: five non-error fail-safe writers set it (deterministic outcomes — max iterations, truncated finish, no tool calls, degenerate finish, no-agree finish); retrying those on every resume is unbounded waste, and #212 established that refusals are deterministic — an incomplete-based classifier would retry them forever.Test plan
5 hermetic tests: the adapter-error shape retries; deterministic incomplete does NOT; the completed shape, the missing shapes, and the legacy
correct_findingshape unchanged. The classifier is extracted from the live source viainspect— no copy-paste predicate.Verification evidence (commands + results)
pytest tests/test_issue286_errored_checkpoint_retry.py -qenv -i HOME=/tmp/fakehome-noconfig python -m pytest tests/test_issue286_errored_checkpoint_retry.pypytest tests/ -qruff check .Adversarial loop: 4-seat wave → the over-retry catch (my first fix classified ALL incomplete as errored — the wave identified five non-error
incomplete=Truewriters and the deterministic-refusal infinite-retry risk; fixed with the surgicalerror-string discriminator copied into the checkpoint by the writer).Fixes #286