Skip to content

fix(verifier): adapter-raise errored verify checkpoints retry on resume - #377

Merged
gadievron merged 2 commits into
masterfrom
fix/issue286-errored-cp-retry
Aug 28, 2026
Merged

fix(verifier): adapter-raise errored verify checkpoints retry on resume#377
gadievron merged 2 commits into
masterfrom
fix/issue286-errored-cp-retry

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

Summary

The verify resume classifier _cp_is_error returned True only when the checkpoint had no verification dict or when verification["correct_finding"] == "error"a value the normal verify error path never writes. That path (an adapter raise — the retryable class) writes the error string on the result dict (finding_verifier.py:800), which the checkpoint writer did not copy, and sets verification["incomplete"] = True. Every errored unit therefore looked like finished work on resume and was adopted, never retried (#286: the reporter's 223/223 adoption with 48 hard errors).

The fix is surgical (per adversarial review): the checkpoint writer — both sequential and parallel — now copies the adapter-raise error string into cp_data["error"], and the classifier retries only checkpoints carrying it (plus the legacy correct_finding == "error" and the missing-verification shapes). verification["incomplete"] alone is deliberately not a retry signal: five non-error fail-safe writers set it (deterministic outcomes — max iterations, truncated finish, no tool calls, degenerate finish, no-agree finish); retrying those on every resume is unbounded waste, and #212 established that refusals are deterministic — an incomplete-based classifier would retry them forever.

Test plan

5 hermetic tests: the adapter-error shape retries; deterministic incomplete does NOT; the completed shape, the missing shapes, and the legacy correct_finding shape unchanged. The classifier is extracted from the live source via inspect — no copy-paste predicate.

Verification evidence (commands + results)
check command result
new tests pytest tests/test_issue286_errored_checkpoint_retry.py -q 5 passed (offline)
hermeticity oracle env -i HOME=/tmp/fakehome-noconfig python -m pytest tests/test_issue286_errored_checkpoint_retry.py 5 passed
mutation smoke revert the error-check the core test fails (restored → green)
full suite pytest tests/ -q 3124 passed, 2 failed (pre-existing zig local-env, stash-replay verified), 32 skipped
lint ruff check . clean

Adversarial loop: 4-seat wave → the over-retry catch (my first fix classified ALL incomplete as errored — the wave identified five non-error incomplete=True writers and the deterministic-refusal infinite-retry risk; fixed with the surgical error-string discriminator copied into the checkpoint by the writer).

Fixes #286

…e IS the error signal

The resume classifier _cp_is_error returned True only when the checkpoint
had no verification dict or when verification['correct_finding'] ==
'error' — a value the normal verify error path never writes. That path
writes verification = {'incomplete': True} (the adapter-raise fail-safe,
finding_verifier.py:802) and puts the error string on the RESULT dict,
which the checkpoint writer does not copy. Every errored unit therefore
looked like finished work on resume and was adopted, never retried (#286:
the reporter's 223/223 adoption with 48 hard errors).

The classifier now also treats verification['incomplete'] as errored —
the exact signal the error path writes. The legacy
correct_finding == 'error' path (written by the Stage-1 consistency
path) is retained.

Counts (commands run at commit time):
- pytest tests/test_issue286_errored_checkpoint_retry.py -q → 4 passed
  (the classifier is extracted from the LIVE source via inspect — no
  copy-paste predicate; the incomplete shape, the completed shape, the
  missing-verification shapes, and the legacy correct_finding shape)
- hermeticity oracle: env -i HOME=/tmp/fakehome-noconfig → 4 passed
- mutation smoke: removing the incomplete branch → the core test fails
  (restored → green)
- pytest tests/ -q → 3123 passed, 2 failed (pre-existing zig local-env,
  stash-verified; CI green with them), 32 skipped
- ruff check . → clean

Fixes #286
…me — the error string IS the retry signal

The verify resume classifier _cp_is_error returned True only when the
checkpoint had no verification dict or when
verification['correct_finding'] == 'error' — a value the normal verify
error path never writes. That path (an adapter raise — the retryable
class) writes the error string on the RESULT dict
(finding_verifier.py:800), which the checkpoint writer did not copy, and
sets verification['incomplete'] = True. Every errored unit therefore
looked like finished work on resume and was adopted, never retried
(#286: the reporter's 223/223 adoption with 48 hard errors).

The fix is SURGICAL (per the wave's catch): the checkpoint writer — both
sequential and parallel — now copies the adapter-raise error string into
cp_data['error'], and the classifier retries ONLY checkpoints carrying
it (plus the legacy correct_finding == 'error' and the
missing-verification shapes). verification['incomplete'] alone is
deliberately NOT a retry signal: five non-error fail-safe writers set it
(deterministic outcomes — max iterations, truncated finish, no tool
calls, degenerate finish, no-agree finish); retrying those on every
resume is unbounded waste, and #212 established that refusals are
DETERMINISTIC — an incomplete-based classifier would retry them forever.

Counts (commands run at commit time):
- pytest tests/test_issue286_errored_checkpoint_retry.py -q → 5 passed
- hermeticity oracle: env -i HOME=/tmp/fakehome-noconfig → 5 passed
- mutation smoke: reverting the error-check → the core test fails
  (restored → green)
- pytest tests/ -q → 3124 passed, 2 failed (pre-existing zig local-env,
  stash-verified; CI green with them), 32 skipped
- ruff check . → clean

Fixes #286
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify checkpoints: errored units are restored as completed on resume, because _cp_is_error tests a value nothing writes on the normal path

1 participant