Verify checkpoints: errored units are restored as completed on resume, because `_cp_is_error` tests a value nothing writes on the normal path

[gadievron]

Evidence provenance (added 2026-08-22). The run-derived figures in this issue come from a
private scan that a reader cannot reproduce (raptor-e2e-20260818, a 9.85-hour run against
a Python target). They are reported for completeness, not as independently checkable evidence.
Every code claim below is pinned to public source at b5019628 and is checkable; where a
conclusion rests on the private numbers rather than on the code, treat it as my report of what
that run showed. This label was missing when the issue was filed.

Summary

FindingVerifier's resume path decides whether a checkpointed unit needs re-verification with
_cp_is_error, which returns True only when the checkpoint has no verification dict, or when
verification["correct_finding"] == "error". The verify error path writes neither: it writes
verification = {"incomplete": True} and puts the error string on the result dict, which the
checkpoint writer does not copy. correct_finding == "error" has no writer on the normal verify
path. Every errored unit therefore looks like finished work on resume and is never retried.

⚠️ Corrected 2026-08-22. This paragraph originally read "no writer anywhere in production
code". That is false — finding_verifier.py:882/:921 assigns correct_finding from an
unvalidated LLM field on the Stage-1 consistency path. The value is unwritten on the path that
produced these records, not unwritable. Full correction at the end of this report; the measured
223/223 adoption result is unaffected.

Evidence

libs/openant-core/utilities/finding_verifier.py:603-630 (excerpt — 6 interior lines elided) (HEAD b501962):

        def _cp_is_error(cp_data):
            """A verify checkpoint is errored if verification is missing/empty
            or correct_finding == 'error'."""
            if not cp_data:
                return True
            v = cp_data.get("verification", {})
            if not v:
                return True
            return v.get("correct_finding") == "error"

        # Separate already-done (successful) from to-do (new + errored)
        results_to_verify = []
        _restored_ok = 0
        for r in results:
            key = r.get("unit_id") or r.get("route_key", "unknown")
            cp_data = checkpointed.get(key)
            if cp_data and not _cp_is_error(cp_data):
                # Restore verification data from checkpoint
                ...
                _restored_ok += 1
            else:
                # Either no checkpoint, or an errored one — re-verify
                results_to_verify.append(r)

The error path, finding_verifier.py:761-768:

            err_msg = f"{type(e).__name__}: {e}"
            result["error"] = err_msg
            # Surface a minimal verification dict marked incomplete so any
            # consumer that branches on ``verification.incomplete`` also treats
            # it as needs-review rather than a clean verdict.
            result.setdefault("verification", {})
            result["verification"]["incomplete"] = True
            result["verification_note"] = f"Verification errored: {err_msg}"

The checkpoint writers, finding_verifier.py:788-792 (sequential) and :819-823 (parallel), copy
three keys and not error:

                    cp_data = {
                        "verification": result.get("verification", {}),
                        "finding": result.get("finding", ""),
                        "verification_note": result.get("verification_note", ""),
                    }

So on an errored unit the checkpoint holds verification = {"incomplete": True} — truthy, and
without a correct_finding key. Both guards in _cp_is_error pass.

correct_finding == "error" has readers, and no writer on the normal path

⚠️ Corrected — it IS writable via the consistency path (finding_verifier.py:882/:921). See the correction at the end of this report; the survey below covers the finish/normalise paths only.

grep -rn 'correct_finding' --include='*.py' libs/openant-core --exclude-dir=tests | grep -i error

core/checkpoint.py:118          # Verify: verification empty or correct_finding == "error"
core/checkpoint.py:121              if not v or v.get("correct_finding") == "error":
core/checkpoint.py:366          #   - verify: verification is empty or verification.correct_finding == "error"
core/checkpoint.py:388          # Verify-style: verification empty or correct_finding == "error"
core/checkpoint.py:391              if not v or v.get("correct_finding") == "error":
utilities/finding_verifier.py:605       or correct_finding == 'error'."""
utilities/finding_verifier.py:611       return v.get("correct_finding") == "error"

Three readers, zero assignments.

Key-shape histogram over the live run's checkpoints

RUN=~/.openant/projects/gadievron/raptor-e2e-20260818/scans/7dbf9c691d7/python
python3 - <<'EOF'
import json, glob, os, collections
D = os.path.expanduser("$RUN/verify_checkpoints")
hist = collections.Counter(); n = 0
for f in glob.glob(D + "/*.json"):
    if os.path.basename(f) in ("_fingerprint.json", "_summary.json"):
        continue
    n += 1
    v = json.load(open(f)).get("verification", {})
    hist[tuple(sorted(v.keys()))] += 1
print("real checkpoints:", n)
for k, c in hist.most_common():
    print(c, k)
EOF

real checkpoints: 223
134 ('agree', 'correct_finding', 'explanation', 'incomplete', 'iterations', 'total_tokens')
 48 ('incomplete',)
 41 ('agree', 'correct_finding', 'explanation', 'exploit_path', 'incomplete',
     'iterations', 'security_weakness', 'total_tokens')

The 48 errored records carry only incomplete. One of them in full:

{
  "verification": { "incomplete": true },
  "finding": "vulnerable",
  "verification_note": "Verification errored: LLMResponseError: AnthropicAdapter returned no usable content (empty completion); the request may have been filtered or the response was malformed",
  "usage": { "input_tokens": 0, "output_tokens": 0, "cost_usd": 0.0 },
  "id": "core/audit/condition_smt.py:disprove_integer_overflow"
}

On resume, all 223 checkpoints satisfy cp_data and not _cp_is_error(cp_data), so all 223 are
restored as completed and 0 are re-verified. The else: branch commented "Either no checkpoint, or
an errored one — re-verify" is unreachable for errored units, and the operator-facing message at
:637-640 (errored_retries = len(checkpointed) - _restored_ok) computes 0.

The same pattern is repeated in the shared checkpoint helper

core/checkpoint.py:99-122 (load_ids(skip_errors=True)) and :388-391 use the identical verify
test, so they also treat errored verify checkpoints as completed.

Why it matters

The whole point of skip_errors / _cp_is_error is that a resume retries transient failures.
Because the test targets a value nothing writes on the path that produced these records, a resumed verify run adopts the previous run's
hard errors as finished verifications instead of retrying them — the failure is not merely
un-repaired, it is promoted to a result. On this run that is 48 of 223 candidates (21.5%), each of
which keeps its Stage-1 finding: "vulnerable" and is then counted as a confirmed vulnerability by
the metrics recount (filed separately).

It also means that any change intended to reduce verify errors cannot be validated by resuming an
affected scan: the errored units will not be re-attempted at all.

Suggested fix

1. Make the checkpoint carry the error. Add "error": result.get("error", "") (or
   verification["error"] = err_msg) to the cp_data dicts at finding_verifier.py:788-792 and
   :819-823.
2. Change _cp_is_error to test what the writer actually produces:
   return bool(cp_data.get("error")) or v.get("correct_finding") == "error". Keep the
   correct_finding clause for backward compatibility with existing checkpoint files.
3. Decide explicitly what to do with verification == {"incomplete": True} and nothing else. A unit
   whose verification never completed is arguably also a retry candidate; at minimum the decision
   should be a named, tested policy rather than a side effect of key absence.
4. Apply the same change to core/checkpoint.py:388-391 (status) — the live site. Its value
   is consumed by the Go CLI at apps/openant-cli/internal/checkpoint/checkpoint.go:112.
   :99-122 (load_ids) is DEAD CODE — zero callers repo-wide (/usr/bin/grep -rn load_ids
   returns only the definition; the two tests/parsers/c/ hits are test_overload_ids…, a
   substring false positive). Fixing it changes no behaviour; deleting it is the better move.
   (Corrected 2026-08-22 — this item previously directed a maintainer at both sites without
   saying one of them is dead. The fact was noted in a comment on this issue but had never
   reached the body, which is what a triaging maintainer actually reads.)
5. Add a regression test that writes an errored checkpoint through the real writer, then asserts
   _cp_is_error returns True for it.

What I am not claiming

• I am not claiming a retry would succeed. Whether these 48 units verify on a second attempt is
  unmeasured here; the defect is that no second attempt happens.
• I am not attributing the 48 errors to any particular cause in this issue. The error string is
  quoted verbatim above and is discussed separately.
• The errored checkpoint records carry a usage sub-object whose fields are zero. They do not
  carry a total_tokens key at all — the successful shapes do.
• The 41/134/48 split is from one Python run against a direct Anthropic binding. I have not checked
  whether other phases' checkpoint shapes differ.

---

Correction 2026-08-21 — the "never written" framing is WRONG. The measurement stands; the mechanism claim does not.

This report says _cp_is_error tests "a value the verifier never writes", and a comment of mine
strengthened that to "a value the finish tool's schema cannot emit". Both are too strong.

There is a writer. utilities/finding_verifier.py:882 and :921:

                    new_verdict = finding_update.get("should_be")
...
                                result["verification"]["correct_finding"] = new_verdict

should_be comes from the Stage-1 consistency resolution, an LLM reply parsed out of a schema-less
simple_text call (:1029-1031). /usr/bin/grep -n "should_be" over the file returns exactly one
site — :882 — and there is no enum, allow-list, or membership check on it anywhere. So
correct_finding can be assigned any string the model returns, including "error".

Why I missed it, stated plainly: I searched for the literal assignment
(correct_finding.*=.*"error") and found only readers. Line 921 assigns a variable, so a literal
search cannot see it. That is a text-match standing in for a semantic question, and it is the same
error class as several other corrections in this corpus.

My enum comment is also narrower than I wrote it. The finish tool's JSON Schema
(finding_verifier.py:158-161) does constrain correct_finding on the finish path. It does not
constrain the consistency path at :921, which bypasses the finish tool entirely.

What survives, and what does not

• STANDS — the measurement. All 223 verify checkpoints in the reference run carry
  correct_finding != "error", and all 223 satisfy cp_data and not _cp_is_error(cp_data), so all
  223 are adopted on resume. That is data, not inference.
• STANDS — the practical effect. Errored verify units are restored as completed, because nothing
  on the normal path writes the value the test looks for.
• FALLS — "never writes" / "can never fire" / "structural". The predicate is unwritten in
  practice, not unwritable. It is reachable through an unvalidated LLM-supplied field.
• CHANGES the fix. Making _cp_is_error correct is not only about giving it a value that gets
  written; the should_be path should also be validated, or an LLM reply could set
  correct_finding to an arbitrary string — the same shape as the synthesised-verdict problem in
  JSON corrector synthesises a verdict from free text and reports it as a successful correction, producing rows no bucket counts #316.

#287 depends on this premise and inherits the correction.

[gadievron]

⚠️ RETRACTED 2026-08-22 — the central claim of this comment is FALSE.
This comment argued the value is "unwritable by schema" and that "nothing can" write it.
There IS a writer: utilities/finding_verifier.py:1094
(correct_finding = finish_result.get("correct_finding", original_finding)), with no allow-list
enforcement — the enum declared at :158-161 is advisory and the repo has no jsonschema
dependency. The value is unwritten in practice, not unwritable, and the word "structural"
below is withdrawn.
What actually keeps the clause unexercised for checkpoint reads is narrower and mechanical:
checkpoint saves are synchronous (core/checkpoint.py:129-140 → utilities/file_io.py:274-293,
which json.dumps, fsyncs and os.replaces), so a later in-memory mutation cannot reach a file
already written. That is the correct grounding — not call-site ordering, which an earlier
correction of mine wrongly relied on.
The original text is left below unaltered so the record of what was claimed stays intact.

---

Two additions from a verification pass: the unwritable value is unwritable by schema, not merely
by absence of a writer — and the defect has a live consumer in the Go CLI that this report did not
trace.

1. correct_finding == "error" is excluded by the tool contract

The report established that nothing writes the value. The stronger fact is that nothing can: the
finish tool's own JSON Schema constrains it.

libs/openant-core/utilities/finding_verifier.py:158-161:

"correct_finding": {
    "type": "string",
    "enum": ["safe", "protected", "bypassable", "vulnerable", "inconclusive"],

The same five values appear in the prose schema at :83-87. "error" is not among them, so a
model-emitted verdict cannot take that value. Enumerating writers repo-wide agrees — every occurrence
of the literal is a reader or a comment (core/checkpoint.py:118, :121, :366, :388, :391;
utilities/finding_verifier.py:605, :611), zero writers.

2. The status() copy has a consumer, and it gates the resume prompt

The report notes the pattern recurs in core/checkpoint.py. The two sites are not equivalent:

• StepCheckpoint.load_ids (:99-127) is dead — zero callers repo-wide (definition only). Its
  copy of the test is inert. (It also handles all four phase formats, so it is not "half-fixed".)
• StepCheckpoint.status (:388-391) is live, and is the one with a user-visible effect.

Executed against this run's real verify_checkpoints (223 unit files):

StepCheckpoint.status(".../verify_checkpoints")
# -> {"step": "verify", "completed": 223, "errors": 0,
#     "total_units": 223, "phase": "done"}   error_breakdown: {}

Ground truth from the same files: 182 of those 223 units carry verification.incomplete == True
— i.e. Stage 2 never reached a verdict on them. They are unadjudicated, not errored; this run's 48
hard failures are a different quantity living in summary["error_count"].

status() has no notion of "incomplete" at all. It tests only error shapes, and its verify shape is
the unwritable one above. So both genuine verify errors and incompletes report as errors: 0.

That value has a consumer. openant/cli.py:796 (cmd_checkpoint_status) exists specifically to serve
it — its docstring (:781-786) reads "Internal subcommand — not user-facing. Called by the Go CLI
to get accurate completed/errored counts by reading actual checkpoint files." The Go side unmarshals
Phase and Errors into an anonymous struct at
apps/openant-cli/internal/checkpoint/checkpoint.go:92-100, and then:

if status.Phase == "done" && status.Errors == 0 {
    return nil
}

(checkpoint.go:112) — the resume prompt is suppressed.

Scope of what was run: the Python above was executed against the real checkpoints. The Go path is
read from source, not executed — no Go binary was run for this comment.

3. Why this raises the priority

Running the verify adopt predicate (finding_verifier.py:603-611) verbatim over the same 223 files:

verification EMPTY (the `not v` arm) :   0
verification.incomplete == True      : 182
  ...of those, _cp_is_error() False  : 182   -> would adopt on resume

So a resumed verify run would adopt all 182 unadjudicated units, and the CLI would not prompt,
because the count it consults reports zero errors on a phase marked done. This needs no model swap,
no new scan, and no spend to reproduce — the artifacts already exist.

Nothing above changes the report's diagnosis; it establishes that the defect is structural rather
than incidental, and that it reaches the operator-facing surface.

[gadievron]

Related: #311 reports the same substrate in the dynamic-test phase — a loop-accumulated _summary.json treated as authoritative where StepCheckpoint.status(), recomputing from the unit files, disagrees.

The mechanisms are different (there, a continue skips the counter increments; here, _cp_is_error tests a value the finish tool's schema cannot emit), and neither fix would close the other. Recording the link because the shared substrate is what makes both hard to see from the artifacts alone: in each case one source says the phase is clean and a recomputation says it is not.

While checking that, I also confirmed the enum constraint behind this issue is stronger than the report states. correct_finding is bounded by the finish tool's JSON Schema at libs/openant-core/utilities/finding_verifier.py:158-161:

                "correct_finding": {
                    "type": "string",
                    "enum": ["safe", "protected", "bypassable", "vulnerable", "inconclusive"],

So "error" is not merely unwritten — it is not an emittable value. The test can never fire.

[gadievron]

Correcting my own comment above — the enum argument is narrower than I stated it.

I wrote that correct_finding == "error" is "not merely unwritten — it is not an emittable value", citing the finish tool's JSON Schema enum at finding_verifier.py:158-161. That is true only for the finish path.

There is a second writer that bypasses the finish tool: finding_verifier.py:882 binds new_verdict = finding_update.get("should_be") and :921 assigns it directly to result["verification"]["correct_finding"]. should_be comes from a schema-less simple_text consistency-resolution call (:1029-1031), and /usr/bin/grep -n "should_be" over the file returns that single site with no enum, allow-list, or membership check anywhere.

So the accurate statement is unwritten in practice, not unwritable. The issue body has been corrected accordingly.

The measurement is unaffected: all 223 verify checkpoints in the reference run carry correct_finding != "error" and all 223 are adopted on resume.

Worth noting the shape this exposes: an unvalidated model-supplied string flows into a verdict field, which is the same class as #316.

[gadievron]

Fix opened in #377 (CI green, all checks passing). The verify checkpoint writer now copies the adapter-raise error string into the checkpoint, and the resume classifier retries ONLY checkpoints carrying it — the surgical signal for the retryable error class (the reporter's 48 hard errors). verification.incomplete alone is deliberately NOT a retry signal: five non-error fail-safe writers set it (deterministic outcomes), and #212 established refusals are deterministic — an incomplete-based classifier would retry them forever.

The sibling stale predicate in checkpoint.py's load_ids (MINOR, currently dead code for verify) and the StepCheckpoint.status() path are noted as follow-ups.