feat(tasks): persist the raw grader reply in LLM-judged tasks

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Adds grader_reply to the feedback record of the whole LLM-graded family — hle_0shot_gen, browsecomp_0shot_gen, simpleqa_verified_0shot_gen, aa_lcr_0shot_gen — holding the grader's reply verbatim and in full on every attempt.
• The parsed verdict was persisted; the text it was derived from was not. A report could say "7 replies didn't parse" (judge_unparsed, from feat(hle): add Humanity's Last Exam dataset + 0-shot LLM-judge task #43) with no way to tell truncation from format drift from an API error from a genuine matcher gap. The HLE regexes have been hardened twice, both times from reasoning about likely reply shapes rather than observed ones — because the failures were never recorded.
• Beyond HLE: parse_grade resolves a reply it cannot read to INCORRECT (browsecomp, aa_lcr) or NOT_ATTEMPTED (simpleqa_verified), so a drifted reply is indistinguishable from a real negative verdict or a real abstention — and the F1 treats abstention very differently from a wrong answer.
• Grading is not reproducible from artifacts: a grader model version is not pinnable like a Hub revision, so re-running a judge need not reproduce a past verdict. The reply is the only durable evidence of what the grader actually said.
• Docstrings and reference_impl.notes on all four claimed only the grade + grader model id were persisted — stale, not merely incomplete. Updated, hence the sieval/meta/index.json regen.

Settling the three questions #48 left open:

Question	Decision	Why
Payload	Full, always	The record already carries predicted, a full generation typically longer than a judge reply, so the marginal cost is small. Storing only unparsed replies leaves a wrong-but-parsed verdict — the failure that actually moves scores — unauditable.
Gating	No knob	A record_* knob would put task-specific state in TaskRunnerConfig (a core/ layer that forbids it) and widen --resume strict matching for no scoring benefit.
Field name	grader_reply	Reads consistently with the existing grader_model.

aa_lcr is now included. #44 merged while this PR was open, so the family-wide pass covers it rather than leaving a fourth task to diverge (plan posted on #44 before it merged). One wrinkle specific to it: the empty-candidate short-circuit grades INCORRECT without calling the checker, so grader_reply = "" is bound explicitly on that path. Leaving the variable unbound there would attribute the previous attempt's real verdict to an ungraded answer when n > 1 — AA-LCR runs n=3. Empty is unambiguous: the branch is exactly not predicted.strip(), so the empty predicted in the same record identifies it. A sentinel string ("<not invoked>") was rejected as a magic value the record does not need.

No behavior change to scores. Purely additive to the artifact; no metric reads the new field.

Related Issues

Closes #48. Refs #43 (which added the judge_unparsed count this makes diagnosable), #44 (whose task this now covers).

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check clean; mypy --strict error count is identical before and after this change on every file touched — all pre-existing unannotated stage overrides, verified by stashing the source changes and re-running)
• Unit tests pass — pytest tests/unit → 2511 passed; the four task files → 58 passed
• python scripts/check_preflight.py → all checks pass, including the meta-drift check after regenerating the index

New assertions cover every branch, and are written to fail against the rejected alternatives:

• Parsed path — a multi-line reply containing reasoning the parse discards is asserted equal in full. This fails a "store only the matched fields" or "store only on parse failure" implementation.
• Drift path — HLE unparseable (judge_parsed is False), browsecomp no-verdict (INCORRECT default), simpleqa_verified non-matching (NOT_ATTEMPTED default), aa_lcr unreadable (INCORRECT default) each assert the reply survives, which is the motivating audit case.
• aa_lcr not-invoked path — feedback(["Rising", ""]) asserts the graded attempt keeps its reply and the short-circuited one is empty. This is the assertion that fails if the reply = "" binding is dropped, so it discriminates the leak described above rather than merely restating the happy path.

Manual

Not applicable — no scoring path changed, so there is no score to compare. The four tasks' existing validation numbers (HLE gpt-oss-20b 12.14/3.61; BrowseComp gemma-3-27b-it 0.316%; SimpleQA Verified gemma-4-31B-it F1 9.95; AA-LCR gpt-oss-120b/20b within ~2–3 pts of the AA leaderboard) are unaffected by an additive feedback field.

Resume compatibility, verified by inspection: records written before this change simply lack grader_reply, and nothing in report() reads it, so resuming an in-flight run does not raise — it yields records mixed on this field, which a cross-version resume already warns about.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — all four modules already carry it; unchanged
• No new upper-layer dependencies added to core/ — core/ is untouched, which is the reason there is no config knob
• Deleted code verified — no remaining call sites depend on it — nothing deleted; change is additive

If: New or Modified Benchmark

• Reference paper/repo linked in Summary — unchanged from each task's existing reference_impl.url (HLE @ 26dca2e, arXiv:2504.12516, arXiv:2509.07968, AA-LCR card @ bdae010b)
• Score comparison table — no scoring path changed
• Dataset loading tested — no dataset change
• Task registered in package-level __init__.py — already registered; no new task

🤖 Generated with Claude Code

[ethan-scitix]

Scope note: aa_lcr_0shot_gen should become the fourth member of this pass, handled here rather than in #44.

Sequencing, since #44 is still open: this PR waits for #44 to merge, then rebases onto the new main (which will have aa_lcr) and adds grader_reply to all four tasks in one place. That keeps one PR as the single owner of the field, the "in full on every attempt, no config knob" convention, and the notes wording — rather than having #44 invent its own copy of the field in parallel.

The cost is that aa_lcr sits on main briefly without the field. That's acceptable: the window is however long this PR takes, and it removes work from a fork PR that still has an outstanding blocker.

Two things to remember when it lands:

1. The empty-candidate short-circuit. Unlike the other three, aa_lcr grades an empty/whitespace candidate INCORRECT without invoking the checker (aa_lcr_0shot_gen.py:182), so there is no reply to store. Going with option 1 from the feat(aa-lcr): add dataset + 0-shot LLM-equality-checker-graded task #44 discussion: "grader_reply": "" plus a comment on that branch noting the checker was never called. grade == "INCORRECT" with an empty predicted already identifies the branch unambiguously, so a sentinel string isn't worth the magic value.
2. reference_impl.notes needs the same edit as the other three. aa_lcr's notes currently say "the per-sample grade and grader model id are persisted in the feedback record" — that becomes stale the moment the field is added, and it drives sieval/meta/index.json, so regen and commit it or preflight's meta-drift check fails.

Also worth knowing for this PR: on aa_lcr the stored reply is what makes the grader's phrasing space auditable at all. Two accuracy-inflating parse bugs have already been found there by reading the prompt rather than the replies (negated verdicts in 216614fc, hedged verdicts in 4a8c858f), and neither could be checked against the 1200 real replies from the validation runs because they were never persisted. Those are gone permanently — this field is what stops that recurring.

[ethan-scitix]

Heads-up on a conflict, and the agreed resolution — this PR goes first.

#60 (stage-output protocol pilot) independently added grader_reply to simpleqa_verified before I spotted this PR, so the two overlap. A trial merge conflicts in three files:

• sieval/tasks/simpleqa_verified_0shot_gen.py
• tests/unit/tasks/test_simpleqa_verified_0shot_gen.py
• sieval/meta/index.json (both PRs rewrote that task's reference_impl.notes, so both regenerated the index)

The other three judge tasks here — hle, browsecomp, aa_lcr — are untouched by #60, so they conflict with nothing.

Plan: this PR merges as-is (it is the family-wide change and it Closes #48), then #60 rebases on top. In that rebase simpleqa_verified's grader_reply moves from a top-level feedback field to extra.grader_reply, because the protocol puts grader-specific detail under extra while correct / reference / the rollout counts stay at the top. The field, its name and its rationale are unchanged — only its position in the record. The four assertions this PR adds for simpleqa_verified get rewritten to the new shape, not dropped, and #60 additionally asserts that the grader's per-call usage (extra.grader_calls) is persisted.

Nothing to do on this PR beyond an Update branch — it is MERGEABLE, just BEHIND main.

Also worth flagging for the eventual judge-family protocol migration (tracked as a follow-up on #60): the profiler only reads task-supplied TaskStageMeta["model_calls"], and protocol records are returned bare rather than boxed in a TaskStageOutput, so grader spend lands on disk but not in profile.json. That needs a core-side fix before all four tasks migrate, otherwise each one inherits the blind spot.