feat(tasks): finish the stage-output protocol migration (33 tasks, 40/40)

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Finishes the migration #60 started: all 40 registered tasks now return PromptRecord / PredictionRecord / JudgementRecord, so a consumer can read any sample's prompt, prediction, ground truth and verdict without knowing which task wrote the row. At 7/40 the protocol paid off nowhere — every consumer still branched per task for the other 33.

• 33 tasks migrated, one commit per family: math pass@k (6), clp/ppl (9), LLM-judge (3), code-execution (4), single-rollout generative (9) + ifbench, and scicode last in its own commit.
• feat(tasks): persist the raw grader reply in LLM-judged tasks #51's documented limit closes. hle/browsecomp/aa_lcr adopt extra.grader_output (the grader's full ModelOutput), so finish_reasons separates a reasoning judge that burned its budget thinking from an empty API response — the case feat(tasks): persist the raw grader reply in LLM-judged tasks #51 flagged as most reachable on HLE.
• The PROTOCOL_TASKS fork is gone. test_math_pass_at_k_family.py no longer builds two feedback shapes by set membership.
• ResourceMetrics is retired from all four code-exec tasks — the coupling [Feature]: finish the stage-output protocol migration — 32 remaining tasks #61 named, where an evaluator gaining n_cases/n_passed forced a client-side TypedDict widening.
• Grader spend now reaches profile.json (core-side; feat(core): uniform stage-output protocol, piloted on 7 tasks #60's accepted workaround, resolved — see below).
• ruler gains a per-sample verdict it never had; ifbench grades in feedback() rather than report(), mirroring feat(core): uniform stage-output protocol, piloted on 7 tasks #60's ifeval relocation.
• scicode is the one non-mechanical port. A sample is a chain of dependent sub-steps, each generating and executing code against the previous step's output — which is not what rollouts[] means (independent attempts at the same thing). So it maps to one rollout per problem: the whole step sequence is the attempt, per-step detail lives in rollouts[0].extra.steps, and metrics carries both axes with the headline derived from them. [Feature]: finish the stage-output protocol migration — 32 remaining tasks #61's inventory of 32 missed it because feat(scicode): add SciCode dataset + 0-shot generation task #42 landed just before feat(core): uniform stage-output protocol, piloted on 7 tasks #60.

On PR shape: #61 asks for one PR per family, and it's right that 33 tasks in one diff is a lot. The commits are exactly that split — one per family, plus the core profiler change and scicode's two — each independently green with its own parity evidence, so this can be landed as-is or peeled apart without rework; the call is the reviewer's. Reviewing commit-by-commit reads the way the issue intended.

Independent of #63 / #64. This branch touches neither pdm.lock nor pyproject.toml, so it does not conflict with the lock revert (#63) and #64's new drift gate does not fire on it. Its suite passes on both dependency sets — see the openai note in the review round.

Related Issues

Closes #61. Refs #60, #51 (its documented grader_reply limit closes here), #49.

Test Plan

Automated

• Lint/format clean (ruff check → all passed; ruff format --check → 368 files already formatted)
• Type check clean (ty check → All checks passed)
• Unit tests pass — 2890 collected across tests/unit + tests/integration + tests/acceptance (was 2827 on main; +63). CI reports it as 2834 passed, 55 skipped, 1 deselected; a local venv with more optional groups installed skips fewer.
• scripts/check_preflight.py all-PASS; sync_meta_index.py --check and sync_package_stubs.py --check clean

Manual — report parity

report.json metrics are unchanged for every task, verified against the pre-migration implementations (loaded from d622448c with the task decorator stripped), not only against new tests. 8,700 randomized differential trials, 300 per task, 8,700 identical. Each family's fixtures deliberately exercise its edge paths:

family	tasks	trials	edge paths deliberately covered
math pass@k	6	1800	k ∈ {1,2,4}, n ≥ k, rollouts < k so the short warning path runs
clp/ppl	6 reports (9 tasks; 4 ARC share arc_report)	1800	cmmlu's raw_sample=None skip; hellaswag acc + acc_norm
LLM-judge	3	900	judge_parsed=False so the unparsed-drop stays out of the calibration arrays
code-execution	4	1200	timeouts substring counter in both directions, incl. the near-miss "timed out"
single-rollout gen	9	2700	theoremqa empty; ruler across both metric families with partial matches; t_eval all 6 axes
scicode	1	300	167 solved problems / 4,529 graded steps; a deliberately sabotaged variant does mismatch, so the differential discriminates

• ifbench is covered instead by its unit test, whose expected report numbers are unchanged across the grading relocation — that equality is the parity check.
• Grader-spend routing verified end-to-end through the real TaskRunner + profiler, not just the helpers: a bare-record judge task's grader tokens land in the feedback stage's usage (20 in / 1 out per sample), the candidate's own spend stays attributed to infer, and the recorded call names the judge model.

Corrections to #61

• It groups hmmt_nov_2025 / imo_answer_bench as pass@k tasks that "also use an LLM grader". They do not — both grade deterministically (math_verify / the vendored verify_math_answer), and imo's own notes say an LLM grader would break the reproducibility contract. No grader detail is recorded because there is no grader.
• It counts 39 registered tasks; there are 40. scicode_0shot_gen (feat(scicode): add SciCode dataset + 0-shot generation task #42) landed just before feat(core): uniform stage-output protocol, piloted on 7 tasks #60 and is in no group — it is migrated here too, in its own commit (see Summary).

Deliberate non-changes (so no denominator moves)

• cmmlu's report still groups by ctx.raw_sample, not the record's extra.subject. The two agree whenever both exist, but a final whose raw_sample is absent has always been dropped from the macro-average; switching the source would silently change the denominator. The record carries subject regardless, so a row on disk is self-describing.
• ruler's report still calls the vendored string_match_* on the whole cell. The per-sample term recorded in feedback is the exact decomposition, but re-deriving the cell mean in the task would fork upstream's scoring — a vendored metric that no longer runs is a reproduction that drifts silently.
• mmmlu_kshot_clp and scicode_0shot_gen keep their TaskStageOutput box on infer. infer is not a record stage, so boxing it does not violate the bare-record rule. Both need the box for the same reason: one sample is several model calls, so the box carries the stage value while meta reports every call's usage to the profiler.
• The ppl tasks' "no option could be picked" sentinel -1 becomes None (the protocol's spelling). Equivalent downstream: both mismatch the gold and both map to "" through choice_text.

The one deliberate denominator change (90981721)

scicode's two accuracies disagreed about what a pipeline failure means: main_problem_accuracy counted a failed problem as unsolved, while sub_problem_accuracy dropped its steps from the denominator entirely. One metric was diluted by a failure while the other was silently inflated by its removal — and a full test split fell below the fixed 288-step denominator the official sub-problem figures are computed over, making the number incomparable to the leaderboard exactly when a run was unhealthy. A failed context still carries its raw_sample, so its tested steps are recoverable; they now stay in the denominator scoring zero, and a new unevaluated_steps field says how many of total_steps never ran.

This is the one place report.json moves, and only when fails > 0 — a clean run is identical. It is a separate commit ahead of the migration so the parity trials above measure the migration alone.

Notes on the metrics tier

metrics carries every multi-metric task, and in each the headline is derived from the mapping so the two cannot drift: hellaswag (acc/acc_norm), gsm8k-base (exact_match/flexible_exact_match), drop (em/f1), ifbench (strict/loose), scicode (main_problem_solved/sub_problem_pass_rate), t_eval (the co-equal axes its configuration scores).

t_eval needed a judgement call: correct is "every axis the evaluator scored came out at 1.0", deliberately not parse_rate. A task whose correct meant "the output parsed" would look near-perfect next to every other task on the one axis that is supposed to be comparable across them. "The axes it scored" is load-bearing rather than decorative — see the review round below.

Review round — fixes in the latest push

Two of these were live bugs in the migration; the rest are latent or type-level.

• t_eval's correct was structurally unreachable. feedback recorded every axis _evaluate returns, including ones the configuration never scores — and correct is derived as "every recorded axis is 1.0", so under the default eval_thought=False a pre-seeded thought: 0.0 pinned correct to False on every sample of every run, emptying the one axis the protocol exists to make comparable across tasks. metrics now carries only the scored axes, from a _metric_keys() shared with the macro-average so the recorded metrics, the derived correct and the report cannot disagree. report.json does not move: the same axes were already being macro-averaged, and the *_parsed variants now read defensively so the str modes that score no args still report 0.0, as before. Fixes a latent NameError on the way through — metric_keys was left unbound for any prompt_type outside json/str, surfacing deep in the macro-average rather than at config time.
• ruler's report crashed on the resume path. report() indexed rollouts[0]["prediction"], but postprocess normalizes a blank response to None and serialization omits None-valued keys — so on the resume-report path (every sample complete, report.json missing) the key is absent, not null, and an already-paid-for long-context run died at the last step with KeyError. Now .get(...) or "". The test helper round-trips both records through obj_to_dict, so every report test that builds a final now asserts against the shape report() actually has to read, not the more forgiving in-memory one.
• Four judge tasks spelled grader_output as a literal instead of records.GRADER_OUTPUT_KEY. The name is what the runner reads back to route grader spend into profile.json, so a divergence is silent — tokens on disk, missing from the profile.
• The five reports that count timeouts read r["extra"]["msg"].lower() unguarded. msg is passed straight through from the evaluator response, so a null makes a fresh run raise AttributeError — not only a resumed one. Now (r["extra"].get("msg") or "").
• The record predicates were annotated Any. Any passes silently in both directions — a caller may pass anything, and the body may do anything with it. object is the accurate spelling, and switching to it surfaced that is_judgement_record's narrowing never reached its caller: only Any made iter_grader_outputs' value.get("rollouts") type-check. Both sniffers now return TypeGuard[Mapping] — deliberately not TypeIs, since a mapping that is not a record still returns False, which detect_empty_postprocess relies on when it tests isinstance(post_result, dict) after a False. Two of the three signatures predate this PR. Also replaces a non-discriminating assertion: iter_grader_outputs({"rollouts": "not a list"}) read as covering the per-rollout isinstance guard, but without n_rollouts the value is not a judgement at all, so it returned at the first guard and deleting the one under test failed no test.
• One new t_eval test could not construct its instance in CI. eval_thought=True builds the embedding client in __init__ with api_key=os.getenv("SIEVAL_EMBED_API_KEY", ""), and an explicit "" is not None — it skips the OpenAI client's own OPENAI_API_KEY fallback, so openai raises Missing credentials at construction. The test now sets the env var: still a real instance, still covering the eval_thought=True branch, no embedding call. Reproduced rather than inferred from the traceback — api_key="" gives the CI error exactly, and it does so on both openai 2.45.0 (locked when this branch was cut) and 2.9.0 (restored by chore(deps): revert the unrequested lock drift from d805418a #63, now on main), so the fix is not version-specific. The whole suite was re-run against the 2.9.0 set before chore(deps): revert the unrequested lock drift from d805418a #63 landed, so this branch is verified on the dependency set it will actually merge into.
• …and the message named a variable that would not help. Follow-on from the above, since the wart is what made that test fail in the first place. Missing credentials names OPENAI_API_KEY, which is useless here: the endpoint is our own embedding service (SIEVAL_EMBED_API), not OpenAI's. The failure also lands at construction, before any run output, so that message is all an operator gets. eval_thought=True now checks the key itself and raises a ValueError naming SIEVAL_EMBED_API_KEY, pointing at SIEVAL_EMBED_API for a non-default endpoint, and saying the axis is optional (leave eval_thought unset). Both None and "" take that path — "" is the reachable case, an exported-but-blank variable. No new exception class: sieval has no config-error base (ResultDirExistsError / ResumeVersionError both subclass builtins), and tasks must not change core/. The new test discriminates — pre-fix the constructor raised OpenAIError, which is not a ValueError subclass, and its message contained OPENAI_API_KEY. Default eval_thought=False is untouched: it builds no client, and nothing in-tree passes eval_thought=True.
• Four migrated tasks still declared the shapes they no longer return. gsm8k_0shot_gen, gsm8k_kshot_base_gen, hendrycks_math_kshot_base_gen and theoremqa_kshot_base_gen had bodies fully on the builders but Task[...] slots still reading list[ChatCompletionUserMessageParam] / str for the prompt, str / list[str] / a per-task Prediction TypedDict for the prediction, and a per-task Feedback TypedDict for the judgement. Nothing caught it: the stage methods carry no return annotations, so the generic slots are never cross-checked against the bodies — and the five dead TypedDicts stayed reachable as those slots' arguments, which is exactly why the deletion sweep in the checklist below read clean. On-disk output was already correct, so no report or shard moves; what changes is that the declared type now matches it, the five unused TypedDicts are gone, and gsm8k_0shot_gen no longer imports openai at all.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ — the core change touches only core/tasks/records.py, core/utils/meta.py, core/runners/runner.py
• Deleted code verified — grepped to 0 remaining references for every removed name: ARCFeedback, JudgeFeedback, GradeFeedback, ResourceMetrics, RulerFeedback, StepFeedback, ifbench._get_report, and the per-task Feedback/Preprocessed/Prediction TypedDicts. Remaining grader_reply hits are prose explaining what replaced it. The per-task TypedDicts held out longest — five of them survived in four files as arguments to stale Task[...] slots, so a reference grep could not see them; removed in 67bd03bd (see the review round above), and the sweep now reads 0 for real.

If: Breaking Change

• Described what breaks and migration path in Summary — see below
• Existing tests updated to reflect new behavior

On-disk schema, for the 33 newly-migrated tasks. Result dirs written by 0.7.x hold the old feedback shape and cannot be re-reported by the new report(). Migration path: start fresh. This rides the 0.8.0 minor bump already staged for #60 and the infer-recipe split, so the --resume version gate rejects a 0.7.x → 0.8.x resume with a version message rather than a confusing shape error. report.json shapes are unchanged for every task except scicode's added unevaluated_steps (above), so leaderboards, alignment cards and resolve_model_name are unaffected.

Module-level names removed from sieval/tasks/* are importable, so their removal is user-visible even though nothing in-tree referenced them (verified above).

Follow-ups (not in this PR)

Three of #61's adjacent items concern the code evaluator. None is blocked upstream — code-evaluator is vendored in this repo (vendor/code-evaluator/, formerly a submodule), so each is a local patch plus a VENDORED.md entry, exactly how the n_cases/n_passed patch already in that copy was done. What actually gates two of them is a redeploy: tasks reach the evaluator over HTTP (SIEVAL_CODE_EVAL_API), so a new response field only appears once the image is rebuilt from vendor/code-evaluator/docker/. The sieval side reads these with .get(), so an unpatched evaluator keeps working with the field absent.

• Land the vendored n_cases/n_passed patch upstream and re-vendor. Pure upstream hygiene — it keeps the fork delta from growing. It gates nothing: the patch is in the vendored copy, and all five tasks that call the evaluator for test-case counts already read both fields (the four migrated here plus livecodebench_code_generation_0shot_gen from feat(core): uniform stage-output protocol, piloted on 7 tasks #60).
• Failure taxonomy belongs on the evaluator response next to n_cases/n_passed — a client-side classifier over free-text msg decays silently, which is why feat(core): uniform stage-output protocol, piloted on 7 tasks #60 removed one. Doable in the vendored copy; needs the redeploy.
• Full per-test-case pass rate needs dropping the evaluator's short-circuit. Also local, but it carries a real runtime cost (every case runs, which can newly trip the shared timeout budget), so it wants an opt-in env flag — precedent: CODE_EVAL_FLOAT_TOL in the existing patch set.

Not code-evaluator, and not deferred any more: the eval_thought credential error is fixed in this PR (see the review round).

Still open, unrelated to the above: task-level service env vars are undocumented. SIEVAL_EMBED_API_KEY, SIEVAL_EMBED_API and SIEVAL_CODE_EVAL_API appear only in code — the README and FAQ cover SIEVAL_DATA_DIR alone. Pre-existing and broader than this PR; worth its own small docs change.

🤖 Generated with Claude Code

[ethan-scitix]

Seventh commit: a scicode metric fix, not part of the 32-task migration

e4d3cecb rides on this branch but is a different concern from the six per-family migration commits the description covers. Flagging it so it isn't read as more of the same.

scicode_0shot_gen reports two accuracies, and they disagreed about what a pipeline failure means:

	before	after
main_problem_accuracy	failed problem = unsolved	unchanged
sub_problem_accuracy	failed problem's steps dropped from the denominator	steps stay in, scoring zero

So a failure diluted one metric and silently inflated the other, and a full test split fell below the fixed 288-step denominator the official sub-problem figures are computed over — the number became incomparable to the leaderboard exactly when a run was unhealthy.

The comment justifying the drop ("their step counts are unknown") was false — a failed TaskContext still carries its raw_sample, so its tested steps are recoverable. They are now recovered, with the three scientist-authored steps (13.6 / 62.1 / 76.3) excluded just as on the success path. A context that failed before its sample was loaded has nothing to recover and is skipped; it still dilutes main-problem accuracy.

New report.json field unevaluated_steps — how many of total_steps never ran. Lets a reader separate "the model failed these steps" from "the pipeline never reached them"; read it alongside fails.

No user-facing break: scicode (#42) is unreleased, so this pins the definition down before it ships rather than changing it under anyone.

Audited against a prior in-house SciCode port — and where I deliberately did not follow it

The rest of the port cleared: identical sub-problem formula, byte-identical code extraction, identical prompt templates, and equivalent special-step gold injection under the default verbatim mode.

Not adopted from it:

• Filtering failures out of both metrics. That hides pipeline failures and makes the denominator depend on run health. Keeping both denominators complete is both self-consistent and faithful to upstream's fixed 65 / 288.
• An empty-step_background guard. Fail-loud is correct here: a silently-empty background yields numbers that aren't comparable to either setting, the field is required in the sample TypedDict, and upstream crashes too.
• Decoupling the multi-step template from with_background. That is an extension over upstream, and would add an uncalibrated score-relevant knob.

Why it is on this branch: it is the prerequisite for scicode's own protocol migration. Putting sub_problem_accuracy into the metrics tier means welding its definition into the shared contract, so the denominator semantics had to be settled first. The remaining open call is scicode's rollout shape (dependent sub-steps as rollouts vs. one rollout per problem with step detail in extra) — untouched here.

Tests: 25 in tests/unit/tasks/test_scicode_0shot_gen.py (was 22); full suite 2931 passed; ruff / ty / check_preflight.py / both sync scripts clean.

[ethan-scitix]

Eighth commit: scicode migrates too — adoption is 40 of 40

c46185e9 closes the one gap I flagged in the previous comment. That comment ended with "the remaining open call is scicode's rollout shape … — untouched here"; it is no longer open, so read this as superseding that sentence.

scicode_0shot_gen is the one registered task #61's inventory of 32 misses (#42 landed it just before #60). It is not a mechanical port — a scicode sample is a chain of dependent sub-steps, each generating and executing code, which is not what rollouts[] means: independent attempts at the same thing.

One rollout per problem. The whole step sequence is the attempt.

	where it goes	why
main_problem_solved, sub_problem_pass_rate	metrics (both tiers, headline derived)	co-equal — neither is a footnote of the other
per-step verdicts	rollouts[0].extra.steps	mechanism detail
correct_steps, total_steps	record extra	report() pools them

That last row is the load-bearing one, and it is the same call ifeval's instruction-level accuracy already rests on: a per-problem rate cannot reconstruct a pooled one, so the raw counts have to survive to report time. Averaging the sub_problem_pass_rate values across problems would produce a different — and wrong — sub_problem_accuracy.

Steps-as-rollouts was the alternative and it loses on semantics, not on convenience: step i's prompt embeds the model's own code from steps 1..i−1, so the steps are neither independent nor attempts at the same thing. Recording them as rollouts would make n_correct / n_rollouts read as a pass-rate over retries for every generic consumer.

Two details worth a second look: what prompt holds, and when prediction is None

prompt is the first generated step's prompt, not the whole problem and not all N step prompts.

It is knowable at preprocess because every step preceding it is a special step whose gold code is static (13.6 / 62.1 / 76.3 are never generated). infer then reuses that recorded prompt for the step rather than rebuilding it, via a shared _gold_code() — so the record structurally cannot disagree with what the model actually saw. Only steps 2..N are rebuilt, since their prompts embed the model's own prior output and cannot exist before inference.

extra carries first_generated_step / n_generated_steps / previous_code, so prompt is not misreadable as the whole model input. The two rejected alternatives: persisting all N per-step prompts is quadratic text (step N's prompt contains every earlier step's), and putting the sample in duplicates raw_sample.

prediction is None iff no tested step produced extractable code. A partial extraction miss is still a scoreable answer and keeps its prediction. This is deliberate anomaly-rule hygiene: detect_extraction_failure reports rollout indices where extracted is falsy, and a multi-step task where any one empty step nulled the prediction would fire that rule on most problems, making it noise. New StepCode.extracted_code holds the model's per-step answer so the prediction doesn't re-run the extractor.

reference is None — the ground truth here is a procedure (this problem's per-step test suites), same as human_eval.

report.json unchanged. 300 randomized differential trials against the pre-migration report() loaded from e4d3cecb, 0 mismatches, covering 167 solved problems and 4,529 graded steps — with new-shape finals produced by running the real feedback() against a stubbed evaluator, not hand-built records. A deliberately sabotaged variant (perturbing correct_problems) did mismatch, so the differential discriminates rather than comparing two copies of the same bug.

Tests: 34 in tests/unit/tasks/test_scicode_0shot_gen.py (was 25), including one that drives the real feedback() twice over scripted verdicts and feeds both records into report() — so report is asserted against what feedback actually writes, not against a fixture that could drift from it. Full suite 2940 passed; ruff / ty / check_preflight.py / both sync scripts clean.