feat(core): uniform stage-output protocol, piloted on 7 tasks

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Stage return types are free-form generics, so every task invented its own shape for "the extracted answer" and "was it right". Nothing downstream could read a sample's answer / ground truth / correctness without a per-task special case — and since raw_sample is never serialized, ground truth reaches disk only if a task puts it in a stage result, which two of the seven tasks below never did.

• New: a record per stage in sieval/core/tasks/records.py — PromptRecord / PredictionRecord / JudgementRecord + builders. infer is deliberately left alone (generative tasks already return ModelOutput uniformly; this does not hold for _ppl/_clp, which fan out over candidates — see Follow-ups).
• Migrated: mmlu_pro, gpqa_diamond, aime_2026, hmmt_feb_2026, livecodebench_code_generation, ifeval, simpleqa_verified. Other ~80 tasks untouched; adoption is per-task and legacy shapes keep working.
• Sample-level envelope with a rollouts[] list, so n=1 and n>1 share one schema and a gold is stored once instead of once per rollout (AIME with n=4 used to repeat it 4×). n_rollouts/n_correct are materialized as the sample-level pass rate. reference=None means the ground truth is a procedure, not a value — LiveCodeBench's is a test suite, described in extra.
• ifeval now grades in feedback() instead of report(), so per-sample strict/loose verdicts + per-instruction results are finally persisted.
• simpleqa_verified now persists the raw grader reply: parse_grade silently defaults to NOT_ATTEMPTED, so a defaulted grade was previously indistinguishable from a real one.
• New extraction_failure anomaly rule reports real per-rollout indices instead of the {0} sentinel.
• vendor/code-evaluator gains {n_cases, n_passed} on the response, at zero extra execution cost.

A migrated record on disk ({iteration}/final/*.jsonl, aime_2026 with n=2):

"preprocess_result": {"prompt": [...], "reference": "42"},
"postprocess_result": {"rollouts": [
    {"index": 0, "prediction": "42", "extracted": true},
    {"index": 1, "extracted": false}]},
"feedback_result": {"reference": "42",
    "rollouts": [{"index": 0, "correct": true}, {"index": 1, "correct": false}],
    "n_rollouts": 2, "n_correct": 1}

Two rules that are easy to break

Both are documented on the records module, because violating either silently undoes the point of the change.

1. Records are returned bare — never wrapped in TaskStageOutput. The runner preserves that box as the stage value, so boxing one task would persist {"value": {...}, "meta": {...}, "__sieval_cls__": ...} while its peers persist a flat record. Accepted cost: the profiler only reads task-supplied TaskStageMeta["model_calls"], so simpleqa_verified carries the grader's ModelCallMeta inside the record (extra.grader_calls) — grader spend is on disk but not in profile.json. The fix belongs in core; boxing would be working around it.
2. obj_to_dict drops None-valued keys, so prediction: None and reference: None are absent on disk, not null (False and 0 survive — the check is is not None). That is why extracted and the n_* counts are explicit fields rather than derived from a missing prediction: the flags and counts are what survive the wire. Pinned by a round-trip test.

Related Issues

Refs #49 (per-stage record schema; the run-level output-file envelope stays open and unprejudiced).

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check) — only pre-existing sieval/_version.py (pdm-generated, gitignored)
• Type check clean (ty check → all passed). mypy --strict adds 9 type-arg errors on bare dict, matching the project's existing style, on top of 63 pre-existing on main
• Unit tests pass — 2787 across tests/unit + tests/integration + tests/acceptance
• scripts/check_preflight.py all-PASS; sync_meta_index.py --check and sync_package_stubs.py --check clean

New tests: tests/unit/core/tasks/test_records.py (27, incl. the None-dropping round trip), tests/unit/tasks/test_stage_protocol_reports.py (21 report-parity), plus detect_extraction_failure coverage in test_anomaly.py.

Manual — report parity

report.json metrics are unchanged for every task. Verified against the pre-migration implementations, not only against new tests:

task	method	result
gpqa_diamond	replayed a real 792-record run; the old report() first had to reproduce its stored report.json to prove the harness valid	identical — 49.621212121212125
aime_2026, hmmt_feb_2026	replayed real AIME runs incl. 30 samples × 64 rollouts (285/1920 correct); pass@8 also checked against an independently computed estimator	identical
ifeval	120 real dataset rows through both the old whole-set grading and the new per-sample grading — all 8 metrics and key order	identical (fixture discriminating: strict 14/120, loose 23/120, differing on 9)
mmlu_pro, livecodebench, simpleqa_verified	300 randomized differential trials each against origin/main's report() (no completed run for these exists on disk)	900/900 identical

IFEval's stage relocation is safe because both graders are pure per-sample — their only use of the response map is prompt_to_response[inp.prompt] — and all 541 IFEval prompts are unique, so the old prompt-keyed dict had no collisions to preserve.

• End-to-end through the real TaskRunner + saver (stubbed model, everything downstream production code): records land self-describing, and extraction_failure fires per rollout — aime_2026 flagged [1] (the rollout with no boxed answer), mmlu_pro flagged [0] on a refusal.
• vendor/code-evaluator exercised on all five paths: all pass → n_passed=4; fail at case 2 → 2; compile error → 0; arity mismatch → 0; subprocess timeout → None.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ — records.py imports only sieval.core.types
• Deleted code verified — Feedback/GradeFeedback/Preprocessed TypedDicts and ifeval._get_report had no call sites outside their own modules (_get_report's tier0_*/tier1_* breakdowns were computed and never returned)

If: community/ Changes

• Upstream diff documented — vendor/code-evaluator/VENDORED.md gains an entry for the test-case-progress patch; still to be landed upstream in scitix/code-evaluator and re-vendored
• License attribution preserved

If: Breaking Change

• Described what breaks and migration path in Summary — see below
• Existing tests updated to reflect new behavior

On-disk schema, for the 7 migrated tasks only. Result dirs written by 0.7.x hold the old feedback shape and cannot be re-reported by the new report(). Migration path: start fresh. This rides the 0.8.0 minor bump already staged for the infer-recipe split, so the resume version gate rejects a 0.7.x → 0.8.x resume with a version message rather than a confusing shape error. report.json shapes are unchanged for every task, so leaderboards, alignment cards and resolve_model_name are unaffected.

Two deliberate ifeval behaviour changes, outside the parity claim: report() no longer raises ZeroDivisionError on an empty final set (returns zeros, matching every sibling task), and the dead tier0/tier1 code is gone.

One fleet-wide effect: adding an anomaly rule changes rules_hash, so every task's next run rotates anomalies.json (_backup_if_rules_changed handles it — harmless, but visible).

Follow-ups (not in this PR)

• Finish the migration. The math family is currently split — aime_2026/hmmt_feb_2026 moved while their near-identical siblings did not; tests/unit/tasks/test_math_pass_at_k_family.py::PROTOCOL_TASKS bridges the gap and should shrink to empty.
• Judge family (hle, browsecomp, aa_lcr) maps exactly as simpleqa_verified did.
• _ppl/_clp are blocked on a design call: they fan out over candidate answers rather than rollouts, so "one rollout + per-candidate scores in extra" needs confirming against a real ppl task.
• Route grader model_calls into the profiler (core-side), ideally before the judge family migrates in bulk.
• Full per-test-case pass rate needs dropping the evaluator's short-circuit — slower, and it can newly trip the shared timeout budget. Ship as an opt-in flag if wanted.

🤖 Generated with Claude Code

[ethan-scitix]

Two review items applied, plus a dependency to record.

1. Case counts now come back on every source, not just LiveCodeBench (890d866d)

n_cases / n_passed are reported for all sources, so a caller can compute a pass rate without branching on source. A direct run (human-eval / mbpp / scicode, or livecodebench without test) is one all-or-nothing case, so the pair is 1/1 or 1/0. For those it is redundant with status by construction — reported anyway, because a field that is sometimes absent is a field every consumer has to special-case. Both stay null only when nothing ran at all (unsupported language or source), where data is itself null. Verified on the direct-run path: clean run → 1/1, raises → 0/1, syntax error → 0/1.

This widened the blast radius on the sieval side, which is worth stating: the four tasks that persist the evaluator payload verbatim as metrics (human_eval ×2, mbpp, livecodebench kshot base) each declare a ResourceMetrics TypedDict listing only the four resource fields, which would now misdescribe what is actually written to disk. Added the two counts as NotRequired so the declared type matches the record. No behaviour change — nothing reads them, and the passthrough was already persisting them.

2. vendor/code-evaluator docs are English now (890d866d)

README.md translated in full; content otherwise unchanged apart from the case-count section. No CJK left anywhere under vendor/code-evaluator/. VENDORED.md records both edits as local patches still to be landed upstream in scitix/code-evaluator.

3. Depends on #51 — that one merges first

#51 adds grader_reply to all four LLM-judged tasks and Closes #48. This PR independently added the same field to simpleqa_verified before I spotted it, so the two overlap. A trial merge conflicts in sieval/tasks/simpleqa_verified_0shot_gen.py, tests/unit/tasks/test_simpleqa_verified_0shot_gen.py, and sieval/meta/index.json (both rewrote that task's reference_impl.notes).

#51 merges as-is; this PR then rebases on top, and simpleqa_verified's grader_reply moves from a top-level feedback field to extra.grader_reply — the protocol keeps grader-specific detail under extra while correct / reference / the counts stay at the top. Same field, same name, same rationale; only its position changes. #51's four simpleqa_verified assertions get rewritten to the new shape rather than dropped.

Do not merge this before #51. Details on #51: #51 (comment)

[ethan-scitix]

Rebased onto main post-#51 (88754304). Three commits, linear on top of it, force-pushed as 79f04d29.

Conflict resolution — all three files, as planned

file	resolution
sieval/tasks/simpleqa_verified_0shot_gen.py	grader_reply moved from a top-level feedback field to extra.grader_reply. #51's rationale wording is kept, not replaced — its docstring and reference_impl.notes explanations were better than mine, and I adapted them to the new location rather than reverting to my terser version.
tests/unit/tasks/test_simpleqa_verified_0shot_gen.py	All three of #51's additions ported to the new shape, including its deliberately multi-line grader reply (the fixture that fails a "store only the matched letter" implementation) and its test_feedback_persists_reply_behind_not_attempted_default. Nothing dropped.
sieval/meta/index.json	Regenerated from the live registry rather than hand-merged; sync_meta_index.py --check clean.

hle / browsecomp / aa_lcr are untouched by this PR, so #51's field is intact on all four tasks.

#51's documented limit is now closed for simpleqa_verified

#51 flagged that the stored reply is ModelOutput.texts — response content only — so a reasoning autorater that spends its whole budget thinking records an empty reply, indistinguishable from an empty API response, and assigned the follow-up to this PR since grader-call metadata lives under extra here. It is closed: extra.grader_calls carries the grader's ModelCallMeta, whose finish_reasons separates the two. An actual on-disk record from a run with a budget-exhausted grader:

"extra": {
  "grade": "NOT_ATTEMPTED",
  "grader_reply": "",
  "grader_model": "grader-reasoning",
  "grader_calls": [{"model": {...}, "usage": {...}, "finish_reasons": ["length"]}]
}

["length"] = burned the budget thinking; ["stop"] = genuinely empty response. Asserted by test_finish_reason_separates_a_thinking_grader_from_an_empty_response, which pins that the two cases are identical on grader_reply and grade and separable only via the call metadata — so it fails if grader_calls is ever dropped.

Two scoping notes on that, stated rather than left implicit:

• reasoning_texts is still not stored. It is unbounded, and the finish reason already answers the question the record needs to answer. If reading a judge's actual reasoning turns out to matter, that is a separate decision about payload weight.
• This closes it for simpleqa_verified only. hle / browsecomp / aa_lcr still carry the limit, and HLE is the case feat(tasks): persist the raw grader reply in LLM-judged tasks #51 named as most reachable since it normally runs a reasoning judge. It closes for them when they migrate to the protocol — tracked with the judge-family follow-up.

Re-verified after the rebase

• 2791 tests pass (tests/unit + tests/integration + tests/acceptance); +4 from the ported and new assertions
• ruff check / ruff format --check / ty check clean; check_preflight.py all-PASS; both sync scripts --check clean
• Parity re-run against the post-feat(tasks): persist the raw grader reply in LLM-judged tasks #51 main, since feat(tasks): persist the raw grader reply in LLM-judged tasks #51 changed that task's feedback dict: 300 randomized differential trials, F1 and every other metric identical, key order included. feat(tasks): persist the raw grader reply in LLM-judged tasks #51 did not touch report(), so the aggregation path is unchanged — but worth confirming rather than assuming.

[ethan-scitix]

Follow-up on the three review questions. Only Q1 needed code; Q2 and Q3 I'm not changing, with reasons below. Folded into the protocol commit (14a90187, force-pushed).

Q1 — grader output: store the full ModelOutput, nothing hand-picked

You were right that the grader is just another model and should be recorded as one. The old shape (grader_reply + grader_calls = a ModelCallMeta projection) dropped exactly one field — reasoning_texts — which is the field that matters for a reasoning autorater. New shape:

"extra": {
  "grade": "CORRECT",
  "grader_output": {
    "model": {"model": "grader-o3", ...},
    "texts": ["A"],
    "reasoning_texts": ["The candidate names Shakespeare, which matches the gold."],
    "usage": {"input_tokens": 55, "output_tokens": 4, "total_tokens": 59}
  }
}

grader_output is the grader's ModelOutput flattened with obj_to_dict(out, add_type=False). That captures every field (reply, reasoning, usage, finish_reasons, request_params, model id), loses nothing, and picks up any future ModelOutput field for free. add_type=False keeps it a plain dict, so the record stays uniformly plain-dict rather than nesting a typed @sieval_record — verified it round-trips through save/load with no type markers. grade stays a separate key because it's the parsed verdict (task logic), not model output. report() reads only grade, so F1 is unchanged.

Two new tests pin the point: test_grader_reasoning_is_persisted_not_dropped (the info-loss fix — reasoning reaches disk) and the finish-reason test now reads it from grader_output.

One consequence to flag: this makes simpleqa_verified the template for the judge family. The three non-migrated judge tasks (hle/browsecomp/aa_lcr) keep #51's flat grader_reply until they migrate, then adopt grader_output. Staged in the changelog note.

Q2 — judge→grader naming: not doing it, and here's why it isn't the clean rename it looked like

I mapped the surface before touching anything. grader (the role — task arg, model, #51's shared field) is already consistent across all four tasks. judge is concentrated in HLE and is not sloppiness:

• It's upstream's own term — JUDGE_PROMPT and the metric kernel are vendored byte-for-byte from hle_eval/run_judge_results.py.
• parse_judge returns (correct, confidence, parsed) — a different contract from the other three modules' parse_grade, which returns a grade string. Renaming parse_judge → parse_grade would put two different signatures under one name: worse than the inconsistency it "fixes".

So a global rename would create same-name/different-contract collisions, fight vendored upstream vocabulary, and drag HLE + its community module + its test into a PR that otherwise doesn't touch them. JudgementRecord/rollout_judgement are the verdict layer (mechanism-agnostic) and are correctly named. If you still want the cosmetic JudgeFeedback → HLEFeedback cleanup, I'd do it as its own small PR rather than smuggle it in here.

Q3 — consolidation: nothing to move

records.py is single-purpose. The only mild incohesion nearby is pre-existing and unrelated: TaskRunMeta (meta.json) and TaskManifest (manifest.json) live in context.py despite being persisted-file schemas, not context state. That's the same group as the deferred run-level "Unified output file envelope" (TODO §1.8) — the right place to move them is alongside that work, gated on sieval analyze, not mixed into this protocol PR. TaskStageMeta (→ context) and ModelCallMeta (→ models) are cohesive where they are, as you noted.

Re-verified after the amend: 2792 tests pass; lint/type/preflight/index-sync clean; end-to-end through the real runner confirms the on-disk record above. Green CI to follow.

[ethan-scitix]

Applied the builder-naming fix; the judge/grade question I'll answer rather than change.

Builder naming → build_*, matching build_model_call_meta / build_stage_meta

You're right, and I checked it's the actual convention, not a two-sample guess. The repo has two factory styles that split by purpose:

• build_* (7) — construct from parts: build_stage_meta, build_model_call_meta, build_prompt
• *_from_* (11) — convert from a source: task_meta_from_dict, dataset_meta_from_dict

My builders construct records from parts, so they're the build_* family — same category as build_model_call_meta (which builds the ModelCallMeta TypedDict from a ModelOutput). Renamed across all 14 files:

• prompt_record → build_prompt_record
• prediction_record → build_prediction_record
• rollout_judgement → build_rollout_judgement
• judgement_record → build_judgement_record

Types stay PascalCase (JudgementRecord, RolloutJudgement), exactly the ModelCallMeta / build_model_call_meta split. is_prediction_record / is_judgement_record keep the is_ predicate convention. Whole-word rename, is_*_record verified untouched; 2792 tests pass, folded into the protocol commit.

judge vs grade — they denote different layers, and within this PR they're already separated correctly

They aren't two words for one thing:

concept	word	scope
the verdict record (feedback output)	judgement	protocol, mechanism-agnostic — string-compare (mmlu), math-verify (aime), test-suite (lcb), or LLM
the LLM autorater's categorical output	grade (CORRECT/INCORRECT/NOT_ATTEMPTED)	LLM-judged tasks only
the LLM actor	grader	LLM-judged tasks only

A JudgementRecord contains a grade for LLM tasks — one nests in the other. A math task produces a judgement with no grade and no grader anywhere, which is exactly why the protocol layer uses the mechanism-neutral word. Within this PR the usage is already consistent: judgement/build_rollout_judgement for the record, grade for the LLM output, grader for the actor.

The genuine inconsistency is a third word — HLE's judge (parse_judge, judge_parsed, JudgeFeedback) used where the other three say grader/grade. That's HLE-internal, out of this PR, and not a blind rename: parse_judge returns (correct, confidence, parsed), a different contract from parse_grade's grade string, so it can't just take that name. It's a scoped follow-up on the judge family, tracked with their protocol migration.

One deliberate call worth surfacing: I kept the protocol record as Judgement rather than renaming it to Verdict. "Verdict" would remove even the shared root with HLE's "judge", but "judgement" is standard for a mechanism-agnostic decision (and it's what the eval field means by "LLM-as-a-judge"), so I didn't think the churn earned it. Say the word if you'd rather have the zero-overlap Verdict.

Force-pushed; CI to follow.

[ethan-scitix]

Swept for more of the same, and answered the judgement/feedback question.

Q2 — judgement vs feedback: different axes, not synonyms

feedback is the stage — the pipeline method that returns (finalize, payload): it decides finalize-vs-iterate and emits the verdict. judgement is the content of that payload — the verdict itself.

The records are named by content, not by stage, and that's systematic:

stage (method / *_result field)	record type (content)
preprocess / preprocess_result	PromptRecord
infer / infer_result	ModelOutput
postprocess / postprocess_result	PredictionRecord
feedback / feedback_result	JudgementRecord

So feedback → JudgementRecord is the same choice as postprocess → PredictionRecord and preprocess → PromptRecord: the stage name describes the pipeline step, the record name describes the data. "PromptRecord" reads better than "PreprocessRecord" on a shard line, and "JudgementRecord" better than "FeedbackRecord" — a reader sees what it holds, not which method emitted it. A JudgementRecord also nests a grade for LLM tasks, so judgement (the verdict record) genuinely sits a layer above grade (the LLM output) and feedback (the stage).

Q1 — one real inconsistency, plus wording nits. Fixed, folded into the protocol commit.

Real, with teeth: is_prediction_record and is_judgement_record had byte-identical bodies ("rollouts" in value), so is_judgement_record(a_prediction) returned True — the names promised a discrimination the code didn't do. And is_judgement_record had no production caller (only is_prediction_record is used, in anomaly.py), so the bug was latent. Fixed to key on n_rollouts, which build_judgement_record always materializes (even at zero rollouts) and a prediction never has:

• is_judgement_record(v) → "n_rollouts" in v
• is_prediction_record(v) → "rollouts" in v and "n_rollouts" not in v

Now mutually exclusive. Added test_sniffs_discriminate_prediction_from_judgement (fails on the old identical bodies) and extended the empty-record test to assert an empty judgement isn't mistaken for a prediction. No behavior change at the one real call site — anomaly.py only ever passes postprocess_result, always a prediction.

Wording, same judge/grade axis as before: RolloutJudgement.extra / JudgementRecord.extra docstrings said "grader-specific detail" / "sample-level grader detail" — but these types are mechanism-agnostic, and a string-compare (mmlu) or math-verify (aime) verdict has no grader. Reworded to "verdict-mechanism-specific" and named the mechanisms (LLM grader / code runner / constraint checker).

Checked and left as-is: the builder asymmetry — build_rollout_judgement exists but there's no build_rollout_prediction — is intentional, not a slip. Prediction rollout items are trivial (wrap a value, derive extracted), so build_prediction_record([...]) builds them internally; judgement items carry caller-decided correct/score/extra, so they get an item builder. Different data, different ergonomics.

2793 tests pass; lint/type clean. Force-pushed; CI to follow.

[ethan-scitix]

Added the vocabulary to sieval/tasks/CLAUDE.md — a new Stage-Output Protocol (opt-in) section, folded into the protocol commit.

Placement rationale: it's the task-author guide, which is who migrates the next task and who writes grade/grader/extra. Kept to the doc-layering already in use — records.py docstrings stay the authoritative type/schema source, the CLAUDE.md section carries only the vocabulary + conventions and points to records.py. It captures the decisions this thread settled that no single code file makes obvious:

• Stage → record map, and the "records named by content, not stage" choice (feedback → JudgementRecord, not FeedbackRecord).
• The layered vocabulary: judgement (verdict record, mechanism-agnostic) vs grade (LLM categorical output, contained by a judgement) vs grader (LLM actor) — plus a one-liner that judge is HLE-only upstream terminology not to be introduced in new tasks.
• Conventions: build_* constructors (matching build_model_call_meta), is_* sniffs, records returned bare, and grader ModelOutput persisted whole via obj_to_dict(out, add_type=False).

No enforcer added — this is descriptive convention (like the existing ppl/clp explanation in the same file), not a checkable rule, so per the engineering-infra coherence rule there's no preflight/pre-commit/CONTRIBUTING coupling to wire. markdownlint passes.

Three commits still; force-pushed.