feat(core): persist task identity in run meta.json

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

A finished run recorded which sieval version produced it, but not which task or under what evaluation protocol it was scored. @sieval_task already builds a full TaskMeta; it just never left the process — reaching only the in-process registry and the static sieval/meta/index.json catalog. Consumers recovered question type and scoring mode by parsing directory names, which fails silently, and that name comes from Task.name — a user-chosen YAML key with no guarantee of matching the registered one.

• meta.json now carries an optional task block, read off the instance via get_task_run_identity(task) rather than looked up by name, so it works identically for plugin tasks — which never appear in index.json at all, making persistence their only channel. Additive on disk; absent means pre-feature run or undecorated class, never backfilled.
• n_shot is the run's, not the class's. Every other field is the task's declaration and matches its catalog row. @sieval_task seeds the declared count onto Task.n_shot, beside cls.tags / cls.model_type; the 15 knob-bearing tasks assign self.n_shot in __init__, shadowing it for that instance, so args: {n_shot: 3} stops recording 5. A task with no shot knob needs no code of its own to be correct, and "n_shot" in task.__dict__ still separates an override from a declaration after the fact.
• The knob is spelled n_shot everywhere now, and a preflight check keeps it that way. Recording it exposed that k meant two unrelated things in the task tree — a few-shot count in 13 tasks, the k of pass@k in 13 others (the metric's own parameter, not the sampling budget; that is n, and k <= n) — the same silent convention this PR exists to remove, one layer down. New check_task_shot_knobs (AST-only). Breaking for YAML args:.
• A resume by a different task is refused, not just noticed. auto_resume short-circuits on a valid report.json before any stage runs, so a task pointed at another task's directory was handed that report as its own result, having evaluated nothing. Neither existing guard caught it. Breaking.
• Four tasks no longer render fewer shots than they record. MMLU/CMMLU silently capped against a short subject pool — 5 shots for a 5-row subject at n_shot=8, 0 for a subject absent from dev, while the recorded count said 8. DROP/OpenBookQA reached the same drift by a different mechanism, and so were missed by the sweep that found the first two: they draw through Dataset.retrieve_samples, which truncates to the split length (k = min(k, len(ds))) and returns [] for a missing split, so DROP at n_shot=3 against a 1-row train rendered 1 shot and recorded 3. All four now abort in setup(), before any inference spend, like gsm8k/hellaswag/arc. Breaking.
• The ARC family moves to sieval/tasks/arc/, which turned out to be load-bearing: it exposed that get_task_class() could not resolve a subpackage-hosted task at all, so all four ARC tasks would have raised KeyError from sieval task show. Now falls back to scanning subpackages.

{
  "version": "0.7.0",
  "deterministic": true,
  "task": {
    "name": "cmmlu_kshot_clp",
    "display_name": "CMMLU (few-shot, base CLP)",
    "dataset": "cmmlu",
    "eval_mode": "clp",
    "n_shot": 5,
    "tags": ["chinese", "multiple-choice", "base-model"],
    "status": "stable"
  }
}

Decisions, with the rationale at the code

Each of these is argued where it binds — the docstring or rule that a later change would have to edit — rather than only here, where nobody re-reads it:

• No backfill into an existing meta.json (TaskRunMeta) — it would stamp the resuming process's identity onto samples another task produced. Absent is honest; wrong is worse.
• Identity is not inherited (get_task_run_identity) — reads cls.__dict__, not the MRO like get_task_meta. An undecorated subclass is a different task. No task-to-task subclassing exists in-tree, so this only bounds plugins.
• Deliberately excluded fields (get_task_run_identity) — model_type, reference_impl, description, deps_group. The block is a chosen subset, not task_meta_to_dict(). tags is persisted but advisory: the one field whose values aren't frozen within schema_version=1.
• Shot count is declared, never inferred (Task.n_shot) — before the rename self._k held a few-shot count in *_kshot_* tasks and pass@k's k in several *_0shot_* ones, so any name-based rule would have recorded that metric parameter as shots on exactly the tasks declaring n_shot=0. Task filenames are no better a proxy: openbookqa_kshot_gen declares n_shot=0.
• n_shot is a plain class attribute, not a ClassVar (Task.n_shot) — tags and model_type are ClassVar because nothing ever assigns them per instance. n_shot is the one field a run changes, and ClassVar makes the shadowing assignment a type error under ty ("Cannot assign to ClassVar n_shot from an instance"). Not a property either: there is no second source to derive it from — the decorator assigns after the class body, so a class-body n_shot cannot advertise a count different from its declaration. Pinned by a test.
• The gate compares name only (gate_resume_identity) — a wider comparison would refuse a task merely redefined between runs, which is the version gate's job. n_shot is excluded for a different reason, now written down at the gate: a same-task resume under a different n_shot is a changed invocation, which the CLI's strict --resume config match already refuses by comparing the persisted YAML body, tasks.*.args included, before a runner is built. What is left for this gate is the mismatch that match cannot see at all. It passes when either side has no block, so it can never reject a pre-feature resume.
• No other two-meaning name in the repo — swept every @sieval_task __init__ by AST: k (26), n (17), fewshot_split (10), then singletons. n is the sampling count in all 17, matching the OpenAI field. k was the only collision, split exactly 13/13.
• Envelope scope and subtask granularity — out of scope, per the issue.

Related Issues

Closes #49
Refs #25 — the task-side capability declaration will extend @sieval_task, touching the same metadata surface. This PR deliberately does not persist model_type for that reason.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check on sieval/ scripts/ tests/; mypy diffed before/after on touched files — no new errors)
• Unit tests pass — full CI scope pytest tests/unit tests/integration tests/acceptance → 2959 passed, re-run after rebasing onto main @ 66d00482 (on top of feat(tasks): finish the stage-output protocol migration (33 tasks, 40/40) #62 and chore(deps): revert the unrequested lock drift from d805418a #63). All 15 commit subjects preserved; this branch touches neither pdm.lock nor pyproject.toml.
  feat(tasks): finish the stage-output protocol migration (33 tasks, 40/40) #62 migrated DROP and OpenBookQA to the stage-output protocol, which overlaps this branch two ways. Two commits conflicted (openbookqa_kshot_gen.py, and the HellaSwag/TheoremQA tests) — resolved as orthogonal overlays: main's record protocol plus this branch's k → n_shot rename, neither reverted. Four further sites were conflict-free but semantically broken: preprocess now returns a PromptRecord, so this branch's new pool-guard tests had to read pre["prompt"][0]["content"], not pre[0]["content"] (KeyError: 0 — caught by running the full suite, not by the rebase). Those four are folded into the commit that introduced the tests rather than appended as a fixup, so every commit from that point to HEAD still passes its own suite (verified per commit).
  Stated plainly because it bounds the claim above: the local run used the pre-chore(deps): revert the unrequested lock drift from d805418a #63 dependency set, since chore(deps): revert the unrequested lock drift from d805418a #63 reverts 71 packages, 16 across a major (datasets 5.0.0 → 4.4.1, transformers 5.14.1 → 4.57.3, openai 2.45.0 → 2.9.0, mypy 2.3.0 → 1.19.1, torch 2.13.0 → 2.9.1). The mypy diff above also predates that revert. CI's clean --frozen-lockfile install on 3.12 and 3.13 is the authority for the reverted set; a shared venv synced to the drifted lock structurally cannot check it, which is the trap chore(deps): revert the unrequested lock drift from d805418a #63 itself documents.
• scripts/check_preflight.py 20/20 PASS, including the new check_task_shot_knobs ("all 36 task constructor(s) spell and wire the shot knob correctly") and check_meta_index_sync — index.json is touched: the rename edits six reference_impl.notes strings embedded in the catalog, regenerated via scripts/sync_meta_index.py in the same commit. No TaskMeta field changes. scripts/sync_package_stubs.py --check, check_layer_imports.py, and scripts/sanitize.sh clean.

New coverage — the regression-critical files first:

File	What it pins
tests/unit/core/runners/test_runner.py	E2E arun() persists registry identity while Task.name differs; a run rendering 7 shots against a class declaring 2 persists 7. TestResumeIdentityGate: refuses a directory another task produced (verified to fail with the gate call removed), allows same-task / pre-feature / undecorated / malformed blocks
tests/unit/core/tasks/test_meta.py	exact projected subset, excluded keys asserted absent, JSON-shaped output, descriptive vs synthesized tags, None for an undecorated subclass with get_task_meta still MRO-reading, n_shot following the instance, class-argument raising. Plus: a class-body n_shot cannot shadow the declaration (the decorator assigns last), and the subpackage fallback with the flat path asserted to genuinely miss first, so it can't pass vacuously
tests/unit/scripts/test_check_preflight.py	27 tests in TestCheckTaskShotKnobs — one per rule and exemption: every misspelling flagged, fewshot_* not flagged, n_shot = len(...) allowed, the knob fed from k (the regression the check exists for), subpackage tasks scanned, plus a live-repo assertion. Both closed holes are pinned: a pass@ mention in a class or method docstring no longer buys an exemption, and a decorated task inheriting __init__ from an undecorated base is now checked at the base
tests/unit/tasks/test_drop_kshot_gen.py, test_openbookqa_kshot_gen.py	20 tests: the pool aborts (split shorter than n_shot, split missing entirely, negative n_shot) fire at setup(), not mid-run; DROP's lazy drop_eval import discipline preserved — the guards land before it, so they cost no optional dependency; and prompts byte-identical with and without setup(), which is what makes the preprocess → setup() hoist score-neutral
tests/unit/core/tasks/test_saver.py, test_context.py	identity on disk, key absent (not null) when undecorated, no backfill, task genuinely optional
tests/unit/tasks/test_mmlu_kshot_clp.py, test_cmmlu_kshot_clp.py	the short-pool abort at setup() (not mid-run), and the residual setup() structurally cannot see — a subject absent from dev entirely, which still surfaces per sample

Every fix in the review round was mutation-checked: revert the fix, the new test fails.

Manual

• Swept all 40 registered tasks: every projected block matches its index.json row field-for-field. Both directions of the shot hook — all 15 knob-bearing tasks report the constructor value while the class declaration stays intact, all 11 pass@k-only tasks project their declaration with no k leaking in as shots. Re-run unchanged after n_shot_used collapsed into Task.n_shot, which is what establishes that refactor as behaviour-preserving.
• Audited every k in tracked YAML and docs before renaming, since task kwargs are user-facing. Exactly two were shot counts (leaderboards/sft_fast_202511.yaml, docs/guide/configuration.md) and are updated; the four in examples/leaderboard-math-sft.yaml are pass@k and deliberately left. Prose kept where k isn't this knob (top-k, k-shot, upstream dev_df[:k]).
• Drove the live cmmlu_kshot_clp through get_task_run_identity + write_run_meta into a temp dir: block matches its catalog row, and a second write leaves the file byte-identical.
• Proved the silent cap was real before fixing it, at n_shot=8 against the live MMLU/CMMLU classes. The follow-up sweep of the six per-subject-pool tasks confirmed those two were the only ones of that mechanism not already fail-closed — it did not cover retrieve_samples-drawn pools, which is how DROP/OpenBookQA were missed and then found separately. Both reproduced live (DROP n_shot=3 on a 1-row train → 1 shot, recorded 3; OpenBookQA n_shot=5 on 2 rows → 2, recorded 5) before fixing.
• Rendered DROP prompts at n_shot 0/1/3/8 before and after moving the few-shot draw out of preprocess: byte-identical, so no score moves. The draw was seeded, so it had been re-shuffling the whole train split once per eval sample to return the same set.
• Verified the ARC move end to end: all four names resolve to sieval.tasks.arc.* from a cold cache, flat tasks still resolve in one import, sieval task show arc_easy_kshot_ppl renders. check_task_shot_knobs held at 36 tasks across the move, which is what proves its recursive scan works on real data.
• Both preflight holes were found by mutating the live repo against the check, not by reading it.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info (scripts/sanitize.sh passes)
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — the only new files are the two empty arc/__init__.py markers, which carry no docstring by convention and no code to attribute; every ARC module keeps its attribution through the move
• No new upper-layer dependencies added to core/ — new imports are core.runners → core.tasks.meta / core.tasks.context, alongside the core.tasks.* imports runner.py already had. The subpackage scan adds no static edge: it goes through pkgutil.iter_modules on an already-dynamically-imported sieval.tasks, the same call-time indirection import_all_tasks() uses
• Deleted code verified — the only removals are branch-local: Task.n_shot_used and the preflight helper that read it, both introduced earlier in this same branch. Swept afterwards: n_shot_used appears nowhere in tracked files

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior

Four parts; the first is additive, the other three break.

On-disk schema — additive. task is NotRequired; readers ignoring unknown keys are unaffected and the version handshake is untouched. Pre-existing runs never gain the block, so consumers must treat it as optional indefinitely — the documented contract, not an oversight.

Resume behaviour — breaking. Resuming into a directory produced by a different registered task raises ResumeIdentityError. What it replaces was not a working workflow: that resume returned the first task's report as the second's result with nothing evaluated. Recovery is the documented pair, no escape hatch — remove the result_dir and start fresh, or give the task its own. Fires only when both sides carry a block, so it can never reject a pre-feature resume.

Constructor + YAML args: — breaking. k → n_shot on 13 tasks: arc_challenge/arc_easy (×clp,ppl, via arc/_base.py), c_eval, cmmlu, drop, gsm8k-base, hellaswag, mmlu, mmmlu, openbookqa, theoremqa-base. args: {k: N} becomes args: {n_shot: N}; the constructor raises TypeError on the old spelling, so this breaks loudly at startup, never as a wrong score. The 13 pass@k tasks keep k. In-tree configs updated in the same commit.

Few-shot pool — breaking, four tasks. All four now refuse a pool that cannot supply the count they will record, at setup(), before any inference spend. Recovery in every case: ask for a count the split can supply.

• MMLU/CMMLU abort when any subject in the few-shot split holds fewer than n_shot exemplars. Only reachable with n_shot > 5, since both shipped dev splits are a uniform 5 rows/subject — no upstream-faithful configuration hits it. Sweeping the whole dev pool rather than only the evaluated subjects is deliberate and inherited from c_eval — stricter than strictly necessary, and unreachable at 5/subject regardless. Residual, documented at the code: a subject present in test but absent from dev has no prefix to build in setup(), so that case still surfaces per sample.
• DROP/OpenBookQA abort when the drawn split is shorter than n_shot, or missing entirely; DROP additionally rejects a negative n_shot, which used to render 0 shots and record the negative value. Reachable through supported config, not only hand-construction: a dataset operations: slice takes a split, so - slice: {num: 2, split: train} shrinks the pool while args: {n_shot: 3} stands. Defaults are safe. DROP's draw also moves from preprocess to setup() — prompts verified byte-identical, preprocess keeps a lazy fallback for callers that skip setup().

Minor API break for direct callers: get_task_run_identity takes a Task instance, not a class (raising TypeError rather than silently reporting "undecorated").

Task.n_shot, no released break. The field started this branch as Task.n_shot_used and was collapsed into a public Task.n_shot before merge, so no release ever carried the old name. Stated for anyone building on the branch, because the failure mode is silent: a task still assigning self.n_shot_used now writes an attribute nobody reads, and meta.json records the declared default as though the run had used it. Assign self.n_shot instead. check_task_shot_knobs catches this for in-tree tasks; it scans sieval/tasks/ only, so an out-of-tree plugin gets no warning.

[ethan-scitix]

Self-review follow-up — three commits addressing five findings, ce8ba123..13bb0159.

bb738fe3 — fix(tasks)!: DROP / OpenBookQA fail closed on a short few-shot pool

The same drift the MMLU/CMMLU commit removes, reached by a mechanism that commit's sweep did not cover: it swept the six per-subject-pool tasks, while these two draw through Dataset.retrieve_samples, which truncates (k = min(k, len(ds))) and returns [] for a missing split.

	shots rendered	meta.json recorded
DROP, 1-row train, n_shot=3	1	3
OpenBookQA, 2-row train, n_shot=5	2	5

Exactly the disagreement n_shot_used exists to remove, one task family over. Reachable through supported config rather than only hand-construction: a dataset operations: slice takes a split, so - slice: {num: 2, split: train} shrinks the few-shot pool while args: {n_shot: 3} stands. Stock configs are safe — DROP's train is ~77k rows and OpenBookQA defaults to n_shot=0.

Both now raise in setup(), before any inference spend, matching gsm8k / hellaswag / arc. DROP additionally rejects a negative n_shot, which previously rendered 0 shots and recorded the negative value.

DROP's exemplar draw also moves from preprocess to setup(): it was re-shuffling the whole train split once per eval sample. The draw is seeded (42), so every per-sample call already returned the same set — prompts verified byte-identical against the pre-change rendering at n_shot 0/1/3/8, so no score moves. preprocess keeps a lazy fallback for callers that skip setup(), and both guards sit after the n_shot == 0 early return and before the lazy drop_eval import, so they cost no optional dependency.

aae59050 — fix(scripts): two coverage holes in check_task_shot_knobs

Both found by mutating the live repo against the new check, and both let through the regression it exists to catch.

1. Rule 3 accepted a docstring as evidence. ast.walk reaches docstring Constants, so a class documenting "pass@1" while computing nothing of the kind flipped a bogus k parameter back to PASS — and k matches no shot-count spelling, so rules 1 and 2 do not catch it either. Class and method docstrings are now excluded; real metric keys (f"pass@{k}" in a report dict) still count.
2. A decorated task inheriting its __init__ was skipped at both ends — the subclass declares no __init__, the base carries no decorator. A knob-bearing constructor then went unchecked with only a silently lower count as a symptom, and nothing asserts that count. This PR introduces both the arc/_base.py shared-base layout and the CLAUDE.md rule counting a shared module toward the ≥5-file threshold, so the pattern is now encouraged. The n_shot rules now bind every constructor under sieval/tasks/; the k rule stays decorated-classes-only, since an undecorated base's pass@k is normally computed by its subclass and judging it from the base's body would be a false positive.

The live repo still PASSes at the same 36 constructors. Two existing tests asserted the old scope — test_undecorated_class_ignored and test_inherited_init_ignored, whose stated expectation ("a subclass with no __init__ of its own has no knob to check") is hole 2 — so they are rewritten to the new contract rather than adjusted around it.

13bb0159 — docs(core,tasks): bound two claims the code makes about its own reach

No behaviour change; both remove a reading a later change would lean on.

• gate_resume_identity justified comparing name only with "a wider comparison would refuse a task merely redefined between runs". Sound for the declaration fields, but n_shot escapes that argument — it is the run's value, and the reason the block records it at all. The docstring now says why it is still excluded: a same-task resume under a different n_shot is a changed invocation, which the CLI's strict --resume config match already refuses by comparing the persisted YAML body, tasks.*.args included, before a runner is built. What is left for this gate is the mismatch that match cannot see at all.
• MMLU/CMMLU setup() claimed a short pool aborts "before any inference spend". True only for subjects the few-shot split contains — one present in test but absent from dev has no prefix to build there, so it still surfaces per sample. The residual is noted in the PR body; the comment is where a reader looks.

Verification

• 2879 → 2896 tests (DROP +7, OpenBookQA +4, preflight +6). Every new test mutation-checked: reverting the corresponding fix fails it, confirmed for all four fixes.
• scripts/check_preflight.py 20/20 PASS, including check_task_shot_knobs and check_meta_index_sync.
• ruff check + ruff format --check, ty check, scripts/sanitize.sh, scripts/check_layer_imports.py, sync_meta_index.py --check, sync_package_stubs.py --check all clean.
• CI green on 3.12 and 3.13.

Body still needs one Breaking Change line

Not yet covered above the fold; suggested entry:

DROP / OpenBookQA few-shot pool — breaking. Both now abort at setup() when the few-shot split holds fewer than n_shot exemplars, instead of silently rendering fewer shots than n_shot_used records. Only reachable when the pool is shrunk below the requested count — e.g. a dataset slice operation on the few-shot split — so no stock configuration hits it. DROP also now rejects a negative n_shot, which previously rendered 0 shots and recorded the negative value. Recovery: ask for a count the split can supply.