feat(platinum): add PlatinumBench dataset + 5 math 0-shot generation tasks

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

• Integrates PlatinumBench (MadryLab/platinum-benchmarks, data madrylab/platinum-bench) — benchmarks whose labels were re-verified so that a remaining failure is a real model failure, not label noise. Scope is the 5 math subsets (gsm8k, svamp, multiarith, singleop, singleq); the other 9 configs load but ship no task.
• One dataset, one task subpackage. PlatinumBenchDataset merges all 14 non-vqa configs into one test split and stamps subset on every row; the caller narrows to one subset with a filter operation (see Selecting a subset). sieval/tasks/platinum_bench/ holds 5 leaves that are 2-line subclasses of one _base.py — same layout as arc/ (feat(core): persist task identity in run meta.json #57). The dataset FK resolves off the base's sample generic through the MRO, so the leaves declare no generic of their own.
• Rejected rows are dropped, which is the point of the benchmark: cleaning_status == "rejected" marks a question the authors found unanswerable/mislabeled. 953 of 1042 rows kept (gsm8k 268/300, svamp 265/300, multiarith 170/174, singleop 150/159, singleq 100/109).
• Prompt, extraction and scoring all follow upstream, not sieval's native math path. The prompts are carried in the data (platinum_prompt / platinum_prompt_no_cot); upstream's get_parse_fn + check_prediction (exact float equality on platinum_target[0]) are vendored byte-faithfully into sieval/community/platinum_bench.py.
• status="stable" on all five, earned by an exact reproduction: replaying upstream's own published inferences (madrylab/platinum-bench-paper-cache) through this pipeline reproduces all 120 math error counts of the paper's Table 3 — 24 models × 5 subsets, every cell. Independently, three live runs on two published checkpoints (Qwen2.5-72B-Instruct and Llama-3.3-70B-Instruct, 968 rows each, on upstream's own serving provider) land within 0.7 σ_D of their Table 3 rows — 8 vs 11, 16 vs 14, 15 vs 14 — confirming the numbers survive real sampling over the wire. Tables below.
• Three prompt variants, not two. prompt_variant now takes cot / no_cot / no_cot_o1. Upstream's platinum_prompt_no_cot column carries a dangling "Then, provide" (a leftover conjunction from the CoT wording) and upstream rewrites it to "Provide" for its o1 snapshots only — not for o1-mini / o3-mini, which ran against the unedited string. Both branches are needed, and having them takes the port from 21 to 24 of 24 published rows.
• No new dependencies, no examples/ YAML, no changes to any existing task. One additive core/ change: Dataset.filter, plus its registration as a YAML operation (see below). The get_task_class() subpackage-lookup fix these tasks need already landed in feat(core): persist task identity in run meta.json #57.

Selecting a subset

A HuggingFace config is a load-time choice, so the loader reads all 14 non-vqa configs and the caller narrows:

datasets:
  platinum_gsm8k:
    class: PlatinumBenchDataset
    path: madrylab/platinum-bench
    operations:
      - filter: {by: subset, value: gsm8k}

This replaced an earlier required subset= constructor argument — see Review Fixes for why and what was re-verified. Omit the operation, or name the wrong subset, and setup() fails before any tokens are spent.

Related Issues

Refs #57 (the arc/ subpackage layout and the get_task_class() subpackage lookup this builds on).

Test Plan

Automated

• Lint/format clean (ruff check + ruff format --check: all checks passed, 386 files formatted)
• Type check clean (ty check: all checks passed). mypy --strict reports 29 diagnostics of 4 classes (no-untyped-def on stage methods, no-untyped-call into the vendor, union-attr on upstream's unguarded re.search(...).group()); the existing gsm8k dataset+task pair reports 37 of the same classes, and [tool.mypy] excludes sieval/community — so this is baseline, not new.
• Unit tests pass: 3067 passed (24 community, 14 dataset, 60 task, 12 Dataset.filter, plus the operation's two CLI whitelists)
• Integration + acceptance: 64 passed
• Full preflight (scripts/check_preflight.py): all PASS, including check_meta_index_sync, check_tasks (45 tasks), check_datasets (70 exports, all sources pinned), check_imports, check_task_shot_knobs

New tests deliberately pin the failure modes that would otherwise be silent:

• Every one of the 5 subset names is on upstream's math_datasets list — a misspelled subset silently degrades scoring from float equality to string membership, and "42" in ["42"] still passes, so nothing else would notice.
• Each leaf's vars() is exactly {subset, tags, model_type, n_shot} (the last three seeded by @sieval_task), so a leaf can never grow behaviour that drifts from its four siblings.
• Row counts are asserted three ways (kept, total, total - kept) so a stale number cannot agree with itself.
• prediction=None and the absent prediction key are covered separately, because obj_to_dict drops None keys on the disk round-trip.
• The vendor's two upstream quirks are pinned as behaviour the task layer absorbs: AttributeError on digit-free output, and the dead prediction != 'Parsing error' guard that lets float('parsing error') raise ValueError.

Manual

• sieval dataset download platinum_bench succeeds (17 files at the pinned revision).
• Real-data load of all 5 subsets: kept-row counts exactly as claimed (953 total); every kept row has platinum_parsing_strategy == "math"; surviving statuses are only consensus/verified/revised; the selected columns are exactly the 6 shared ones plus the stamped subset; and all 953 gold answers round-trip through upstream's parse + check_prediction (953/953), so the vendored extractor and the pinned data agree.
• sieval task show platinum_gsm8k_0shot_gen resolves by name (the subpackage-hosted lookup path) and renders the full reference-impl notes.
• Score comparison: all 120 cells of Table 3's math block reproduce exactly. See the table below.
• Live end-to-end run against a real endpoint — all 5 subsets, 953 questions, Qwen/Qwen3-32B over an OpenAI-protocol endpoint, temperature=0.5, one sample, driven through sieval eval + YAML. 0 pipeline failures, 6 errors: singleq 0/100, multiarith 1/170, singleop 1/150, gsm8k 2/268, svamp 2/265. Persisted request_params confirm max_tokens=6000 and temperature=0.5 actually reach the wire. (That run predates the review fix below, which moved the budget out of the task; the request it sent is byte-identical either way, since the value is the same 6000 — only its source changed, from infer() to the model config.) Five of the six errors are ordinary wrong answers with a clean Answer: line.
  • The sixth is a finding, and it is now documented. Upstream sized max_tokens=6000 for non-thinking models. Qwen3-32B spent the entire budget inside the reasoning channel on one singleop question (finish_reason="length", 10k characters of "Wait, no, 4 + 7 is 11?") and returned an empty answer — which scores as an error indistinguishable from a wrong one unless you know to look. anomalies.json does distinguish it: that row is the only anomaly across all 953, flagged truncated_output + extraction_failure. The docstring and reference notes carry this as an infer prerequisite: set max_tokens=6000 to reproduce, raise it on a thinking model, and read errors alongside anomalies.json. Tests pin the wording.
  • This run alone cannot confirm a published number — Qwen3-32B is not a model upstream evaluated, so it pins the transport but has no number to be checked against. The run below closes that gap.
• Live runs on models upstream did publish, aligned against Table 3. Qwen2.5-72B-Instruct and meta-llama/Llama-3.3-70B-Instruct — byte-identical to the paper cache's model keys — all 5 subsets over the paper version's 968 questions, temperature=0.5, one sample. Upstream's model factory routes both of its open-weight rows through DeepInfra, so the runs pin that provider with fallbacks disabled and the pin lands in the persisted request_params; a third run on a bf16 endpoint brackets the serving-precision effect. 0 pipeline failures, full 968-row coverage, all three within 0.7 σ_D of their published rows, and the live misses overlap upstream's own while moving in both directions — numbers in Live alignment below.
  • Exact equality is not reachable and is not the claim: upstream samples at temperature=0.5 with no seed. The two checks answer different questions on purpose — the replay pins scoring fidelity against upstream's own inferences, a live run pins that the transport and stage plumbing work on real completions.

Score Comparison

Three checks, because none alone is sufficient: a replay of upstream's published inferences pins scoring fidelity against Table 3; live runs on two published models pin that the same numbers survive real sampling over the wire; and a live run on a thinking model exercises a decoding regime upstream never covered. All three are described below.

Why replay and not a fresh run

Upstream decodes at temperature=0.5, one sample, no seed — so a fresh run against any single model is unreproducible by construction, and the metric makes that fatal rather than merely annoying. Upstream reports raw error counts, 0–19 out of 100–274 rows; for GSM8K (n=274, p≈0.011) the binomial σ is ≈1.7 errors, so "our count is within ±2 of theirs" would be unfalsifiable noise dressed up as agreement.

Upstream also published the actual completions behind the paper's numbers (madrylab/platinum-bench-paper-cache, 15 pickles). Replaying those through this pipeline is deterministic, costs nothing, and is a strictly stronger check than any live run:

• The stub model looks up (prompt, temperature, 0, model) and raises on a miss. All 968 prompts × 25 cached models hit — which is what proves our prompt string is byte-identical to upstream's, not merely equivalent.
• Everything else is production code: the dataset's rejected-row filter, the leaf task, preprocess/infer/postprocess/feedback/report, and the on-disk record round-trip TaskRunner drives. fails == 0 on every run.
• Run against madrylab/platinum-bench-paper-version (the revision the paper's numbers are computed on), which keeps 968 math rows.

Table 3, math block — errors per subset (lower is better)

# Platinum Questions: SingleOp 150 · SingleEq 100 · MultiArith 171 · SVAMP 273 · GSM8K 274.

Model	variant	SingleOp	SingleEq	MultiArith	SVAMP	GSM8K
o1-2024-12-17 (high)	no_cot_o1 @ 1	0	0	0	0	2
Claude 3.5 Sonnet (Oct)		0	0	0	1	3
o1-2024-12-17 (med)	no_cot_o1 @ 1	0	0	0	1	2
DeepSeek-R1	no_cot	0	0	1	1	1
Claude 3.5 Sonnet (June)		0	0	0	2	5
Llama 3.1 405B Inst		0	0	0	3	2
GPT-4o (Aug)		0	0	0	7	4
GPT-4o (Nov)		0	0	0	6	7
o1-preview	no_cot_o1 @ 1	0	0	0	1	2
DeepSeek-V3		1	0	0	3	3
o1-mini	no_cot @ 1	1	0	1	1	2
Gemini Thinking (12/19)	no_cot	0	0	1	0	4
Qwen 2.5 72B Inst		0	0	0	4	7
o3-mini-2025-01-31 (high)	no_cot @ 1	0	0	2	1	1
Grok 2		1	0	0	4	3
Mistral Large		0	0	0	7	3
Gemini 2.0 Flash		0	0	1	4	8
Llama 3.3 70B Inst		0	0	0	7	7
Llama 3.1 70B Inst		2	0	0	7	7
Gemini 1.5 Pro		0	0	1	6	6
GPT-4o mini		0	1	1	6	6
Claude 3.5 Haiku		0	0	1	8	10
Gemini 1.5 Flash		0	1	0	13	11
Mistral Small		1	0	0	11	19

Every cell above is both ours and the paper's — rows compared: 24, exact: 24, mismatched: 0; cells compared: 120, exact: 120. Blank variant = the default cot @ 0.5. The cache holds a 25th model (gemini-2.0-flash-thinking-01-21) that Table 3 does not print; it replays cleanly too but has nothing to be compared against.

Live alignment on published models

The replay stubs the model layer, so it cannot catch a defect that only appears with real sampling over the wire. Two of the checkpoints Table 3 scores are still served, so both were run live on the paper version's 968 rows at temperature=0.5, one sample.

Which provider is not a detail. An aggregator routes across providers that quantize differently, and quantization changes completions, so every run pins its provider with allow_fallbacks: false — a wrong-silicon serve fails instead of silently routing, and the pin lands in the persisted request_params where it stays auditable. The first run picked a bf16 endpoint on the reasoning that released precision is the right standard. Upstream's own model factory says otherwise: both open-weight rows in Table 3 were served through DeepInfra. So both models were run there, and the bf16 run is kept alongside rather than discarded — together they bracket the serving-precision effect instead of hiding it.

model	provider	singleop	singleq	multiarith	svamp	gsm8k	total	Table 3	Δ	Δ/σ_D
Qwen2.5-72B-Instruct	DeepInfra (fp8)	0	0	0	4	4	8	11	−3	0.69
Llama-3.3-70B-Instruct	DeepInfra (fp8)	1	0	0	6	9	16	14	+2	0.37
Llama-3.3-70B-Instruct	Crusoe (bf16)	0	0	0	8	7	15	14	+1	0.19

The yardstick is σ_D = √(paper + live) ≈ 4.4–5.5, not √paper. The paper's own count is a single unseeded draw at temperature 0.5, so the comparison is a difference of two noisy draws and its spread is the sum of both variances. This also settles whether more sampling would help: it would not. Repeats shrink only our side of the variance, and the published number's own σ ≈ 3.7 is an irreducible floor — 5 repeats would move σ_D from 5.5 only to ≈ 4.2. Buying a different published model adds information; re-drawing the same one does not.

Three of the five cells publish 0 errors (singleop, singleq, multiarith — 421 rows at a near-zero true error rate), so agreeing there is close to automatic and should not be counted as evidence. Only svamp and gsm8k carry signal.

Counts alone cannot tell sampling noise from a scoring difference — both look like "a different number". The identity of the missed questions can. Upstream's cache is keyed by prompt, so its own completion for every live row is directly retrievable:

run	live misses	also missed by upstream	live-only	paper-only
Qwen2.5-72B @ DeepInfra	8	7	1	4
Llama-3.3-70B @ DeepInfra	16	9	7	5
Llama-3.3-70B @ Crusoe	15	9	6	5

Misses overlap heavily and move in both directions in every run. That is the signature of resampling the same model; a scorer that had drifted looser or stricter would push one-sidedly. The strongest form of the argument: the same scorer applied to upstream's own completions reproduces Table 3 exactly (the 120/120 replay), and applied to fresh completions from the same model, provider and decoding it differs — so the delta lives entirely in the sampled text, not in the scoring. All 968 live prompts were verified byte-identical to the cache's keys.

Plumbing is clean in all three runs: 0 pipeline failures, full 968-row coverage, and max_tokens=6000 / temperature=0.5 / the provider pin on every persisted record. Every miss finished on a stop token with a parseable integer prediction, so none is a truncation or a parse artifact — precisely where a defect in the vendored extractor or the max_tokens budget would surface. The one non-saturated surprise, a singleop miss in the fp8 Llama run where Table 3 has 0, is a genuine arithmetic slip (4754 vs 4764), not a pipeline failure.

Scored on the shipped 953-row pin from the same records, no extra inference: Qwen 0/0/0/4/4 = 8, Llama@DeepInfra 1/0/0/5/9 = 15, Llama@Crusoe 0/0/0/7/6 = 13.

Exact equality is not achievable here and is not the bar: upstream samples at temperature 0.5 with no seed. The replay is what pins scoring fidelity; these runs pin the live path.

The one caveat, and why the pin does not change

The counts above belong to platinum-bench-paper-version (968 math rows). The shipped pin stays on the current madrylab/platinum-bench, matching upstream's own default, which rejects 15 more math rows — 953 total (multiarith 171→170, svamp 273→265, gsm8k 274→268). Raw row counts are identical in both repos, so the delta is purely additional rejected labels from later cleaning. Upstream publishes no table for the current revision, so its scores will land near Table 3 but not on it. That is recorded in ReferenceImpl.notes (with the reproduction recipe) and pinned by a test, so nobody reads Table 3 as an exact target for the default pin.

Review Fixes

Round 1 — the token budget

Landed in refactor(platinum): hand the token budget back to the model layer. The reproduction evidence above is unchanged — re-running the paper-cache replay against the modified code returns the same Table 3 cells (30 spot-checked cells across all three prompt variants, 968 cache hits per model, 0 fails), because the budget never reached the replay's stub model in the first place.

max_tokens is no longer forwarded by the task. infer() used to pass max_tokens=6000 as a call-time kwarg. agenerate merges {**self._kwargs, **kwargs}, so that did not supply a default — it overrode the caller: a max_tokens set in models: / infer_args was silently discarded and 6000 went on the wire regardless. That is the one knob this benchmark most needs a caller to turn, since the score is budget-sensitive, and the docstring was telling readers to raise the budget for a thinking model without saying that the obvious place to raise it does nothing.

The max_tokens task arg and the UPSTREAM_MAX_TOKENS constant are gone; 6000 is documented as an infer prerequisite instead (the imo_answer_bench pattern). The task now injects no decoding params, matching gsm8k_0shot_gen / human_eval_0shot_base_gen / livecodebench_*. The tasks that legitimately do pass max_tokens are the ones where it is task-coupled — *_clp scoring a single next token, ruler reading a per-sample budget from the row.

⚠️ Behaviour change for callers: these five tasks now inherit whatever budget the model config supplies. Reproducing upstream requires max_tokens: 6000 on the model; omit it and you get the backend default, which can truncate CoT and under-score.

The test stub is why this was invisible. CapturingChatModel overrode _agenerate_impl, which sits below the kwargs merge, so last_kwargs recorded what the task passed rather than what the model would send — making every "the task injects no decoding params" assertion vacuous. It now records the merged request and accepts model-side kwargs, and a new test pins that a configured budget survives. Both infer tests fail against the previous code (verified by reverting).

Also in that commit:

• get_parse_fn("math") is resolved once at import instead of rebuilding the strategy table per sample.
• A comment records that PLATINUM_BENCH_REVISION is not forwarded to load_dataset — the pin is honoured by sieval dataset download staging, so passing the bare hub id gets whatever is current (same wording as _arc.py).
• A comment records that the dataset's categories / tags cover only the subsets that currently ship a task; one sample TypedDict is the FK for all 14 configs, so a future drop / squad leaf must widen them or render as ElementaryMath.

Re-verified after the change: ruff + ruff format, ty, package stubs, 3010 unit and 64 integration/acceptance tests, and full preflight all clean; sieval/meta/index.json regenerated for the notes edit.

Round 2 — the subset is selected, not constructed

Landed in refactor(platinum): select the subset with a Dataset.filter operation (plus a one-line comment fix). The reproduction is unchanged, and that was verified rather than argued — see the re-verification list below.

PlatinumBenchDataset no longer takes a subset= argument. It was the only custom __init__ among the repo's 37 dataset modules and the only required dataset constructor argument anywhere. The cost was a redundant knob: each leaf already pins subset: ClassVar[str] = "gsm8k", so args.subset was a second copy of a fact the class fixes, and setup() spent a runtime check confirming the two copies agreed. Mis-wiring was guarded, not unrepresentable.

Dataset.filter(by, value, *, split="test") -> Self is the transform that was missing. It joins repeat/slice/shuffle/stratified_sample and follows their conventions: a bare verb, by= spelled as in stratified_sample, an immutable clone via _clone_with_new_dict, self returned for an absent or empty split. It raises on an unknown column and — deliberately, unlike the size transforms — on zero matches, because an empty split is a misspelled value far more often than an intent and would otherwise surface as a run that silently scores nothing. value takes a scalar or a list; a string stays atomic rather than becoming a set of characters.

It does not subsume stratified_sample: filter drops non-matching groups, stratified_sample never drops a group. Opposite purposes, no overlap.

Filtering in the outer layer is what keeps this simple. Task.dataset is read-only, so a task cannot adopt a clone; narrowing before the dataset reaches the task means no Task.dataset setter and no task-side mutation.

load() merges the 14 non-vqa configs. They measurably share one HuggingFace feature signature across the six platinum columns, so they concatenate without a cast. Each config is loaded, cleaned and stamped in full isolation and only then concatenated, which makes "narrow back to one subset == the rows a single-config load produced, in order" structural rather than merely observed. vqa stays out: it misspells the column as platinum_parsing_stratagy and fails select_columns outright. PLATINUM_SUBSETS is a frozenset, so the merge sorts it — without that the row order would follow the string hash seed and sample ids would shift between processes.

setup() now checks the wired split via unique("subset"). A missing split, a different Dataset class and an empty split all collapse to [], so one comparison covers every way the wiring can be wrong, and the message names the filter operation that fixes it. A stale args.subset gets a migration error rather than load_dataset's opaque "unexpected keyword argument" — the same courtesy session.py extends to the select → slice rename.

filter is registered in both operation whitelists: the dispatch in session.py and _VALID_OPERATIONS in sieval/cli/validation.py. Only the first would have let --dry-run reject a config the run executes fine. Its value is required by presence, not truthiness, so value: 0 and value: false stay usable.

A corrected number. The loader docstring claimed 2752 merged rows; that was the paper version's total. Re-measured against the pinned snapshot it is 2725 kept of 3062 across the 14 configs — later cleaning rejects 27 more non-math rows. The five math subsets (953) were always right. Caught by re-verifying against the real pinned data rather than the local paper-version copy.

Re-verified after the change:

Check	Result
Row lists, old per-config path vs merge-then-filter, pinned revision	byte-identical, content and order — 268/265/170/150/100 = 953
Same, paper version	byte-identical — 274/273/171/150/100 = 968
Paper-cache replay re-run through the new code	120/120 cells, 24/24 models exact against Table 3
Shipped counts vs the pinned snapshot	2725 of 3062 merged; every math subset platinum_parsing_strategy == "math", no rejected survivors
Wiring smoke through the real session layer	filter → 268 rows, setup() passes; operation omitted → 2725 rows, setup() rejects; wrong subset → 265 rows, setup() rejects
Suites	3067 unit, 64 integration/acceptance
ruff / ty / full preflight / both sync scripts	all clean, all PASS

No CHANGELOG entry: the dataset and all five tasks are new in this same unmerged PR, so no shipped config breaks.

Follow-up, not this PR: mmmlu_kshot_clp mutates dataset_dict["test"] in place (:466, :550) for want of exactly this transform. Migrating it needs a fraction= budget on stratified_sample and a decision about task-state-dependent narrowing, so it ships separately.

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ — the one core/ change (Dataset.filter) adds no imports at all, only a method beside the existing transforms
• Deleted code verified — nothing deleted

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table — 24 models × 5 subsets, 120/120 cells exact against the paper's Table 3 by replay, plus a live run on a published checkpoint (Llama-3.3-70B-Instruct) landing 4/5 cells exact and +1 pooled (see §Score Comparison)
• Dataset loading tested (sieval dataset download platinum_bench succeeds)
• Task registered in package-level __init__.py (lazy AST scan picks up the subpackage; .pyi stubs regenerated, sieval/meta/index.json synced)

If: community/ Changes

• Upstream diff documented — sieval/community/platinum_bench.py is a byte-faithful copy of get_parse_fn + check_prediction from src/utils.py@8fd2f82; its docstring names the two behaviours callers must absorb rather than fixing them in place. Two deliberate deviations, both recorded in each task's ReferenceImpl.notes (so sieval task show surfaces them):
  1. prompt_variant (task arg, default cot) replaces upstream's hardcoded reasoning-model name list. All three of upstream's prompt strings are reachable explicitly — cot, no_cot, and no_cot_o1 (upstream's o1-only "Then, provide" → "Provide" edit) — so a name list that ages badly is traded for an argument the caller sets. The reference notes say which value reproduces which published row.
  2. A parse failure is recorded as prediction=None / correct=False instead of upstream's swallowed AttributeError.
• License attribution preserved — upstream harness code is CC-BY-4.0, attributed with a commit-pinned permalink in the vendor's module docstring. Note the data is cc-by-sa-4.0 (that's what DatasetMeta.license records); the two differ.

If: New Dependency

Not applicable — no new dependencies.

🤖 Generated with Claude Code

[ethan-scitix]

Cross-reference: #70 fixes the fleet-wide rollout["prediction"] resume KeyError this PR's task sidesteps with .get(), and adds check_preflight.py --check check_record_key_access to enforce it.

Suggested order: #70 first, then this one. This PR is already compliant either way — it is the only module in the repo that was — so nothing here needs to change. Merging the guard first just means this lands in a tree where the rule is enforced rather than one where it is a comment.