feat(tasks): add 5 MathArena competitions (BRUMO, SMT, CMIMC, Apex, Apex Shortlist) - #73
Merged
Merged
Conversation
…pex Shortlist) Adds datasets + 0-shot generative tasks for five MathArena final-answer competitions, cloning the existing AIME/HMMT pass@k shape. Takes the pass@k math family from 8 to 13 members; no new dependencies. brumo_2025 30 problems Brown University Math Olympiad smt_2025 53 Stanford Math Tournament cmimc_2025 40 Carnegie Mellon Informatics and Math Competition apex_2025 12 MathArena-curated, very hard for models apex_shortlist_2025 47 MathArena-curated, ~50% for frontier models All five pin their HF snapshot and cite their upstream competition config at the same matharena commit the HMMT ports already reference. Validated against MathArena's published outputs, the bar HMMT Nov 2025 set: replaying the five `MathArena/*_outputs` datasets (39,131 rollouts) through each task's real extraction + grading reproduces upstream's recorded `correct` on 98.3 / 99.1 / 99.1 / 99.9 / 99.4% — inside the 96.2-99.7% band the four already-shipped ports occupy (re-measured on 17,367 rollouts), with four of five above the family's worst. Per-task figures live in reference_impl.notes. `HMMT_INSTRUCTION` is renamed `BOXED_INSTRUCTION`: seven upstream configs carry that string byte-identical, so four new tasks would otherwise import an HMMT-named constant. New `CMIMC_INSTRUCTION` covers the one ported competition whose instruction differs (it mandates a `### Final answer` section). The string each existing task sends is byte-identical, so no score moves. Also documented, verified by exact problem-text match: upstream curates Apex from other 2025 contests, so apex_2025 shares 3 of 12 problems with smt_2025 and apex_shortlist_2025 shares 3 of 47 with brumo_2025 / hmmt_feb_2025 — evaluating those together scores the shared problems twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task's reference_impl.notes told users to set n=4 to compare against matharena.ai. Measured over MathArena/apex_2025_outputs, that is wrong for Apex: 35 of 46 scored models are run at 16 samples/problem, 10 at 8, and only one at 4 — a 12-problem set needs the extra samples to produce a usable score. The other four competitions added in the previous commit are n=4 as documented (BRUMO and CMIMC uniformly; SMT and Apex Shortlist for the large majority). Adds examples/leaderboard-matharena.yaml, which runs the five new benchmarks at their published repeat counts and records the two per-model things a leaderboard comparison also has to match (temperature/top_p/max_tokens come from the model's own matharena config, and the model must be one MathArena actually scored). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…config Measured while reproducing MathArena's GPT OSS 120B numbers through an OpenAI-compatible gateway. Three things cost real time and none of them were written down anywhere: sieval asks for all `n` rollouts in ONE request where matharena issues `n` separate ones, so with a reasoning model the single response is n x longer and a gateway that drops a long stream mid-body kills the whole sample (`RemoteProtocolError: peer closed connection without sending complete message body`). The OpenAI client's `max_retries` cannot retry that — it only covers failures before the response body starts — so only `max_iterations` recovers. Failure rate tracked response length: 0/30 and 0/53 failed at ~17-21k output tokens/sample, 3/40 at ~29k, 6/12 at Apex's n=16 (~679k). And the sharp edge: a failed sample scores 0 rather than abstaining, because report() keeps it in the denominator. Apex reported 0.5 against an official 1.0 purely because 6 of 12 samples died; on the 6 that completed it matched exactly. So `fails` has to be read before any score is compared to a published number, and a non-zero count means recomputing over the completed problems and comparing against the official number on that same subset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure-mode note I added in c0fcc79 got two things wrong. `max_iterations` does not recover a dropped stream. A stage exception is terminal within a run (runner.py: `except Exception -> ctx.to_failed`); `max_iterations` bounds the feedback/iterate loop, and these tasks finalize on the first pass, so it was dead config. The actual recovery is re-invoking with `auto_resume: true`, which sends every retriable failure through the loader's `_prepare_failed_retries` -> rolled back to its pre-infer stage with `retry_count+1`, bounded by `runner_config.max_retries` (a different knob from the per-model one). Verified against this run's on-disk records: all 9 checked came back `preprocessed rc=1`, since `exception::RemoteProtocolError` is not in `ERROR_REASONS_NON_RETRIABLE`. The token figures also mixed units — per-rollout for three tasks, per-request for Apex — and omitted apex_shortlist, which also had failures. Replaced with one table in consistent tok/request, recomputed from the finished run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ame from The shortlist loader's docstring claimed it was "the shortlist Apex was drawn from". It is not. The two sets are disjoint: no shared problem at exact, normalized, or fuzzy (>=0.80) statement match, no shared answer string, and not one shared entry between their `source` columns (12 vs 47, zero intersection). Neither HF dataset card states the relationship, so the claim was an inference from the name. MathArena's own Apex writeup gives the real one: both sets come off the same 2025-contest sweep, split by difficulty — Apex kept only problems that Grok 4, GPT-5 (High), Gemini 2.5 Pro and GLM 4.5 all failed across 4 attempts (~100 competitions reviewed, 12 survived), while the shortlist is the companion band where SOTA models score ~50%. Also records the adversarial filter and its stated model bias in the Apex docstring, and cross-links the two so the naming does not re-invite the subset assumption. The 3+3 byte-identical sibling overlaps already documented (apex -> smt_2025 8/42/43; shortlist -> brumo_2025 30 and hmmt_feb_2025 19/20) are unaffected and independently confirmed by the `source` columns. meta/index.json unchanged: module docstrings are not embedded there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Byte-equality undercounted the overlap. Reconciling every `source` string in both Apex sets against the sibling datasets sieval actually ships turns up two more duplicates in the shortlist: problems 25/26 are AIME 2025 P14/P15, matching aime_2025 problems 14/15 at 0.992/0.970 after normalization — identical statements with identical golds (60, 735), differing only because that loader mirrors opencompass/AIME2025, which re-typesets. A run covering both scores them twice, exactly as the three byte-identical ones do. The same reconciliation clears two false candidates: the shortlist attributes problems 4 and 5 to the HMMT 2025 team round, which is not in hmmt_feb_2025 (individual rounds only, 30 problems) — statement match 0.43/0.34, not duplicates. And it confirms apex_2025's three SMT overlaps are the complete set there. Both notes now also state that the two Apex sets are disjoint, since the naming invites the opposite assumption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… rollout key These five were authored in parallel with #70, so they landed the same defect that PR had just removed from 38 other modules: `rollout["prediction"]` raises KeyError on resume, because `build_prediction_record` spells "could not extract" as `prediction=None` and `obj_to_dict` drops None-valued keys — the key is gone from disk, not null. A fresh run never sees it, which is why every unit test and both type checkers stayed green. Caught by rebasing onto main: #70 shipped `check_preflight.py --check check_record_key_access` as the enforcer, and it flagged all five on the first run. That is the enforcer working exactly as intended on code written after the contract but before the check. Fix is `.get("prediction")`, identical to the 39 sites #70 converted — equivalent when the key is present, and otherwise yielding the same None the fresh path produced. Not hypothetical for this branch: the live reproduction run resumed 11 stream-dropped samples through `feedback`. It survived only because a rolled-back sample re-runs postprocess in memory rather than hydrating it from disk; a sample resumed from a persisted postprocessed record with a failed extraction would have crashed. Two such rollouts exist in that run's output (apex #2, shortlist #25). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s prompt cohort Review follow-ups on the five new competitions, plus the two already-merged siblings that turned out to share the same conditions. 1. `list_answer` is now derived from the gold, as upstream's grader.py does. The branch was already vendored in `find_last_boxed_content`, but unreachable: `extract_boxed_answer` never passed the flag and `extract_answer` did not expose it. So sieval carried upstream's list logic and could never run it, while the BRUMO/SMT notes described the resulting gap as an inherent limitation. Both wrappers now take the flag and all 9 matharena-sourced tasks derive it the way `grader.py:178` does — a comma in the gold. Measured by replaying upstream's own stored rollouts: brumo_2025 98.30% -> 98.83% 28/5,280 moved (all problem 23) hmmt_feb_2025 99.39% -> 99.60% 16/7,680 moved (all problem 10) smt_2025 99.13% -> 99.13% 0/10,875 (every model boxed the whole list) cmimc/apex/shortlist unchanged — no comma golds, so the flag is always False 44 rollouts changed verdict, all 44 toward upstream and none away. `hmmt_feb_2025` had the same comma gold as the two new sets and no note; it has one now. 2. PROMPT COHORT recorded on all seven affected ports. Upstream changed its `instruction` string and did not re-run the earlier rows, so the published tables mix two prompts and the port only sends one of them: brumo 5/44 models on the ported instruction (600/5,280 rollouts) smt 5/43 (1,060/10,875) cmimc 4/35 (640/5,600) — 1,088 predate the `### Final answer` section apex 25/46 (3,830/7,717) apex_shortlist 30/42 (4,813/9,659) hmmt_feb_2025 5/64 (600/7,680) hmmt_nov_2025 5/22 (600/2,640) `hmmt_feb_2026` (30/30) and `aime_2026` (30/30) are clean and get no clause. The positioning is unchanged — sieval tracks the pinned config — so this is stated as a property of the comparison, not a defect. It does bound what a live delta means: every row in this branch's alignment table, including BRUMO's +3.3 pp, was produced under the older prefix. 3. `apex_shortlist_2025` loader docstring still said the overlap was three. 377299e corrected it to five in the task notes and index.json but not here. Tests: 4 extractor cases for the list branch, plus a family-level check that each matharena task derives the flag from the gold and degrades to upstream's default when `raw_sample` is absent (the resume path #70 lived on). Verified discriminating by reverting one task. 3,156 pass; ruff, ty and all 22 preflight checks clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…easured
Ran the A/B the review asked for: BRUMO 2025, gpt-oss-120b at reasoning_effort=high,
n=4, one gateway and one session, both arms config-identical except the instruction.
old prefix ("Please reason step by step, and ...") 90.83
ported instruction (what this task sends) 90.00
official 92.50
The prompt is worth +0.83 pp. Four of thirty problems moved and two of those went
each way, giving t=0.37 over the per-problem differences (SE 2.24 pp, df=29). That
is indistinguishable from resampling, so the cohort split — real, and still worth
recording — is not an explanation for a delta of any size. brumo's note now carries
the measurement; the other six say the magnitude is known only from this one set
rather than asserting an unquantified confound.
Two things the run also settles, neither of which changes code:
* The saturation argument holds and is if anything understated. Only 4-5 of the 30
problems are non-degenerate at n=4 (24/30 and 25/30 come back 4/4), so the score
is carried by a handful of problems and one problem is worth 3.33 pp. Computing
sigma_D from the observed per-problem binomial spread gives 2.20 pp against the
2.57 pp the PR states — same ballpark, PR's is the conservative one.
* Neither arm reproduces this branch's own earlier 95.8. Both land near 90, i.e.
2.5 pp *below* official where the earlier run was 3.3 pp above, a 5.8 pp gap at
2.6 sigma_D between two sieval runs of the same cell. That run used
max_tokens=32768 and a set temperature; this one used 131072 and left temperature
unset to match upstream's oss-120b.yaml. So the gap is decoding config, not
sampling — which is the sharper form of "this set cannot rank models".
Raw arms, configs and log: /volume/ai-infra/ylsun/brumo-prompt-ab (out of tree).
3,156 tests pass; ruff, project-wide ty and all 22 preflight checks clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pped The notes had grown to 2,232 words across the nine matharena tasks, and most of the growth was restatement rather than content: the PROMPT COHORT tail was 54 identical words in seven files and the REPEATS clause another 44, each paraphrased slightly differently in every copy. Rewrote the shared clauses once and applied them identically, then compressed the per-set prose. 2,232 -> 1,761 words (-21%). Every measured figure survives; verified mechanically by diffing the numeric tokens of each notes block against HEAD (thousands separators normalized) rather than by eye -- no number is lost or invented in any of the nine. The four already-merged siblings (hmmt_feb_2025, hmmt_nov_2025, aime_2026, hmmt_feb_2026) are included deliberately. They share the REPEATS and DEVIATION clauses with the five new ports; shortening only the new files would leave two wordings of the same fact in the same directory. meta/index.json embeds reference_impl.notes, so it is regenerated here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things that both amount to "do not keep a roster in two places". `examples/leaderboard-matharena.yaml` is removed: it was added by this branch and should not have been. Nothing referenced it but itself, and the net diff against main now shows no change under `examples/`. What it carried that is not recorded elsewhere is the long-stream failure mode — sieval issues all `n` rollouts as one streamed request where matharena issues `n`, so tokens/request scales with `n` and a gateway that hangs up mid-body fails the whole sample, which `max_retries` cannot retry; plus the measured tok/request table and the warning that a failed sample scores 0 rather than abstaining. That is runner behaviour rather than per-task metadata, so it wants its own home rather than a silent reinstatement here. README no longer lists benchmark names. The Features bullet named seven math sets and would have needed an edit for every future one; it now names the six top-level categories of the shipped taxonomy (`Level1Category`), which changes only when the taxonomy does, and points at `sieval dataset list` / `sieval task list` / `sieval task show <name>` for what a given build actually ships. The `[math]` extra comment described its ten current consumers; it now describes what the extra provides. The remaining extras are one-to-one with their benchmark and are unchanged. The `DEPS_GROUP` pointer already below the code block is what makes the enumeration redundant in the first place. ruff clean, all 22 preflight checks pass (`check_examples` now covers 6 files), 3,026 unit tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ethan-scitix
force-pushed
the
feat/matharena-5-benchmarks
branch
from
August 6, 2026 14:51
24b2e89 to
eebe2b2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
Adds five MathArena competition benchmarks — BRUMO 2025, SMT 2025, CMIMC 2025,
Apex 2025 and Apex Shortlist 2025 — following the existing HMMT/AIME port pattern
(boxed prompt, last-boxed extraction,
math-verifyequivalence).reference_impl.notesrecords what deviates from upstream and whatwas measured, not just what was intended: replay agreement against MathArena's
own stored rollouts (99.9% on Apex, 99.4% on Shortlist), the un-vendored
normalize_stringgap, and the per-set repeat protocol.apex_2025_outputsare at 16 runs/problem. A 12-problem set needs it. Settingn=4there would silently compare against a different protocol.apex_2025shares3 of its 12 problems with
smt_2025;apex_shortlist_2025shares 5 of 47 withhmmt_feb_2025(2),aime_2025(2) andbrumo_2025(1). Despite the name thetwo Apex sets are disjoint — not sub/superset. Verified by full statement
cross-product at three match strengths plus reconciliation against every
sourcestring, so the counts are complete rather than whatever byte-equalityhappened to catch.
rollout["prediction"]defect: these five wereauthored in parallel with fix(tasks): stop indexing a rollout key that is absent on disk #70 and reintroduced it. Caught by that PR's own new
preflight check on rebase.
Review follow-ups (
12564412)Upstream's
list_answerrule is applied instead of vendored-but-unreachable.The branch was already ported into
find_last_boxed_content, but no caller couldreach it:
extract_boxed_answernever passed the flag andextract_answerdidnot expose it — so sieval carried upstream's list logic and could never run it,
while the BRUMO/SMT notes described the resulting gap as inherent. All 9
matharena-sourced tasks now derive it the way
grader.py:178does: a comma inthe gold. Replaying upstream's own stored rollouts, this moves 44 verdicts
across the family, all 44 toward upstream and none away — BRUMO 98.30% →
98.83% (28 rollouts, all problem 23), HMMT Feb 2025 99.39% → 99.60% (16
rollouts, all problem 10), SMT unchanged (every model boxed the whole list),
CMIMC/Apex/Shortlist unchanged (no comma golds).
hmmt_feb_2025had the samecomma gold as the two new sets and no annotation; it has one now.
Every published table these ports compare against mixes two prompts.
MathArena changed its
instructionstring and did not re-run the earlier rows,so a
PROMPT COHORT:clause now records the split on all seven affected ports.Share of each published table generated with the instruction this port sends:
The rest carry a leading
Please reason step by step, and; on CMIMC a further1,088 rollouts predate the
### Final answersection entirely.hmmt_feb_2026(30/30) and
aime_2026(30/30) are clean and get no clause. Positioning isunchanged — sieval tracks the pinned config — so this is recorded as a
property of the comparison, not a defect in the port.
apex_shortlist_2025's loader docstring still said the overlap was three; thecorrection to five had reached the task notes and
index.jsonbut not it.examples/leaderboard-matharena.yamlis no longer added by this PR(
24b2e893), and the README stopped enumerating benchmark names: the Featuresbullet now names the six top-level categories of the shipped taxonomy rather
than seven math sets, so onboarding a benchmark no longer edits the README.
sieval dataset list/task list/task showare the roster.Prompt A/B (
8d55fe12) — the cohort split does not explain BRUMORan the experiment the cohort finding implied: BRUMO 2025,
gpt-oss-120batreasoning_effort=high, n=4, one gateway and one session, both armsconfig-identical except the instruction.
Please reason step by step, and put your final answer within \boxed{}.Put your final answer within \boxed{}.(what the port sends)The prompt is worth +0.83 pp. Four of thirty problems moved and two of those
went each way: t = 0.37 over the per-problem differences (SE 2.24 pp, df = 29).
Indistinguishable from resampling. The cohort split is real and still worth
recording, but it is not the explanation for a delta of any size —
brumo's notenow carries the measurement, and the other six say the magnitude is known only
from this one set instead of asserting an unquantified confound.
Two things the run settles as a side effect:
problems are non-degenerate at n=4 (24/30 and 25/30 come back 4/4), so the
score rides on a handful of problems and one problem is worth 3.33 pp.
Computing σ_D from the observed per-problem binomial spread gives 2.20 pp
against the 2.57 pp stated above — same ballpark, and the stated one is the
conservative choice.
2.5 pp below official where the earlier run was 3.3 pp above. That is a
5.8 pp gap at 2.6σ_D between two sieval runs of the same cell, and it is
larger than the sampling noise either run carries. What it is not is attributed:
this run used
max_tokens=131072with temperature unset, to match upstream'soss-120b.yaml, but the earlier run's exact decoding config isn't recoverablehere (its output directory isn't in this checkout). So the honest reading is
that this cell moves by ~6 pp
across two plausible sieval configurations, not that any one knob caused it.
Pinning that down needs the earlier run's
effective_config.yaml.Raw arms, configs and log are out of tree at
/volume/ai-infra/ylsun/brumo-prompt-ab.Related Issues
Refs #70 (the
check_record_key_accessenforcer that caught the rollout-key commit).Test Plan
Automated
ruff check && ruff format --check)ty check— clean;check_preflight.pyall 22 checks pass)pytest— 3156 passed, +23 for thelist_answerrule:4 extractor cases plus a family-level check that each matharena task derives
the flag from the gold and degrades to upstream's default when
raw_sampleis absent — the resume path fix(tasks): stop indexing a rollout key that is absent on disk #70 lived on. Verified discriminating by
reverting one task: exactly that parametrization fails.)
Manual
Live reproduction against MathArena's published
GPT OSS 120B (high)row, servedby the scitix gateway as
openai/gpt-oss-120b, at each set's publishedn(Apex 16, rest 4).
Suite mean Δ = +0.84 pp; σ of that mean = 1.22 pp; z = +0.69.
target and I am not hiding it — that cell has 23 of 30 problems saturated
at n=4, so one run's noise is 2.57 pp and +3.3 is 1.30σ. The <3% rule is the
wrong yardstick for a near-saturated set; σ_D is. Now tested (see the
Prompt A/B above): the older prefix is worth only +0.8 pp, so the prompt is
ruled out and saturation stands — but the A/B also puts two runs of this same
cell 5.8 pp apart across two plausible decoding configurations, so read the
+3.3 as one draw from a wide distribution rather than as a measured offset.
so the paired difference carries noise from both draws.
Apex), ruling out "scored higher because it generated longer".
port scores −0.84 pp vs their published table, while the live run is
+0.84 pp. The two offsets point opposite ways, so the residual is the
serving path, not grader drift.
sieval dataset downloadresolves for all five — everysourceis anhf:ref pinned to a commit, not a branch:MathArena/brumo_2025@6de9bc09,MathArena/smt_2025@d8e99d83,MathArena/cmimc_2025@8f27517c,MathArena/apex_2025@ac8a641d,MathArena/apex-shortlist@f3efdf22.Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstringcore/If: New or Modified Benchmark
sieval dataset download <name>succeeds)__init__.pyNote for the reviewer
BRUMO and CMIMC cannot rank frontier models. Across every model MathArena
scored, their top-5 spread is 1.7 pp and 2.5 pp against σ_D of 2.57 and 3.47 —
the spread is below the noise, so re-running reorders the leaderboard. Apex is
27.1 pp spread against 1.25 pp noise. This does not argue against merging the
saturated sets — they are good delivery-regression tripwires, which is SiEval's
actual job — but they should not be read as capability rankings. The A/B
above turned this from an argument into a measurement: on BRUMO only 4–5 of
30 problems are live at n=4, one problem is worth 3.33 pp, and two runs of the
same model landed 5.8 pp apart across two configurations — against a top-5
leaderboard spread of 1.7 pp. Still not encoded in the repo. The proposal
is a
NOISE FLOOR:clause in eachreference_impl.notes, besideSMALL N:—same reader, same field, travelling with the shipped metadata instead of living
in a doc. Not done in this PR; say the word and it goes in with the per-set
numbers.
Whether the live alignment table belongs inMoot — theexamples/.example config is no longer part of this PR (
24b2e893removes it; the netdiff under
examples/is now empty). Replay evidence and one-gateway-one-dayevidence should not share a field at equal apparent authority anyway. What did
land in
reference_impl.notesis the reproducible half — the prompt cohort,the
list_answerdeltas and the A/B — all re-derivable by anyone.One thing left homeless by the removal, flagged rather than silently
reinstated: the long-stream failure mode. sieval issues all
nrollouts asone streamed request where matharena issues
n, so tokens/request scales withn, a gateway that hangs up mid-body fails the whole sample, andmax_retriescannot retry it — and a failed sample scores 0 rather than abstaining, so
failshas to be read before any score is compared. That is runner behaviour,not per-task metadata; say where it should live and I'll put it there.
🤖 Generated with Claude Code