[RFC]: Sampled-answer metric family for math and MCQ — avg@k / pass@k / pass^k, consistency, and health signals

[ethan-scitix]

Motivation.

Every generative task in SiEval reports a single headline accuracy. For the
sampled-answer families — math and MCQ — one number per (model, task) is not
enough to do what SiEval exists to do: decide whether a converted / quantized /
re-served model has degraded. A model whose mean accuracy is unchanged but
whose answer distribution has widened is a real delivery defect, and today it is
invisible.

Three concrete gaps in the current code:

1. pass@1 is already avg@n, but nothing says so. 13 tasks carry a
private _pass_at_k (identical copy in each, e.g. tasks/aime_2025_0shot_gen.py).
_pass_at_k(n, c, 1) reduces to c / n — i.e. the mean over n rollouts, which
is what MathArena publishes as avg@k and what simple-evals / Codex publish as
pass@1. The formula is right; what is missing is that n appears nowhere in
report.json. A run at n=4 and a paper number at n=16 land in the same
column with no way to tell them apart, which is exactly where published-number
alignment goes wrong.

2. pass@k alone is the wrong direction for delivery verification. It
measures the best-of-k upper bound. A model with higher sampling variance can
score a higher pass@k while being worse to ship. The reliability direction
(pass^k: all k rollouts correct) is the one that catches the failure mode we
actually care about, and we do not compute it.

3. Health signals are logged, not measured. PredictionRecord carries a
per-rollout extracted flag, and the n_rollouts < k case (a model or a
mid-stream drop returning fewer than the requested n) only produces a
logger.warning — neither reaches report.json. So a report cannot distinguish
"the model got it wrong" from "our extractor broke" or "the stream died and the
sample was scored 0".

Scope note: this RFC deliberately targets math and MCQ only. Code tasks
(human_eval ×2, mbpp, livecodebench ×2, scicode) keep their existing
pass@k unchanged — majority voting over programs has no principled definition
short of AlphaCode-style behavioral clustering, which is a separate project.

Proposed Change.

A. One estimator family, one module

Add sieval/core/tasks/metrics.py and route every task through it, replacing the
13 copies of _pass_at_k. This is extraction on coupling, not on call count:
the estimator must be identical everywhere or cross-task numbers stop being
comparable.

Implement the generalized form once and derive the rest:

G-Pass@k_tau(n, c, tau) = P[ at least ceil(tau * k) of k sampled rollouts are correct ]

  tau = 1/k  ->  pass@k        (optimistic / best-of-k, today's metric)
  tau = 1    ->  pass^k        (pessimistic / all-k-correct, reliability)
  mean       ->  avg@k = c/n   (today's `pass@1`)

Requires n >= k; hypergeometric, same shape as the existing Chen et al.
estimator.

B. Name the sampling budget in the report

• Keep the key pass@1 (it matches upstream convention; renaming it would
  silently invalidate every stored leaderboard row).
• Do not add a separate avg@k key with the same value — two names for one
  number reads as two independent pieces of evidence.
• Add n_rollouts (the actual per-sample rollout count, or its distribution when
  it varies) to every report that samples n > 1.

C. Health metrics as first-class report keys

• n_unextracted — rollouts whose prediction is None. Separates model error
  from parser error. Also a precondition for D: an unextracted rollout must not
  form an answer cluster.
• n_short — samples that came back with fewer than the requested n rollouts.
  These currently score 0 for pass@k and bias every metric downward, silently.

D. Consistency, and only then majority

Add self_consistency (modal-cluster share, averaged over samples) as the
primary dispersion metric. It is a continuous quantity, so it shows the
"same mean, wider spread" degradation that a thresholded majority@k scalar
can miss entirely.

majority@k (a.k.a. cons@k / maj@k) is proposed with three restrictions:

1. Single-answer tasks only — math and MCQ. Not code, not free-form.
2. k == n only. Sub-sampling k < n would need either an unbiased
   estimator or a seed; a seed in the metric layer is a new source of
   irreproducibility and is not worth it here.
3. Equivalence, not string equality. For math, \frac{1}{2} and 0.5 must
   land in one cluster; clustering on raw strings systematically under-reports
   majority@k. Clustering goes through the same verifier the task already uses
   (math_verify for the math family), with a deterministic tie-break
   (lowest rollout index).

E. MCQ tasks must stop hard-coding rollout 0

gpqa_diamond, mmlu, mmlu_pro, openbookqa all read
ctx.feedback_result["rollouts"][0]["correct"]. Under n > 1 they would
silently score only the first rollout and discard the rest. These need a k/n
knob and a rollout-aware report() before any of A–D applies to them.

(gpqa_diamond already carries a comment flagging exactly this.)

F. Declare the denominator; do not silently unify it

Reports currently split two ways:

• len(finals) + len(fails) — a pipeline failure counts as wrong. 17 tasks;
  gsm8k_0shot_gen and hendrycks_math_kshot_base_gen document this as
  deliberate ("matching DeepSeek's full-set accuracy").
• len(finals) — failures are excluded. gpqa_diamond, mmlu, mmlu_pro,
  mmmlu_clp, openbookqa, drop, gsm8k_kshot_base, theoremqa.

So the divergence is upstream-convention-driven, not accidental, and unifying it
would change score for 8 tasks and break comparability with every stored
number. Proposal: leave the values alone, make the convention explicit —
record the denominator policy in the report (and/or in TaskMeta) so a reader
knows which population a number is over. If a value change is ever justified, it
ships as a separate, versioned break with a quantified delta, not folded into
this work.

G. Offline recompute over shards

Implement the metric layer so it can run against an existing result_dir,
not only inline in report():

• JudgementRecord already persists n_rollouts and n_correct per sample
  (core/tasks/records.py), so A/B are pure functions of what is already on
  disk — every past n > 1 run can be backfilled with zero model calls.
• PredictionRecord persists per-rollout prediction, and record_each_stage
  defaults to True, so D is backfillable too for tasks already on the
  stage-output protocol.

Benefits: validate the estimators against stored runs before spending any GPU;
old runs gain the metrics instead of merely staying valid; a later bug fix in a
metric costs a recompute, not a re-run. Consistent with "disk state is source of
truth" (core/CLAUDE.md).

Feedback Period.

One week.

CC List.

(none)

Any Other Things.

In scope

Wave	Tasks	State
1 — math, already sampled	aime_2024/2025/2026, hmmt_feb_2025/feb_2026/nov_2025, math_500, imo_answer_bench	have k/n + pass@k; migrate to the shared estimator, add C/D
2 — math, not yet sampled	gsm8k_0shot_gen, gsm8k_kshot_base_gen, hendrycks_math_kshot_base_gen, theoremqa_kshot_base_gen	need k/n knobs first
3 — MCQ	gpqa_diamond, mmlu_0shot_gen, mmlu_pro_0shot_gen, openbookqa_kshot_gen	need E first

Out of scope

• Code — human_eval ×2, mbpp, livecodebench ×2, scicode. Existing
  pass@k migrates to the shared estimator (A) and picks up C, but no
  majority@k / self_consistency: there is nothing well-defined to vote on.
• clp / ppl — mmlu_kshot_clp, cmmlu_kshot_clp, c_eval_kshot_clp,
  mmmlu_kshot_clp, hellaswag_kshot_ppl. Deterministic scoring, no sampling;
  every metric here is an identity.
• Constraint satisfaction — ifeval, ifbench. The score is a
  per-instruction rate, not a binary verdict; pass@k would need "pass" defined
  first.
• Partial credit — drop, ruler. F1/EM would need binarizing.
• LLM-judged free-form — hle, simpleqa_verified, browsecomp, aa_lcr.
  Deferred: grader cost scales with n, clustering needs the judge, and
  simpleqa is a three-way grade (correct / incorrect / not_attempted) so
  "pass" needs defining. Revisit after the math/MCQ waves land.

Data compatibility

Checked artifact by artifact. Adding report keys is non-destructive:

• profile.json — unaffected. It holds only token_usage / io / stages;
  zero coupling to report metrics.
• report.json — additive. cli/leaderboard/scanner.py reads report["score"]
  and nothing else, so stored reports stay readable as long as score keeps its
  meaning (which is why F leaves values alone).
• meta.json — unaffected. The run-identity block records
  name / display_name / dataset / eval_mode / n_shot / tags /
  status; new metric knobs are not identity.
• Persisted effective_config.yaml — unaffected. It dumps the reified config
  (raw YAML + CLI overrides), not constructor defaults, so a new task arg with
  a default does not perturb the strict-match body.
• anomalies.json — unaffected by metrics alone (rules_hash hashes rule
  definitions). But adding any anomaly rule keyed on the new metrics
  rotates the hash fleet-wide and marks every stored anomalies.json stale —
  so if we want such a rule, it should be bundled deliberately, not trickled in.

Two things that do bite:

• Version series gate. Current tag is v0.7.0; below 1.0 the break axis is
  (major, minor) (core/runners/resume_gate.py, RFC [RFC]: Resume version-compatibility gate + per-record version provenance #24). Shipping this as
  0.8.0 rejects --resume for every unfinished 0.7.x run (completed runs
  stay fully readable — the gate only blocks resume). Either land it as 0.7.x
  patches, or announce a drain window. This is a release-sequencing decision,
  independent of the metric design.
• _STRICT_RUNNER_KEYS. Any new runner-level config for this work must be
  classified in cli/leaderboard/session.py's three-way partition (enforced by
  test_every_field_classified_exactly_once), and anything touching on-disk
  content goes strict — i.e. metrics cannot be toggled on mid-run.

Also needs updating

scripts/check_preflight.py --check check_task_shot_knobs enforces "a task
taking k must compute a pass@k metric" (sieval/tasks/CLAUDE.md). New metric
knobs need that check extended, or it will either false-fail or wave through a
knob that computes nothing.

Deliberately not proposed here

A standard error / confidence interval on the headline score. It is arguably the
highest-value addition of all — without it neither avg@k nor pass@k can
answer "is A better than B, or is this noise?", which is the question that
matters when top-N spread approaches the run-to-run noise floor. It is left out
only because it is a separable design (bootstrap scheme, clustering over
problems, where the interval is stored) and would double the surface of this RFC.
Proposed as a follow-up RFC once the estimator module in A exists.