v2 board condition: move every model to tool_choice auto

[MaxGhenis]

Why

Under the canonical forced tool_choice, Anthropic's thinking-by-default models skip thinking while every other reasoning-by-default provider reasons regardless — verified and measured in sensitivity/claude-thinking-2026-08.md (Claude Opus 5: 79.8 forced vs 85.6 under auto, 1,984/1,984 parsed both ways).

The harness already accommodates this exact incompatibility when it fails loudly: Moonshot rejects forced tool_choice with thinking enabled, so Kimi models run the JSON contract and reason freely. Anthropic's version fails silently, so it never got the same treatment. Like cases should get like treatment — and tool_choice: auto is less configuration than forcing, so the provider-defaults principle gets stronger, not weaker.

This also aligns the board with field practice: Artificial Analysis and Aider publish reasoning configurations as labeled entries, and MLPerf's closed/open division split is the archetype for a fixed-condition board plus documented variants. The Evals-Consensus reporting-standards effort (evals-consensus.ai) is pushing the same direction on configuration disclosure.

Gates

• Roster-wide probes confirming every model reliably emits the tool call under tool_choice: auto (Opus 5 went 1,984/1,984; the rest are unverified). Any model that answers in text instead is the blocker to resolve or document.
• Ships as a versioned re-run of the full roster with a new dataset version — never an edit to existing scores. The current board stays published as-is.

Release split (2026-08-09)

The two change axes ship as separate versions so each release is decomposable against the last:

• v2 — condition fix, expedited: tool_choice: auto for every model, whole-scenario for the formerly chunked grandfathers, and the harness-audit scoring fixes — on the SAME households and references as v1. No upstream dependency; the three Claude thinking sensitivity runs are already valid v2 rows. Target: as soon as probes and reruns complete.
• v3 — microdata refresh: fresh household draw from the corrected microcosm release, references regenerated on then-current policyengine-us (retiring scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP #92 and the Scenario refresh: state SSA disability status and dependent relationships explicitly #111 semantics queue), new weighting population — same condition as v2. Gated on the microcosm release.

The interim site note (leaderboard callout + methodology corrections) covers the gap until v2 ships.

Scope expansion (2026-08-08)

A full redo also refreshes the underlying microdata (Max). v2 therefore re-derives everything, not just the request condition:

• Households: a fresh scenario draw from the corrected microcosm sparse release (the successor to the June populace build the current 100 households come from). Gated on that release landing.
• References: regenerated on then-current policyengine-us, which also retires the queued scenario fixes (scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP #92 scenario_121 SC SNAP encoding, the Scenario refresh: state SSA disability status and dependent relationships explicitly #111 semantics items) that were waiting for the next pinned-version regen.
• Condition: tool_choice: auto for every model, per below.
• Pipeline: the harness-audit fixes land before the rerun so v2 is scored once on the corrected pipeline — filed as Repair rounds resample and overwrite valid answers #143 (repair overwrite), Explanation trailer overrides the structured tool-call value #144 (trailer override), Explanation contract is unenforced — set the v2 scoring policy #145 (explanation-contract policy decision), Resume validation ignores model identity and serving treatment #146 (resume identity), Cost and latency accounting integrity #147 (cost/latency accounting), Serialization order, export guards, batch parity, cache hygiene #148 (serialization/export/batch/cache bundle).

Because households and condition both change, v1 and v2 rows are not comparable at all — the bridge is the sensitivity family: the three Claude thinking runs already measure v1-households + v2-condition, and keeping one or two non-Claude bridge runs in that cell would let us decompose condition effects from scenario effects.

Probe data point: under auto, Claude Sonnet 5 failed to emit a parseable tool call on 56/1,984 answers (Fable 5 and Opus 5: 0/1,984). "Reliable tool calling under auto" is therefore not universal even within one provider — v2 needs the parse-failure policy stated up front (score-as-miss, matching how JSON-contract parse failures are treated today).

Checklist

• Corrected microcosm sparse release lands (upstream gate)
• Fresh scenario draw + reference regeneration (retires scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP #92 and the Scenario refresh: state SSA disability status and dependent relationships explicitly #111 semantics queue)
• Harness-audit scoring fixes merged before the rerun — v2 scoring-pipeline fixes: repair immutability, structured-canonical, resume fingerprints, export guards, spend ledgers #149 (ea67093) closes Repair rounds resample and overwrite valid answers #143 Explanation trailer overrides the structured tool-call value #144 Resume validation ignores model identity and serving treatment #146 Cost and latency accounting integrity #147 Serialization order, export guards, batch parity, cache hygiene #148; Explanation contract is unenforced — set the v2 scoring policy #145 stays open as the explanation-policy decision
• Probe all roster models under tool_choice: auto (tool-call reliability + cost/latency deltas)
• Decide contract for any model that fails the probe
• Full-roster re-run under the v2 condition
• Per-row reasoning/serving-configuration labels on the site (not only the paper's Table 2)
• Labeled sensitivity/variant rows surface (e.g. "Claude Opus 5 (thinking)" while v1 remains canonical)
• Paper refreeze with the v2 condition documented and the v1→v2 bridge explained

🤖 Generated with Claude Code

[MaxGhenis]

v2 condition probes complete (2026-08-09)

All 29 board models covered. Verdict: no blocker to the auto condition — every flag resolves with a model-card change or the stated score-as-miss policy. Raw rows live in results/local/v2_probes/ on the probe machine; summary:

One full whole-scenario request (all 18 headline variables, scenario_000,
the harness's own request builder) per board model, tool_choice: auto
for tool-contract models, POLICYBENCH_CHUNK_OVERRIDE=none. Raw rows:
probes.jsonl, followups.jsonl, json_probes.jsonl in this directory.
Total probe spend ≈ $1.20.

Tool-contract models, sync transport (16)

model	tool call under auto	notes
claude-fable-5	✅ 18/18 outputs	comp 3,368 tok (vs ~2,000 for thinking-off 4.x)
claude-opus-5	✅	comp 3,986
claude-sonnet-5	✅	comp 3,883
claude-opus-4.8	✅	comp 1,975 (thinking-off default, unchanged)
claude-opus-4.7	✅	comp 1,974
claude-sonnet-4.6	✅	comp 2,452
claude-haiku-4.5	✅	comp 2,102
gemini-3.6-flash	✅	reasons (4,554) and calls
gpt-5.4-mini	✅	4.6s
gpt-5.4-nano	✅	5.8s
grok-4.3	✅	reasons (460) and calls
grok-build-0.1	✅	reasons (19,253) and calls
inkling	✅	reasons (12,612) and calls
grok-4.5	⚠️ 4/5 across samples	reasons 4–9k tok; 1/5 answers in prose without the tool call (finish=stop). Repairs re-ask; residual = score-as-miss per stated policy
minimax-m3	⚠️ cap, fixed	at v1 cap 16,384: burns it all reasoning, no call. At 3× cap: reasons 22,718, calls tool, 282s. v2 card: cap ≥32k, timeout ≥300s
gpt-5.5	⚠️ timeout, fixed	v1 card timeout 60s (chunk=3 in v1); whole-scenario run takes 70.6s (reasons 3,582, calls tool). v2 card: thinking-class timeout

Anthropic reasoning-token column reads 0 for all rows — litellm never
populates it (BerriAI/litellm#36290); completion-token magnitude is the
signal until #140 records it from the raw response.

GPT-5.6 family (3) — transport constraint

Sync chat/completions rejects function tools for gpt-5.6-sol/terra/luna
("Function tools with reasoning_effort are not supported … use
/v1/responses or set reasoning_effort to 'none'"), identical under forced
and auto — transport-level, not condition-level. Their v1 board rows ran
via the native batch/responses adapter; v2 applies tool_choice: auto
inside that same transport. Needs a one-line methodology note.

JSON-contract models (10) — no tool_choice; de-chunk coverage

Whole-scenario parse probes for the three v1-chunked json models:

model	whole-scenario parse	notes
kimi-k2.6	✅ 18/18	26,355 comp tok, 626s — v2 card needs a long timeout
qwen-3.7-max	✅ 18/18	8,007 comp tok, 140s
glm-5.2	✅ at 3× cap (18/18)	at v1 cap 16,384: finish=length, zero content (budget consumed by reasoning). At 3× cap: parses 18/18 at 11,105 comp tok, 66.7s — the overrun is reasoning-length variance; the raised cap removes the failure mode

Remaining seven json models already run whole-scenario on the board; no
condition change for them in v2 beyond the corrected pipeline.

v2 model-card changes implied

• minimax-m3: completion cap ≥32k, timeout ≥300s
• gpt-5.5: thinking-class timeout (observed 70.6s whole-scenario)
• kimi-k2.6: timeout ≥900s whole-scenario
• glm-5.2: completion cap ≥32k (confirmed at 3×)
• grok-4.5: no card change; document ~80% single-shot tool-call rate under
  auto, repairs + score-as-miss policy absorb the rest