You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Gates the #139 v2 rerun. Principle (from the onboarding gauntlet, applied at run time): a completion-cap exhaustion is an environment error, not a serving fact — it must not become a scored miss.
Symptom
A response with finish_reason=length and no parseable payload (every completion token spent on reasoning) goes through repair rounds at the same cap, dies the same way, and its cells score as misses. The miss measures our budget, not the model.
Evidence (v2 probes, 2026-08-09/11)
minimax-m3 at its 16,384 cap: all tokens on reasoning, no tool call. At 3× cap: reasons 22,718, calls the tool, parses 18/18.
glm-5.2 at 16,384: finish=length, zero content. At 3×: parses 18/18 at 11,105 tokens (reasoning-length variance makes a fixed tight cap a lottery).
Contrast grok-4.5: finish=stop with budget to spare and no tool call — genuine formatting behavior, correctly retried then scored per policy.
Audit finding: nominal caps aren't even provider-enforced uniformly (K3 observed 132,376 vs 49,152 nominal), so a tight cap is neither a control nor a constraint — just noise.
Fix
Runner detects finish_reason=length with no valid payload → retries with an escalated budget (×2 steps, bounded by a generous ceiling, e.g. 128k or the provider max), reusing the onboard ladder logic.
Gates the #139 v2 rerun. Principle (from the onboarding gauntlet, applied at run time): a completion-cap exhaustion is an environment error, not a serving fact — it must not become a scored miss.
Symptom
A response with
finish_reason=lengthand no parseable payload (every completion token spent on reasoning) goes through repair rounds at the same cap, dies the same way, and its cells score as misses. The miss measures our budget, not the model.Evidence (v2 probes, 2026-08-09/11)
finish=length, zero content. At 3×: parses 18/18 at 11,105 tokens (reasoning-length variance makes a fixed tight cap a lottery).finish=stopwith budget to spare and no tool call — genuine formatting behavior, correctly retried then scored per policy.Fix
finish_reason=lengthwith no valid payload → retries with an escalated budget (×2 steps, bounded by a generous ceiling, e.g. 128k or the provider max), reusing the onboard ladder logic.budget_exhausted_at_ceilingbucket vsformat_no_tool_call.Methodology note for v2: caps are serving accommodations, not controlled variables — the controlled contract is the prompt and the answer schema.