Skip to content

Completion-cap exhaustion is an environment error: escalate the budget, never score the death #150

Description

@MaxGhenis

Gates the #139 v2 rerun. Principle (from the onboarding gauntlet, applied at run time): a completion-cap exhaustion is an environment error, not a serving fact — it must not become a scored miss.

Symptom

A response with finish_reason=length and no parseable payload (every completion token spent on reasoning) goes through repair rounds at the same cap, dies the same way, and its cells score as misses. The miss measures our budget, not the model.

Evidence (v2 probes, 2026-08-09/11)

  • minimax-m3 at its 16,384 cap: all tokens on reasoning, no tool call. At 3× cap: reasons 22,718, calls the tool, parses 18/18.
  • glm-5.2 at 16,384: finish=length, zero content. At 3×: parses 18/18 at 11,105 tokens (reasoning-length variance makes a fixed tight cap a lottery).
  • Contrast grok-4.5: finish=stop with budget to spare and no tool call — genuine formatting behavior, correctly retried then scored per policy.
  • Audit finding: nominal caps aren't even provider-enforced uniformly (K3 observed 132,376 vs 49,152 nominal), so a tight cap is neither a control nor a constraint — just noise.

Fix

  1. Runner detects finish_reason=length with no valid payload → retries with an escalated budget (×2 steps, bounded by a generous ceiling, e.g. 128k or the provider max), reusing the onboard ladder logic.
  2. Cells that still fail at the ceiling are real misses; the per-model parse-failure column (Explanation contract is unenforced — set the v2 scoring policy #145) reports them in a distinct budget_exhausted_at_ceiling bucket vs format_no_tool_call.
  3. Escalations are logged per model in the run manifest; the cost column reflects actual spend as always.
  4. v2 model cards still start from probe-informed caps (minimax/glm ≥32k) so escalation is the safety net, not the norm.

Methodology note for v2: caps are serving accommodations, not controlled variables — the controlled contract is the prompt and the answer schema.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions