Skip to content

fix(llm): recover from empty responses caused by reasoning token budgets - #537

Merged
ajianaz merged 1 commit into
developfrom
fix/536-empty-llm-response
Aug 28, 2026
Merged

fix(llm): recover from empty responses caused by reasoning token budgets#537
ajianaz merged 1 commit into
developfrom
fix/536-empty-llm-response

Conversation

@ajianaz

@ajianaz ajianaz commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Fixes #536

What

  • ChatResponse now captures finish_reason and reasoning_content (string or parts array) from LLM responses
  • Empty content + finish_reason=length triggers an automatic retry with doubled max_tokens (…→32768 ceiling) inside chat_completion
  • Last-resort salvage: JSON-looking reasoning text is used as the raw response (parse layer still validates it)
  • Parse layer reports an explicit "provider returned an EMPTY response" error instead of serde EOF noise
  • Default max_tokens raised 4096 → 8192

Why

Reasoning models (e.g. glm/glm-5.3 via Bifrost) can spend the entire output budget on chain-of-thought and return HTTP 200 with content: "" and finish_reason: "length". cora never read those fields, silently fed an empty string to the JSON parser, and surfaced the misleading EOF while parsing a value at line 1 column 0 — while the model's actual findings JSON sat in the backend logs (codecoradev/cosy#63). The old retry re-sent the same 4096 budget, guaranteeing identical failure.

Testing

  • New unit tests: budget escalation table (length/stop/none × 32768 ceiling), reasoning salvage (plain JSON / fenced / parts array / prose / none), explicit empty-raw parse error
  • Full suite green (912 tests, tree-sitter feature); cargo fmt clean; no new clippy findings vs develop
  • Pre-commit hook (Cora review) passed locally

All commits signed-off (DCO).

Reasoning models (e.g. GLM via Bifrost) can spend the entire
max_tokens budget on chain-of-thought and return HTTP 200 with
content: "" and finish_reason: "length". cora never read finish_reason
or reasoning_content and silently fed an empty string to the JSON
parser, surfacing the misleading 'EOF while parsing a value at line 1
column 0' — while the model's actual answer sat in the backend logs
(cosy#63) (#536).

Recovery now follows productivity over token frugality:
- response parsing captures finish_reason and reasoning_content
- empty content + finish_reason=length auto-retries with doubled
  budget (4096 -> ... -> 32768 ceiling) inside chat_completion
- last resort: JSON-looking reasoning text is salvaged as the raw
  response; the parse layer still validates it
- parse layer reports an explicit 'provider returned an EMPTY
  response' instead of serde EOF noise
- default max_tokens raised 4096 -> 8192 for reasoning headroom

Regression tests: budget escalation table, reasoning salvage
(string/parts/fenced/prose), explicit empty-raw error.

Signed-off-by: ajianaz <ajianaz@users.noreply.github.com>
@ajianaz
ajianaz merged commit 2813186 into develop Aug 28, 2026
17 of 18 checks passed
@ajianaz
ajianaz deleted the fix/536-empty-llm-response branch August 28, 2026 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

review/scan: reasoning models can return empty content — misleading 'EOF while parsing' error

1 participant