fix(llm): recover from empty responses caused by reasoning token budgets - #537
Merged
Conversation
Reasoning models (e.g. GLM via Bifrost) can spend the entire max_tokens budget on chain-of-thought and return HTTP 200 with content: "" and finish_reason: "length". cora never read finish_reason or reasoning_content and silently fed an empty string to the JSON parser, surfacing the misleading 'EOF while parsing a value at line 1 column 0' — while the model's actual answer sat in the backend logs (cosy#63) (#536). Recovery now follows productivity over token frugality: - response parsing captures finish_reason and reasoning_content - empty content + finish_reason=length auto-retries with doubled budget (4096 -> ... -> 32768 ceiling) inside chat_completion - last resort: JSON-looking reasoning text is salvaged as the raw response; the parse layer still validates it - parse layer reports an explicit 'provider returned an EMPTY response' instead of serde EOF noise - default max_tokens raised 4096 -> 8192 for reasoning headroom Regression tests: budget escalation table, reasoning salvage (string/parts/fenced/prose), explicit empty-raw error. Signed-off-by: ajianaz <ajianaz@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #536
What
ChatResponsenow capturesfinish_reasonandreasoning_content(string or parts array) from LLM responsesfinish_reason=lengthtriggers an automatic retry with doubledmax_tokens(…→32768 ceiling) insidechat_completionmax_tokensraised 4096 → 8192Why
Reasoning models (e.g. glm/glm-5.3 via Bifrost) can spend the entire output budget on chain-of-thought and return HTTP 200 with
content: ""andfinish_reason: "length". cora never read those fields, silently fed an empty string to the JSON parser, and surfaced the misleadingEOF while parsing a value at line 1 column 0— while the model's actual findings JSON sat in the backend logs (codecoradev/cosy#63). The old retry re-sent the same 4096 budget, guaranteeing identical failure.Testing
length/stop/none × 32768 ceiling), reasoning salvage (plain JSON / fenced / parts array / prose / none), explicit empty-raw parse errorcargo fmtclean; no new clippy findings vs developAll commits signed-off (DCO).