Skip to content

server: surface unclosed thinking as reasoning_content, not content#524

Open
stefanwichmann wants to merge 1 commit into
antirez:mainfrom
stefanwichmann:fix/unclosed-thinking-reasoning-content
Open

server: surface unclosed thinking as reasoning_content, not content#524
stefanwichmann wants to merge 1 commit into
antirez:mainfrom
stefanwichmann:fix/unclosed-thinking-reasoning-content

Conversation

@stefanwichmann

Copy link
Copy Markdown

Fixes #509.

Problem

When thinking mode is on and generation stops before </think> (e.g. the response is truncated at max_tokens mid-thought), the non-streaming path in parse_generated_message_ex routed the partial chain-of-thought into content (via split_reasoning_content) with an empty reasoning_content. Clients received raw, unfinished CoT as if it were the assistant's final answer. The live streaming classifier already handles this correctly — only the blocking path was affected.

Fix

In the require_thinking_closed && !think_end branch, treat the accumulated text as unfinished reasoning: surface it as reasoning_content and leave content empty (stripping a leading <think>). Any DSML in that text is unclosed reasoning too, so it's deliberately not executed as a tool call — composing with the existing unclosed-think handling from #318.

Testing

  • Adds test_thinking_unclosed_is_reasoning_not_content.
  • ./ds4_test --server passes.
  • Built with make cuda-spark and validated live on a DGX Spark (GB10): truncated non-streaming responses now return empty content + populated reasoning_content; completed responses unchanged.

Single file, +35/−3.

When thinking mode is on and generation stops before </think> (e.g. the
response is truncated at max_tokens mid-thought), parse_generated_message_ex
dumped the partial reasoning into content with an empty reasoning_content, so
clients received raw chain-of-thought as the assistant's answer.

Route the accumulated text to reasoning_content and leave content empty,
matching what the live streaming classifier already does. This composes with
the existing tool-call handling in the same branch: unclosed DSML is still
treated as reasoning rather than executed.

Adds a regression test (test_thinking_unclosed_is_reasoning_not_content).

Fixes antirez#509.
@Smallfu666

Copy link
Copy Markdown

Independent DGX Spark validation for this PR.

Test environment:

  • NVIDIA DGX Spark / GB10 (sm_121)
  • Ubuntu 24.04.4 LTS, ARM64
  • Kernel 6.17.0-1026-nvidia
  • CUDA 13.0
  • NVIDIA driver 580.159.03
  • DeepSeek V4 Flash Q2 imatrix
  • Baseline: main at 80ebbc396aee40eedc1d829222f3362d10fa4c6c
  • PR head: 76573b091887d18d2aa3066f7d6da1593191764b

Baseline reproduction:

  • A truncated non-streaming thinking response returned finish_reason: "length".
  • Partial reasoning was exposed through message.content.
  • reasoning_content was empty or absent.

Validation results with this PR:

  • make cuda-spark: PASS
  • ./ds4_test --server: PASS
  • Truncated non-streaming thinking response: PASS — content was empty and partial reasoning was returned through reasoning_content.
  • Completed non-streaming thinking response: PASS — final content and reasoning_content remained correctly separated.
  • Truncated streaming response: PASS — no reasoning leaked into content.
  • No-thinking non-streaming response: PASS — normal assistant output remained in content, without unexpected reasoning_content.
  • Unclosed thinking containing DSML-like tool-call text: PASS — no tool call was executed.

The baseline issue was reproducible on this machine, and the tested PR head corrected it without a regression in these five API paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Non-streaming /v1/chat/completions leaks thinking into content when truncated (finish_reason: length) before </think>

2 participants