Provider API: SSE stream ends silently mid-tool-call during large tool-call argument generation (no [DONE], no finish_reason)

[seanpham99]

Summary

Streaming chat.completions against the Provider API intermittently ends the SSE stream mid-tool-call while the model is generating a large tool-call JSON object. The stream stops cleanly — no [DONE], no finish_reason chunk, no error — with the tool-call arguments still incomplete. A client cannot distinguish this from a network drop.

Correlated with:

• Large tool-call argument payloads (~10k–50k chars, e.g. a write_file/terminal call carrying a long body)
• Large prompt context (~50k+ tokens)

Small tool-call arguments, plain text output of any length, and the same request in non-streaming mode all complete reliably. This looks like a streaming-path-specific drop in the upstream SSE implementation, not a hard output-token cap.

Expected Behavior

The SSE stream should run to completion ([DONE], finish_reason: "tool_calls") with complete JSON tool-call arguments ending in }, regardless of argument size.

Actual Behavior

The stream ends after a few thousand chunks with no terminator of any kind — no [DONE], no finish_reason chunk. The tool-call arguments are truncated mid-string. Total delivered arguments ~20–50KB (roughly 5–9k completion tokens), far below the requested max_tokens.

Steps to reproduce the issue

Send a streaming chat completion that forces the model to emit one big tool call:

{
  "model": "deepseek/deepseek-v4-flash",
  "messages": [
    {
      "role": "user",
      "content": "Call write_file with path '/tmp/big.txt' and content being a 20,000-character essay about Vietnamese coffee culture. The ENTIRE content must be inside the tool call arguments. Do not abbreviate."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "write_file",
        "description": "Write a file to disk",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {"type": "string"},
            "content": {"type": "string"}
          },
          "required": ["path", "content"]
        }
      }
    }
  ],
  "tool_choice": {"type": "function", "function": {"name": "write_file"}},
  "max_tokens": 131072,
  "stream": true
}

Python repro (stdlib only):

import json, urllib.request, os

key = os.environ["COMMANDCODE_API_KEY"]
url = "https://api.commandcode.ai/provider/v1/chat/completions"
body = {
    "model": "deepseek/deepseek-v4-flash",
    "messages": [{"role": "user", "content": "Call write_file with path '/tmp/big.txt' and content being a 20,000-character essay about Vietnamese coffee culture. The ENTIRE content must be inside the tool call arguments. Do not abbreviate."}],
    "tools": [{"type": "function", "function": {"name": "write_file", "description": "Write a file to disk", "parameters": {"type": "object", "properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}}],
    "tool_choice": {"type": "function", "function": {"name": "write_file"}},
    "max_tokens": 131072,
    "stream": True,
}
req = urllib.request.Request(url, data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json", "Authorization": "Bearer " + key})
with urllib.request.urlopen(req, timeout=300) as r:
    chunks = 0
    for raw in r:
        line = raw.decode("utf-8", "replace").strip()
        if line.startswith("data: "):
            line = line[6:]
        if line == "[DONE]":
            break
        chunks += 1
        print(line[:200])
    print("chunks:", chunks)

Expected: stream continues to [DONE] with finish_reason: "tool_calls", complete JSON args ending in }.
Observed (when it fails): stream simply ends after a few thousand chunks — no [DONE], no finish_reason chunk, tool arguments truncated mid-string.

Command Code Version

Provider API (gateway) — api.commandcode.ai/provider/v1/chat/completions, model deepseek/deepseek-v4-flash

Operating System

Linux

Terminal/IDE

OpenAI-compatible client (raw SSE reader)

Shell

No response

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context

• The non-streaming equivalent of the same request completes fine — a 190KB tool-call payload via stream: false succeeds. So any cap is in the streaming path only.
• The failure reproduces through a plain OpenAI-compatible client; no client SDK or router involved.
• Frequency appears to increase with prompt context size; the smallest repro used an ~8k-token prompt.
• This is distinct from Error: 500 Upstream stream ended before terminal chunk #628 and CLI reports a fabricated HTTP 500 when the response stream drops mid-flight #755 (both CLI-side handling of a mid-flight drop that already ended with an error). Here the stream ends silently with no terminator at all.

Happy to run additional repros (specific context sizes, different models in the catalog, non-SSE streaming) on request.

[seanpham99]

Follow-up: direct repro passes today — updating with honest findings

I ran the exact repro from this issue today against api.commandcode.ai/provider/v1/chat/completions (model deepseek/deepseek-v4-flash, stream: true, 20k-char tool-call request, browser UA):

Result: PASS — no drop.

• 3,973 chunks, finish_reason: "tool_calls", [DONE] received, ~18.5KB of tool-call arguments delivered, complete.

Also tested via an OpenAI-compatible relay (the environment where the drop was originally observed):

• Same 20k-char tool-call request, ~7k chunks, finish_reason: "tool_calls", complete args — no drop.
• Same request with a ~50k-token context prefix: 6,504 chunks, finish_reason: "tool_calls", complete args — no drop.

What this means

The silent mid-tool-call stream end is intermittent / environment-dependent, not a deterministic failure of a big tool-call request. The original observations came from a production agent harness at 53–64k prompt tokens, with multi-turn tool-call history (assistant tool_calls → tool results → next assistant turn), not from a single-shot request. I have not yet reduced it to a minimal standalone repro.

What I suspect (unverified)

• The drop may require a combination of: large context + multi-turn tool history + reasoning/thinking mode (deepseek-v4-flash has thinking on by default).
• I did hit a related but distinct 400 on the same model: "Thinking mode does not support this tool_choice" when sending an explicit tool_choice (that's the deepseek-v4-flash thinking-mode constraint, separate from this stream-drop issue).
• In the failing harness, the drop appeared as the stream ending with no [DONE] and no finish_reason — indistinguishable from a network/upstream cut, which is the hardest part to debug.

Request to maintainers

If you have internal visibility into deepseek/deepseek-v4-flash streaming (or upstream DeepSeek SSE), a likely culprit is the SSE/streaming path dropping the connection when reasoning + tool-call generation runs long at high context. Anything you can share about known streaming stability limits (context size × reasoning × tool-call interplay) would help.

I'll keep this open and will reduce to a minimal repro (large context + multi-turn tool loop) if I can catch it again; happy to provide trace IDs from the harness when it recurs.

[seanpham99]

Stronger repro found: drop correlates with prompt context size, not tool-call size

Catching up on the earlier thread: the silent mid-stream drop reproduced again in a live agent session today, and the trigger is now clearer.

What happened (production harness, model = deepseek/deepseek-v4-flash via an OpenAI-compatible relay):

• Session context reached ~150k–169k prompt tokens per request.
• Two API calls at 166k/169k in → 3.7k/4.2k completion tokens: OK.
• Immediately after, three consecutive streaming calls each emitted ~1 completion token and the stream ended with no finish_reason / no [DONE] (logged as "Stream ended with no finish_reason ... treating as a mid-stream drop").
• The client fell back to another provider and the turn eventually completed.

Relay usage history for the failed calls (all status "ok", i.e. no HTTP error, stream just ended):

• prompt=156150, completion=1
• prompt=156210, completion=1
• prompt=156270, completion=1

Correlation:

• Same model, same client, same tool-call shape — the only material difference between working and failing calls was prompt context size.
• The earlier 20k-char tool-call repro at ~8k context passes cleanly (recorded above).
• Failing calls were at ~156k+ prompt tokens. Working calls were at ≤169k in the same session, then it flipped to 1-token drops.

So the drop appears tied to context length, not output/tool-call size. DeepSeek's own context window for this model is large (claims 128k+), so 156k may exceed the effective upstream limit — the relay may be sending a request past the model's real window and getting a near-empty stream back instead of a 400.

Ask: could CommandCode clamp/reject requests that exceed the upstream model's actual context window instead of streaming 1 token and ending silently? A clear 400 context_length_exceeded (or a finish_reason/[DONE] with an error payload) would be dramatically easier for clients to handle than an indistinguishable stream cut.

I'll keep the earlier repro too — both are useful. Happy to add exact request headers/timestamps if you want them.