HTTP 400 (INVALID_REQUEST) on every turn after a tool result contains lone UTF-16 surrogates

[cattfan]

Symptom

Every new turn in a session fails immediately with:

This turn failed: DeepSeek API error (HTTP 400)   [code: INVALID_REQUEST, status: 400]

It starts happening after one particular step and then every later turn of that session fails, even after resuming the session.

Reproduction

1. Run the Web GUI (dsh web) with llm-deepseek pointed at https://api.deepseek.com, model deepseek-v4-pro, reasoning_effort: max.
2. Have the agent run a tool that prints raw binary-derived text (e.g. a tool decodes dex/APK bytes as UTF-8 and dumps string tables) so the tool result contains lone UTF-16 surrogates (unpaired \uD800–\uDFFF, e.g. \ud980, \ud9bf).
3. Continue the conversation. The next request that replays that tool result (and every request after) returns HTTP 400.

Root cause

The adapter serializes the body with JSON.stringify (packages/llm/llm-deepseek/src/adapter.ts). A lone surrogate in a message string becomes a \uXXXX escape — valid JSON per Node's parser, but the DeepSeek API's JSON parser rejects unpaired surrogate escapes with HTTP 400 and a non-JSON body:

Failed to parse the request body as JSON: messages[89].content: unexpected end of hex escape at line 1 column 753189

Confirmed with a minimal request:

POST /chat/completions  {"messages":[{"role":"user","content":"\ud980 hi"}]}
-> 400  Failed to parse the request body as JSON: messages[0].content: unexpected end of hex escape

Because the error body is plain text (not JSON), adapter.ts falls back to the generic DeepSeek API error (HTTP 400) message, which is why the real reason is invisible. Since the poisoned message sits durably in the session log, the whole session is bricked — the same failure as the earlier content or tool_calls must be set null-content bug.

Expected behavior

• A lone surrogate must never reach the wire: replace each unpaired surrogate with U+FFFD during serialization.
• The adapter should surface non-JSON provider error bodies (e.g. the real parse error) instead of the generic HTTP-status fallback, so the true cause is visible.

Proposed fix

packages/llm/llm-deepseek/src/serialize.ts: add sanitizeSurrogates() (replaces lone high/low surrogates with U+FFFD, preserves valid pairs) and apply it to every string that goes on the wire (user/system/tool content, reasoning_content, tool-call arguments, tool descriptions). Verified live: the previously-failing request returns HTTP 200 and streams after the fix.

I have the change ready locally (source + tests + rebuilt bundle) and can open a PR if useful.

[hungnnvidia]

Proposed fix (fork PR): hungnnvidia#3 — sanitize unpaired UTF-16 surrogates to U+FFFD during wire serialization so sessions are not bricked by HTTP 400.

[hungnnvidia]

Proposed fix (fork PR): hungnnvidia#3 — sanitize unpaired UTF-16 surrogates to U+FFFD during DeepSeek wire serialization.

[weijiafu14]

Cross-route A/B for exactly this poison (lone UTF-16 surrogate in history), which corroborates where the fix belongs and offers an unlock for stuck sessions:

We injected an assistant history message containing a lone high surrogate (\ud83d, a slice()-cut emoji — same family as your tool-result case) into the real pi-ai route and sent it to the real api.deepseek.com (request body copied, forwarded verbatim): HTTP 200, follow-up turn completes (case loneSurrogate, results). Same-family report on the deepseek-official route observes deterministic 400s: #967. Same dirty input, one serializer produces a body DeepSeek's strict parser accepts, the other doesn't — so the fix is squarely serialization-layer (well-forming before send), not "the API just won't take it".

For a session already bricked by this: re-routing it through a declared llm-pi-ai provider (recipe) gets the history sending again — a downgrade from "session permanently dead" to "switch route and continue", not a root fix.

Scope note: our injection is the minimal single-lone-high-surrogate shape; if your tool result carries other malformed sequences, verify against your reproduction.