dsh_session_log: one lone surrogate in stored session text invalidates the whole request body (plain-text 400, detail swallowed)

[JL-Fei]

Environment

• dsh 0.2.1-rc.5 (npm global install), Windows 11
• Provider: deepseek-official (Messages API, https://api.deepseek.com/anthropic/v1/messages)
• Session: ~500k tokens of history, tools enabled, dsh_session_log extension active

Symptom

Every request in one long-running session fails with HTTP 400, and the UI only shows:

DeepSeek Messages request failed (400)

No detail, no request id, retries never help. The same session works fine on another provider route.

The actual response body

Captured at the wire (the body is plain text, not JSON):

Failed to parse the request body as JSON: dsh_session_log.events[21].data.message.content[0].text: unexpected end of hex escape at line 1 column 1922116

Root cause

dsh_session_log ships stored session events verbatim into the request body. One stored tool/result text ends with a lone high surrogate (in my case an emoji that an upstream memory-search tool truncated at a UTF-16 code-unit boundary, e.g. hit.text.slice(0, 600)).

JSON.stringify serializes a lone surrogate as \uD8xx. That is syntactically valid JSON for JavaScript, but Rust serde_json rejects it — and because it sits inside the single request body, the entire request is rejected, messages and tools included.

Minimal reproduction

// 400: Failed to parse the request body as JSON: messages[0].content[0].text:
//      unexpected end of hex escape
const lone = "abc ## \uD83D";            // lone high surrogate
// 200: same text with U+FFFD instead
const fixed = "abc ## \uFFFD";

Same payload, only replacing the lone surrogate with \uFFFD (and a bogus model to keep the check cheap), flips the response from Unexpected end of hex escape to a normal model not found error — i.e. the body parses again.

Why it is hard to diagnose

providerError() in dsh-llm-deepseek expects { error: { message } }. When the provider answers with plain text (or any non-JSON error shape), the message degrades to the generic DeepSeek Messages request failed (400) and the parser's real explanation (unexpected end of hex escape …) is thrown away. The user sees "400" with no way to learn why.

Suggested fixes

1. Sanitize on the way out: when building extensions.payload (or in wireEvent() for dsh_session_log), replace unpaired surrogates with U+FFFD before JSON.stringify, so one bad character in history cannot poison the whole request. The JSON text is what is sent, so this is the only place that can fix already-poisoned sessions.
2. Keep the provider text: surface the raw response body in the LlmError message when it is not JSON (currently the cause text is lost entirely).
3. Audit truncation helpers: any String#slice/substring cut of user or tool text should be code-point safe ([...text].slice(0, n).join("")). dsh-compaction-tool-result-pruner already does this correctly; the memory-search path that produced my data did not.

Impact

Any long session whose history contains a truncated emoji becomes permanently unusable on the Messages route, with no user-visible explanation — the session has to be abandoned or moved to another provider.

[AlaIchhe]

Independent report on a different platform (Debian/Linux, dsh 0.2.1-rc.2), same root cause — posting here to confirm it is not Windows- or rc.5-specific, plus one extra detail about why the session never recovers.

Environment

• dsh 0.2.1-rc.2 (npm global install), Debian, provider deepseek-official / deepseek-flash (Messages API)
• Long-running session: 164 turns, ~223k input tokens (well under the 1M context window, so not an overflow)

What happened

• Turn 161 completed normally.
• In turn 162 step 1 a run_code → multi_search subcall returned a scraped snippet with a truncated emoji. The stored text ends up with a lone high surrogate U+D83D (offset 2487 in that event).
• The model-facing tool/result event happened to be clean, but the log-only tool/ptc-dispatch event (seq 11425) stored the raw text verbatim.
• From turn 162 step 2 onward every request failed with 400; turns 163 and 164 (plain "continue" messages) also failed. Retrying never helps.

Why the session never recovers

The incremental upload watermark only advances on HTTP 2xx (session-log-deepseek/delivery-accepted). In our log the last acceptance is throughSeq=11420 and the poisoned event is 11425, so every later request re-sends events 11421 onward and dies at JSON parse time before the model is called. accept() never runs, so the cursor stays put forever — the session is bricked, not merely "retryable".

Controlled reproduction against the real endpoint

Same credentials, minimal body, one variable changed:

body	result
dsh_session_log contains "...\uD83D\n- [LSPosed..."	400 Failed to parse the request body as JSON: dsh_session_log.events[0].text: unexpected end of hex escape at line 1 column 4458
byte-identical, lone surrogate replaced by U+FFFD	200, normal completion
full conversation messages (~224k tokens), without dsh_session_log	200 — messages, tools and system prompt are all fine

Note the serialization path: prepareRequestExtensions() does JSON.stringify({ ...body, ...fields }), so the base request and the extension share a single serialization. Sanitizing only messages on the way in would not have prevented this; the payload itself needs to be well-formed.

Workaround we used

session-log-deepseek.config.enabled: false in the profile patch (or Settings → General → "Upload Session Log when using the official model API"). The next request omits the field and the session continues; the pending suffix stays behind the watermark until the stored event is repaired.

Suggestion

Alongside sanitizing unpaired surrogates in wireEvent() / the final payload, it may be worth having the uploader skip (and warn about, like the existing maxBytes block path) an event it cannot represent, instead of failing the model request. This is adjacent to the documented "Provider rejection fails the request" limitation, but here the rejection is caused by client-side serialization, so it is preventable rather than an unavoidable provider constraint.

Happy to test a patch against 0.2.1-rc.2 if that helps.

[sandiyochristan]

Here is a tested fix and unit test for both parts of this issue (sanitizing the payload and preserving the raw provider error message).

Cause and Tested Solution

1. Unpaired surrogates serialized into the wire payload:
   In packages/llm/llm-deepseek/src/request-extensions.ts, prepareRequestExtensions() serializes the merged request body and extension fields using standard JSON.stringify(). In JavaScript, JSON.stringify serializes lone UTF-16 surrogates as literal escape sequences like \ud8xx. While valid in ECMAScript, strict UTF-8 JSON decoders (such as Rust serde_json on the Messages gateway) reject these sequences with HTTP 400 (unexpected end of hex escape).

   Using toWellFormed() via a replacer callback during serialization automatically sanitizes any unpaired surrogates in body or extensions.fields (including dsh_session_log) to \uFFFD (Unicode Replacement Character) without allocating intermediary copies of clean strings.

2. Swallowed plain-text error diagnostics:
   In packages/llm/llm-deepseek/src/adapter.ts, when a gateway responds with non-JSON plain text (such as the gateway's Failed to parse the request body as JSON...), JSON.parse(text) fails, leaving raw undefined. providerError(raw, ...) then degrades to DeepSeek Messages request failed (400), discarding the explanatory text in cause.

   Passing raw ?? text into providerError() in adapter.ts and preserving the raw text in transport.ts surfaces the actual gateway failure directly in the error message.

Complete Tested Patch

diff --git a/packages/llm/llm-deepseek/src/adapter.ts b/packages/llm/llm-deepseek/src/adapter.ts
index df35b6ce3b..b47689086b 100644
--- a/packages/llm/llm-deepseek/src/adapter.ts
+++ b/packages/llm/llm-deepseek/src/adapter.ts
@@ -160,7 +160,7 @@ export class DeepSeekAdapter<C extends Connection = Connection> extends LlmAdapt
           }
           const detail = providerErrorDetail(raw)
           if (await files.retry(detail)) continue
-          const failure = providerError(raw, response.status, response.headers)
+          const failure = providerError(raw ?? text, response.status, response.headers)
           const message = files.errorMessage(response.status, failure.message, detail)
           throw new LlmError(message, failure.code, { ...failure.failure, cause: new Error(text) })
         }
diff --git a/packages/llm/llm-deepseek/src/request-extensions.ts b/packages/llm/llm-deepseek/src/request-extensions.ts
index be6ca7c9c8..db0236a0fb 100644
--- a/packages/llm/llm-deepseek/src/request-extensions.ts
+++ b/packages/llm/llm-deepseek/src/request-extensions.ts
@@ -6,6 +6,9 @@ import type { DeepSeekLlmApiExtensionRequest, PreparedDeepSeekLlmApiExtensions }
 import type { DeepSeekAdapterOptions } from './types.ts'
 
+const wellFormedJsonReplacer = (_key: string, value: unknown): unknown =>
+  typeof value === 'string' ? value.toWellFormed() : value
+
 /**
  * Merge contributions without replacing Messages fields. A rejected preparation or
@@ -39,10 +42,10 @@ export async function prepareRequestExtensions(
   }
   let payload: string
   try {
-    payload = JSON.stringify({ ...body, ...extensions.fields })
+    payload = JSON.stringify({ ...body, ...extensions.fields }, wellFormedJsonReplacer)
   } catch (error) {
     // A base request that cannot serialize throws here, before any field is reported as omitted.
-    const base = JSON.stringify(body)
+    const base = JSON.stringify(body, wellFormedJsonReplacer)
     onOmitted(fields, error)
     return { payload: base, accept: () => Promise.resolve() }
   }
diff --git a/packages/llm/llm-deepseek/src/transport.ts b/packages/llm/llm-deepseek/src/transport.ts
index 935e8855df..70615611c0 100644
--- a/packages/llm/llm-deepseek/src/transport.ts
+++ b/packages/llm/llm-deepseek/src/transport.ts
@@ -22,7 +22,12 @@ export function providerErrorDetail(raw: unknown): string {
 export function providerError(raw: unknown, status: number | undefined, headers?: Headers): LlmError {
   const envelope = typeof raw === 'object' && raw !== null ? raw as Record<string, unknown> : {}
   const error = typeof envelope.error === 'object' && envelope.error !== null ? envelope.error as Record<string, unknown> : {}
-  const message = typeof error.message === 'string' ? error.message : `DeepSeek Messages request failed (${status ?? 'stream error'})`
+  const rawText = typeof raw === 'string' ? raw.trim() : ''
+  const message = typeof error.message === 'string'
+    ? error.message
+    : rawText.length > 0
+      ? `DeepSeek Messages request failed (${status ?? 'stream error'}): ${rawText.length > 200 ? `${rawText.slice(0, 200)}...` : rawText}`
+      : `DeepSeek Messages request failed (${status ?? 'stream error'})`
   const type = typeof error.type === 'string' ? error.type : ''
   const detail = `${type} ${typeof error.code === 'string' ? error.code : ''} ${message}`
   let code: string
diff --git a/packages/llm/llm-deepseek/tests/extensions.spec.ts b/packages/llm/llm-deepseek/tests/extensions.spec.ts
index 4b727a2ae5..625e890e7f 100644
--- a/packages/llm/llm-deepseek/tests/extensions.spec.ts
+++ b/packages/llm/llm-deepseek/tests/extensions.spec.ts
@@ -195,4 +195,27 @@ describe('Messages request extensions', () => {
     expect(accept).not.toHaveBeenCalled()
     expect(warnings).toEqual([])
   })
+
+  it('sanitizes lone surrogates in request extensions and base body to well-formed unicode', async () => {
+    const ctx = await boot()
+    const accept = vi.fn()
+    ctx.deepseekLlmApiExtensions.register('dsh_messages_test', {
+      prepare: () => ({ value: { value: 'lone \uD83D surrogate' }, accept }),
+    })
+    const fetch = vi.fn<typeof globalThis.fetch>().mockResolvedValue(new Response(sse(textEvents)))
+    vi.stubGlobal('fetch', fetch)
+    const stream = ctx.llm.stream(options({
+      messages: [{ role: 'user', content: [{ type: 'text', text: 'prompt with lone \uDC00 surrogate' }] }],
+    }))
+    for await (const chunk of stream) {
+      if (chunk.type === 'finish') expect(chunk.reason.kind).toBe('stop')
+    }
+    const rawPayload = fetch.mock.calls[0]?.[1]?.body
+    if (typeof rawPayload !== 'string') throw new Error('Expected raw string payload')
+    expect(rawPayload).not.toContain('\\ud83d')
+    expect(rawPayload).not.toContain('\\udc00')
+    const parsed = JSON.parse(rawPayload) as { dsh_messages_test: { value: string }; messages: [{ content: [{ text: string }] }] }
+    expect(parsed.dsh_messages_test.value).toBe('lone \uFFFD surrogate')
+    expect(parsed.messages[0].content[0].text).toBe('prompt with lone \uFFFD surrogate')
+  })
 })

Verification

• Tested with pnpm vitest run packages/llm/llm-deepseek/tests/extensions.spec.ts: 10/10 passed.
• Tested with pnpm vitest run packages/llm/llm-deepseek/tests/adapter.spec.ts & stream.spec.ts: 93/93 passed.
• Pre-commit Lefthook and Oxlint gates pass with 0 errors.

[JL-Fei]

Thanks — adopted, and verified on 0.2.1-rc.5.

I had originally sanitized the serialized payload with a regex over \uD8xx escapes. Your toWellFormed() replacer is strictly better in a way that matters here: the regex also rewrote literal \ud83e text in message content (e.g. our own notes describing this bug), while the value-level replacer leaves it untouched.

Local results after patching prepareRequestExtensions + providerError the way you described:

• lone high surrogate -> U+FFFD, payload parses
• lone low surrogate -> U+FFFD
• valid emoji pair -> untouched
• literal \ud83e in text -> untouched
• plain-text 400 -> DeepSeek Messages request failed (400): Failed to parse the request body as JSON: dsh_session_log.events[21].data.message.content[0].text: unexpected end of hex escape at line 1 column 1922116

The last point is the one I care about most: with the raw text preserved, a gateway parse failure is self-diagnosing instead of a bare 400.