web_fetch: one unpaired surrogate from HTML truncation permanently breaks a session (HTTP 400)

[userszhangkefan]

Summary

A single web_fetch of an ordinary third-party page can permanently break a session. The tool result ends up containing an unpaired UTF-16 surrogate; JSON.stringify emits it as \ud83c in every later request body, the gateway rejects the request with a bare 400 INVALID_REQUEST (no error.message, so the UI can only show DeepSeek Messages request failed (400)), and because the character stays in session history every subsequent request fails the same way. The session is unrecoverable; only a new session works.

Environment

• DeepSeek Harness 0.2.0-rc.2 (nightly), Windows
• deepseek-official / deepseek-flash
• web-fetch-http defaults: maxBodyChars = 100_000

Reproduction

A purpose-built, deterministic page (no dependency on a third-party site whose HTML may change): http://weswms.art/slice_errors/

It is 100 053 UTF-16 code units long, and code unit 99 999 is the high surrogate of an emoji, so the default 100 000 cap always lands between the two halves of that character.

1. Open a new session.
2. web_fetch http://weswms.art/slice_errors/ → HTTP 200, 4758 characters, rendered markdown ends with [\ud83c](...).
3. Send any message → {"message":"DeepSeek Messages request failed (400)","code":"INVALID_REQUEST","status":400}, and the same on every later message.

Isolated: a fresh session running only pwsh → web_fetch (that URL) → pwsh fails on the third step.
Control: the same three steps against a page without an astral character at the boundary pass with no errors.

Root cause

packages/web/web-fetch-http/src/provider.ts:168 bounds the decoded body with

const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded

String.prototype.slice counts UTF-16 code units. When the cap lands between the two halves of an astral character, the body ends on an unpaired high surrogate.

packages/web/tool-web/src/fetch.ts has the same pattern twice more: the source cut at line 243 and the output cut at lines 335–336.

Proposed fix

Bound every cut on a code point boundary, and sanitize the model-facing text as a backstop. Diff against master (base blobs a6ddbe0 / f4a644d / 1a134f2 / ee1b478):

--- a/packages/web/web-fetch-http/src/provider.ts
+++ b/packages/web/web-fetch-http/src/provider.ts
@@ -166,7 +166,7 @@
     const decoded = decoder.decode(bytes)
     const truncatedByChars = decoded.length > this.limits.maxBodyChars
-    const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded
+    const content = truncatedByChars ? truncateText(decoded, this.limits.maxBodyChars) : decoded

--- a/packages/web/tool-web/src/fetch.ts
+++ b/packages/web/tool-web/src/fetch.ts
@@ -240,7 +240,7 @@
 function renderBody(body: WebFetchBody, maxInputChars: number): RenderedBody {
-  const content = body.content.slice(0, maxInputChars)
+  const content = boundText(body.content, maxInputChars)
@@ -332,9 +332,11 @@
   const truncated = result.truncated || rendered.sourceTruncated || prefix.length > maxOutputChars
   const full = `${prefix}${truncated ? TRUNCATION_FOOTER : ''}`
-  if (full.length <= maxOutputChars) return { text: full, truncated }
-  if (maxOutputChars < TRUNCATION_FOOTER.length) return { text: full.slice(0, maxOutputChars), truncated }
-  return { text: `${prefix.slice(0, maxOutputChars - TRUNCATION_FOOTER.length)}${TRUNCATION_FOOTER}`, truncated }
+  if (full.length <= maxOutputChars) return { text: wellFormed(full), truncated }
+  const bounded = maxOutputChars < TRUNCATION_FOOTER.length
+    ? boundText(full, maxOutputChars)
+    : `${boundText(prefix, maxOutputChars - TRUNCATION_FOOTER.length)}${TRUNCATION_FOOTER}`
+  return { text: wellFormed(bounded), truncated }

/** Cut to at most `max` UTF-16 code units without splitting a surrogate pair. */
function truncateText(text: string, max: number): string {
  if (text.length <= max) return text
  const last = text.charCodeAt(max - 1)
  const next = text.charCodeAt(max)
  const splitsPair = last >= 0xd800 && last <= 0xdbff && next >= 0xdc00 && next <= 0xdfff
  return text.slice(0, splitsPair ? max - 1 : max)
}

const LONE_SURROGATE = /[\uD800-\uDFFF]/u // `u`: a well-formed pair is one astral code point

/** Replace unpaired surrogate code units with U+FFFD (backstop for remote providers). */
function wellFormed(text: string): string {
  if (!LONE_SURROGATE.test(text)) return text
  let sanitized = ''
  for (const character of text) {
    const code = character.codePointAt(0) ?? 0
    sanitized += code >= 0xd800 && code <= 0xdfff ? '\uFFFD' : character
  }
  return sanitized
}

boundText is the same helper under tool-web's local name; two regression tests come with it (fetch-http.spec.ts: a body capped mid-pair must not end on an unpaired surrogate; tool-web.spec.ts: formatFetchOutput must never emit one).

Full branch: https://github.com/userszhangkefan/deepseek-harness/tree/fix/web-fetch-lone-surrogate

Why the session dies (and what recovery takes)

The bad code unit is stored in the session log, not just generated on the fly. Reopening the session re-serializes it into every request, so the only recovery is rewriting the stored transcript (replace the escape with \uFFFD). Worth keeping in mind for users who hit this: new sessions work, the poisoned one never recovers on its own.

Two smaller issues this surfaced

1. The error is undiagnosable. providerError() falls back to DeepSeek Messages request failed (<status>) whenever the response body has no error.message, which makes a malformed request body indistinguishable from a model-side rejection. Keeping the response body (or x-deepseek-request-id) on the failure event would have pointed at the real cause immediately.
2. Nothing sanitizes tool results at the boundary. A single toWellFormed() before a tool result enters history would cap the blast radius of every provider, including ones that cannot be patched.

Interim mitigation I am running locally

Until a release carries the fix, a one-line change to the packaged @deepseek-ai/dsh-tool-web keeps the session alive:

 function formatFetchOutput(result, maxOutputChars) {
-  return renderFetchOutput(result, maxOutputChars).text;
+  return renderFetchOutput(result, maxOutputChars).text.toWellFormed();
 }

With it, fetching the reproduction page yields a U+FFFD in the markdown instead of a lone surrogate, and the next request succeeds (verified end to end on 0.2.0-rc.2). It is a personal workaround for this install, not a substitute for the source fix.

Handover

CONTRIBUTING says external pull requests are not accepted right now, so this is posted here. The branch above is ready to open as a PR the moment that changes, or the diff can be applied directly. Happy to adjust the shape of the fix to whatever the codebase prefers.

[yingcai233333]

补充一条链接：这条与 #11476、#11831 是同一个根因（一次 UTF-16 码元切把代理对劈开）。

#11831 上整理了：

• 仓库里已有的安全实现 truncateWithoutSplittingSurrogatePair（PR 今天拉去最新代码后，重新安装构建，运行后抛出 '@deepseek-ai/dsh-llm' does not provide an export named 'CallId' 错误 #4827），以及仍然没接上它的 read-render.ts:70
• 一行级修复建议（工具结果落盘前 toWellFormed()、请求序列化加 replacer）
• 一份实测有效的临时兜底（会话日志监视器，约 3 秒就地修复）

如果 web_fetch 的 HTML 截断有自己的切片点，也值得一并检查是否走码元切。