[BUG] dsh-llm-pi-ai: tool_call arguments O(n²) reparse in long sessions causes 500 Upstream gateway error

[Ansenlee-dock]

Summary

@deepseek-ai/dsh-llm-pi-ai's OpenAI Chat Completions adapter accumulates
streaming tool_calls[].function.arguments deltas into a single string,
then re-parses the entire accumulated string on every chunk using
partial-json. This is O(n²) in the length of arguments per tool call.

In long-running sessions (e.g., agent sessions that last hours and
accumulate large assistant message histories), this combines with the
session's growing messages payload to produce requests whose bodies
exceed provider gateway limits, causing 500/503 Upstream gateway error
responses from relay hosts such as relay.framefly.com.cn.

Environment

• DSH: 0.1.0-rc.6 (@deepseek-ai/dsh v0.1.0-rc.6)
• Bundle: @deepseek-ai/dsh-web-app
• Adapter: @deepseek-ai/dsh-llm-pi-ai
• Provider: a custom route registered via llm-pi-ai.providers.<route> using
  api: openai-completions against a relay that fronts OpenAI's
  chat/completions (e.g., relay.framefly.com.cn)
• Tested model: gpt-5.5 (and gpt-5.4-mini, gpt-5.6-sol, etc.)
• Provider confirms the LLM is healthy: 8/8 standalone curl calls succeed
  with sub-second TTFT and no 500 responses
• Provider: dsh-llm-deepseek against the same provider has no such failures

Symptoms

• Sessions running 1+ hours on openai-relay/gpt-5.5 see occasional
  turn/end kind=error events with reason
  {"code":"SERVER","message":"500: {\"message\":\"Upstream gateway error\",\"type\":\"api_error\"}"}
  or 503: {"message":"Service temporarily unavailable"}
• Frequency is low (we observed ~2 errors in 19,589 events on one session,
  ≈0.01%), but accumulates the longer the session runs
• Errors appear after many assistant turns, typically when the
  accumulated history plus tool_calls makes the request body large
• The companion adapter dsh-llm-deepseek on the same provider path
  shows no such errors in equivalent workloads

Reproduction

Direct LLM access (relay.framefly.com.cn) succeeds in 8/8 cases.

$ curl -s -X POST "https://relay.framefly.com.cn/v1/chat/completions" \
    -H "Authorization: Bearer …" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-5.5","messages":[…],"max_tokens":50}' | jq
# → 200 OK, "42"

The same call initiated through DSH via the llm-pi-ai provider eventually
returns a 500. The session's request.context event shows the same provider
and model are in use, and the body of the failing request is significantly
larger than a fresh session's (long history + many tool_calls).

Root cause

In @deepseek-ai/dsh-llm-pi-ai's OpenAI Chat Completions stream handler
(dist/api/openai-completions.js), tool_call argument deltas are concatenated
into a single string and re-parsed on every chunk:

for (const toolCall of choice.delta.tool_calls) {
    const block = ensureToolCallBlock(toolCall);
    …
    let delta = "";
    if (toolCall.function?.arguments) {
        delta = toolCall.function.arguments;
        block.partialArgs = (block.partialArgs ?? "") + toolCall.function.arguments;
        block.arguments = parseStreamingJson(block.partialArgs);   // ← O(n²) per chunk
    }
    …
}

parseStreamingJson invokes partial-json's parse() on the full
accumulated string, so the cost of streaming one tool_call with n
delta characters is O(n²) in n. Across hundreds of tool calls in a long
session, this is both slow and (more importantly) the streaming handler
holds on to ever-growing partialArgs strings per call, and the assembled
history sent to the provider is bloated by stringified reparses.

The harness Blockassembler already provides incremental block assembly
that tolerates delta-only protocols. The adapter does not need to
accumulate partialArgs at all — it could yield raw tool-call-delta
chunks and let the harness's assembler manage concatenation (this is
exactly what dsh-llm-deepseek does).

Workaround

Two practical mitigations until this is fixed upstream:

1. Use dsh-llm-deepseek for long sessions (it does not have this issue).
2. Add an explicit retryPolicy to the affected provider in
   ~/.dsh/settings.yaml so the harness's dsh-llm-retry recovers from
   the 500/503 instead of failing the turn:

   llm-pi-ai:
     providers:
       <route>:
         retryPolicy:
           mode: normal
           maxRetries: 5
           retryableCodes: [429, 500, 502, 503, 504]
           backoff:
             initialDelayMs: 1000
             maxDelayMs: 60000
             jitterRatio: 0.2

   Note: dsh-llm-pi-ai forces the inner pi-ai SDK's maxRetries to 0,
   so the retry must be configured at the harness layer.

Suggested fix

Replace the per-chunk partialArgs accumulator + parseStreamingJson
with raw tool-call-delta yields and let the harness Blockassembler
handle the concatenation. The adapter should yield one
tool-call-delta chunk per incoming tool_calls[i].function.arguments
delta, with index, id, name, arguments fields populated, and rely
on the harness to assemble the final arguments string for each
tool-call block (this is what dsh-llm-deepseek already does).

The same fix applies to the tool-call-delta accumulation that
pi-messages.js performs downstream of the adapter — once the adapter
stops accumulating, the entire pi-ai pipeline becomes delta-only and the
Blockassembler's deltas arriving for an index already closed by block-end are ignored (malformed stream) invariant holds.

Reference

• DSH harness: https://github.com/deepseek-ai/deepseek-harness
• Affected file: packages/llm-pi-ai/dist/api/openai-completions.js
• Reference implementation that does NOT have this issue:
  packages/llm-deepseek/lib/index.js (see async *stream + translatePayload)
• DSH Blockassembler (correct incremental assembly):
  packages/llm/lib/types/assembler.ts

---

Filed by: 老六 (AvatarV5)
Date: 2026-08-19
Affected version: @deepseek-ai/dsh-llm-pi-ai shipped with
@deepseek-ai/dsh v0.1.0-rc.6

[weijiafu14]

Independently reproduced with numbers — your O(n²) diagnosis holds, and it bites well before "long session" territory.

Measurement (fault injection at the wire only; the OpenAI client, SSE parser and accumulator under test are unmodified pi-ai 0.84.1): one tool call whose JSON arguments arrive split across N deltas, timing the whole stream.

chunks	argument chars	wall time	per-chunk
400	8,034	33 ms	0.081 ms
1,600	32,034	421 ms	0.263 ms
3,200	64,034	1,578 ms	0.493 ms

8× the chunks → 48× the time, and per-chunk cost rises with the accumulated length (0.081 → 0.493 ms) — the quadratic signature, exactly as you predicted. Harness + raw verdicts: args-reparse-scaling.mjs / results.

Two things worth adding to your report:

1. The threshold is low. A 64KB argument (an edit/write payload with a decent file body is easily that) already costs ~1.6s of pure CPU in the accumulator on a fast laptop — that's a stall in the middle of a stream, before any upstream slowness. Your "long sessions → 500 Upstream" symptom is plausible as the downstream effect: the read loop stops draining while the parser churns.
2. Cheap fix shape: the intermediate parse only exists to expose partial arguments while streaming. Parsing on a bounded cadence (every k chunks / every m bytes / on a microtask when idle) instead of every delta keeps the partial-preview feature and turns the cost linear; a full parse still happens once at block end.

Happy to re-run this harness against a candidate patch if it helps — it's self-contained (REPO=<repo> PI_AI=<pi-ai dir> node args-reparse-scaling.mjs) and prints the growth verdict directly.