[Stall investigation] Streaming turns stall across 4 unrelated channels; pseudo-completed & missing turn/end in agent-loop — full forensic report

[Dhuahai]

Posted on behalf of the affected user (Dhuahai). All findings below were produced by the harness agent itself over 4 days of forensics (2026-11-05 to 11-08), with every claim tied to session-file events, proxy packet logs, or in-asar source reads. Full report follows.

[Draft] Streaming turns stall silently after the first chunk: idle watchdog + always-retry hang turns with turnEnds=null; cancel() drops queued messages

Summary

DeepSeek Harness intermittently hangs streaming turns: the upstream sends the opening chunk (block-start plus one or two thinking tokens) and then goes completely silent — no further chunks, no stream close, no error. After 300s an idle watchdog kills the silent stream, but retryPolicy mode:"always" retries indefinitely and every retry stalls the same way, so the turn never terminates; turnEnds stays null and the frontend shows an endless "stopping" spinner until the user manually cancels. The final turn record looks aborted-but-completed (interrupted:true while turn/end reason says completed in 97/102 cases), and manual cancel silently drops messages the user queued while the turn was hung. We have ruled out the upstream providers, the network stack, our context-compression plugin, and undici's default dispatcher timeout, and we ship five local patches we'd like upstream to consider.

Environment

• DeepSeek Harness 0.2.1-rc.2, Windows
• Multi-provider setup: several free and paid OpenAI-compatible upstream channels
• Intermittent, load-dependent; no stable minimal reproduction
• Suspected trigger: long multi-turn sessions with in-process concurrency (multiple sessions/subagents issuing model requests inside the same harness process)

Reproduction

No deterministic reproducer — bare probes against the same channels (see "What we ruled out") all complete normally. Observed in production sessions, with three symptom shapes:

• Thinking emits 1–2 tokens, then nothing; the turn hangs forever.
• Output stops mid-stream, goes quiet for minutes, then self-terminates as "stopped".
• A request that reads a large image spins for minutes and then dies.

In all shapes the UI shows "stopped", visually identical to a user-initiated stop, with no error surfaced anywhere. Frequency increases with long multi-turn sessions and concurrent in-process activity.

Evidence

Death chain (event layer):

1. Upstream emits block-start + 1–2 thinking tokens instantly, then the stream goes completely silent (no data, no close).
2. The stream-idle watchdog in dsh-llm-pi-ai (300s; logs "pi-ai stream idle timeout after 300000ms") kills the silent stream.
3. retryPolicy mode:"always" retries — but each retry stalls identically and burns another 300s, so the turn loops indefinitely instead of failing.
4. While this loop runs, turnEnds remains null; the frontend "stopping" state never resolves.
5. The user eventually cancels manually; cancel() calls inbox.clear(), silently discarding messages the user queued during the hang.
6. The recorded turn ends on the abort path, not the error path: the retry hook in agent-loop (L1130) only engages on finish.kind === "error" || "aborted", and the watchdog-killed termination shape records zero retries despite interrupted:true — a user-less abort-shaped termination that retryPolicy never covers.

Turn 106 (session 33dfa911) — the cleanest sample:

• assistant/message only arrives 507s after the turn starts.
• Body: 0 chars. Thinking: 2 chars. Tool calls: 0.
• interrupted:true, zero retry records.
• Precise timeline: turn/start → request/header (+272ms) → 507.7s silence → first chunk (block-start + reasoning "这位", 2 chars) → 4ms later assistant/message lands (interrupted) → turn/end completed. Zero attempts, zero retries.
• Key finding: the 300s idle watchdog does NOT protect the time-to-first-byte (TTFT). In the same session, turn 23 died to the watchdog (TIMEOUT) while streaming; turn 106 sat silent for 507.7s before the first byte and the watchdog never fired — because it starts counting from the first stream event. TTFT stalls have no guard at all; they end whenever the server (or an intermediate layer) finally answers with a fragment and immediately closes, which pi-ai classifies as cancel (no finish event) instead of error. Six such samples (turns 104–110) waited 22s/45s/46s/128s/507s — no fixed host-side timeout fits.

Abort vs error vs silent-stall — event-signature taxonomy (from a full 111-turn session, 3714 events):

	error path	user abort	silent stall
stream tail	finish{kind:"error",failure}	no finish	content chunks, no finish/usage chunk
assistant/attempt + llm/retry	present	absent	absent
assistant/message	not written on failed step	interrupted:true	interrupted:true, no usage
turn/end	completed (after retry succeeds)	{"kind":"aborted","reason":{"kind":"user"}}	{"kind":"completed"} (pseudo-completed)

The silent-stall column is the bug: a server-side close with no finish event is classified as cancel (user-shaped), which retryPolicy never covers and which writes a completed turn-end — so the record looks "done" and nothing ever retries. Note cancelCause does not exist anywhere in the schema; the cancel reason is inferred by pi-ai from the stream shape and is not recorded.

102 interrupted turns (fleet-wide stats):

• 46/102 turn-end timestamps land in 45.0–45.8s; the remainder cluster at larger values consistent with repeated timeout cycles (90/135/180/225/420/507s observed).
• 97/102 have turn/end reason written as {"kind":"completed"} despite interrupted:true — so from the session record these turns look "done" and never retry.
• Note: no event-layer component fires at 45s (the only relevant timer is the 300s idle watchdog). We treat the 45s cluster as a UI-perceived coincidence (likely several stacked timers), not a distinct 45s timeout layer.

What we ruled out

• Upstream providers: bare probe requests against the same channels — various payload sizes, malformed structures, tools, reasoning, protocol violations — all complete normally; symptoms are identical across every channel, including first-party vendor upstreams.
• Network stack: DNS/TCP/TLS all fast; no proxy in path.
• billion-context (context-compression plugin): no truncate/discard events logged.
• undici default dispatcher timeout (300s): after patching the global Agent to 900s, symptoms still reproduce — so it is not merely undici's default dispatcher timeout, though the patch reduces exposure.
• thinking-guard: evaluate() is not invoked while the stream is silent, so it cannot be the killer.

Transport-layer finding: response tail loss (new, 2026-11-07)

A response-side logging proxy (localhost :8402 in front of the OpenAI-compatible gateway) captured 218 POST /chat/completions streaming requests over ~9 hours. 212 got a stream END; 6 never did — three of the six timestamps match user-visible stalls to the minute:

REQ → RES 200 (headers@3.8s) → FIRST-BYTE → [body chunks flow] → ... no END, ever

The stalled requests received headers and partial body, then the body stream simply stopped while the connection stayed open — so the consuming client (pi-ai) waits on a stream that looks alive until the 300s idle watchdog kills it. The UI state this produces is "neither completed nor stopped". Whether the tail is dropped by the upstream, by the gateway (8326), or between the proxy and the harness is the last open hop; a third-level proxy can disambiguate, but the loss itself is already captured.

Pairing artifact to ignore when reading such logs: END total=12-27ms / chunks=2 / bytes=126694 / finish=null lines are GET /v1/models response bodies mis-paired onto POSTs (models keep-alive responses never close), not real late ENDs.

Transport-layer finding #2: random per-request stall, not size- or time-correlated (2026-11-08)

Response-side statistics over 84 streamed POSTs on one channel (freesharenow / deepseek-v4.1-flash, ~9h of real harness traffic):

• TTFT: min 6.1s, p50 20.0s, p75 36.1s, p90 62.9s, max 253.8s. Not one request under 2s.
• Spearman(body-size, TTFT) rho = 0.087 — payload size does not predict TTFT (buckets 0.5–0.9MB median 17.4s vs >1.2MB median 27.3s, no monotonic relation).
• Stall rate does not cluster in time (per-10-min buckets 15–39s median throughout), yet >30s TTFT occurs in 26/84 (31%).

A/B probe against the same channel + model within minutes (small ~30 tok / mid ~13KB / large ~130KB, two rounds each): S1 11.1s, M1 7.1s, B1 18.8s, S2 7.9s, M2 >120s with zero bytes (hung), B2 12.8s — small and large both fine, the mid request hung. Combined with the bare-probe results (small requests never reproduce), the data points to a per-request random stall in the upstream serving pool (bad LB node / queue), not payload size, not time-of-day, not the client.

Cross-channel rates: the first-party vendor channel (zcode/GLM via local gateway, 12 streamed POSTs) had median TTFT 5.4s with 1/12 over 30s (~8%); the reseller channel 31%. Same disease, different incidence — which matches the user-visible impression that "the official channel is just a bit more stable".

Client-side consequence (the part the harness owns): a zero-byte stall is indistinguishable from a dead stream, so the 300s idle watchdog kills it, the error overwrites whatever thinking content already arrived, and mode:always retries — drawing the same lottery again, which is exactly the "keeps disconnecting" loop users see. A stalled-but-recovering sample: 7 chunks → 184s silence → END total=239s finish=null done=true — the stream was slow, not dead; a 900s watchdog let it self-heal.

Mitigation that works: config-level 45s idle cap, hot-reloaded (2026-11-08)

The watchdog value is resolved through a zod profile schema (streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS)) — so it is settable per provider in the plugin patch YAML without touching the asar. The in-asar fallback source.streamIdleTimeoutMs ?? 9e5 (L1129) is dead code: the schema's .default() fills the value first. We set streamIdleTimeoutMs: 45000 (45s) on all 13 providers plus maxRequestImageBytes: 41943040.

Verified live with a fake upstream (a local server that accepts the first POST /chat/completions and holds it silent): the harness opened the stream at 22:43:06.947 and actively closed it after 44819ms (~45s), then immediately retried against the next provider — the full stall→fail-fast→retry loop now completes in under a minute instead of hanging for 300s+ per cycle. This converts "user stares at a frozen turn for 5-20 minutes" into "at most 45s per attempt, with visible retry activity". Note the config is hot-read per request (a baseURL change took effect on the very next request without restart), which also disproves "provider config requires a restart".

Trade-off: 45s kills slow-but-alive streams that would self-heal (the 239s sample above would now be cut at 45s and retried). With mode:"always" retry this is usually fine, but for channels with very slow TTFT (p90 60s+) a 45s cap trades availability for responsiveness — per-provider tuning is therefore essential, and is now possible.

Symptom-shape confounder: the reseller silently rewrites the model (2026-11-08)

Requests sent as model=deepseek-v4.1-flash to the reseller channel came back with usage blocks naming four different backends: sn-deepseek-v4-1-flash-1017 (SenseNova), DeepSeek-V4.1-Flash, DeepSeek-V4-Flash, and glm-5.3-flash (Zhipu). Prompt-token accounting differs by ~20% between backends (170708 vs 205883 for the same request — different tokenizers). The three symptom shapes the user reported are therefore not necessarily three client behaviors: the same stall death chain landing on different backends produces different-looking skins (GLM and DS thinking cadence differ). Anyone debugging "three shapes" against a reseller should check the usage-block model field first. (Also observed: vip/* model names bypass the subscription and require a separate paid wallet — a 403 with exact price-per-request in the message; the reseller's shared pool is the only lane a subscription covers, which increases its load and stall rate.)

Separating an unrelated incident: regional packet loss on one reseller (2026-11-08)

During the same window the reseller's endpoint (freesharenow.cc.cd, 93.177.76.157) became unreachable by region: direct TCP connect from a CN residential IP timed out on 443/80, traceroute died after the China-Mobile backbone hop, while the same endpoint via an overseas proxy connected fine (TCP 0.2s / TLS 0.9s / HTTP 200). Site web login also failed from CN. We initially suspected this was the stall root cause; it is not — it is a separate regional-routing incident at one reseller. We flag it because it produces "channel down" errors (connect ETIMEDOUT, read ECONNRESET) that look like the stall disease in logs. Distinguish by: regional packet loss fails at connect/TLS (fast, deterministic), while the stall disease gets headers + partial body then silence (slow, random).

Updated ruled-out list (post 11-08)

• Proxy layer (8401/8402): the two logging proxies were suspected of polluting the shared dispatcher via half-open keep-alive connections — disproved: qnaigc (which bypasses both proxies, direct connection) stalls identically, and a decisive control showed the harness completing 14 healthy proxied requests on 8401 during the exact 209s window one qnaigc request sat silent.
• Process-wide resource saturation: 6s CPU sampling showed no process near a full core (max 12.2% of one core); the same 209s window proves the event loop, connection pool and egress were all healthy while one request was stalled — the stall is per-request, not process-wide.
• Connection-pool exhaustion: undici Agent/Pool are per-origin with no connection cap in this version; the harness held only ~17 established sockets total (16 loopback).
• Network layer: TCP retransmission rate 0.37%, DNS 6-14ms, TCP handshake 42ms to the stalling channel; IPv6 and IPv4 both healthy; ephemeral port space (16384) far from exhaustion.
• Four symptom variants mapped to one chain: ①"completed+stopped, thinking only" ②"stopped only, thinking cut mid-stream" ③"body bubble shown then stopped" ④"memory-reflection turn visibly executed but vanished". All four are the same stall chain surfaced at different cut points (TTFT / mid-thinking / post-body) plus the pseudo-completed turn-end. ④ additionally involves the swallowed-reply shape above (turn archived, UI never shows it).

Proposed fixes / workarounds

All five applied locally (①–④ on 11-06/07, abortfix ⑤ on 11-08); symptoms reduced, though the underlying stall still occasionally reproduces. We'd like upstream to consider:

1. Raise / make configurable the stream-idle watchdog (300s → 900s) in dsh-llm-deepseek (L1127) and dsh-llm-pi-ai. A silent-but-open stream should not be killed at 300s by default; ideally the idle timeout is configurable per channel.
2. Raise the image request body cap 20971520 → 41943040 (two places) — large image payloads were a recurring trigger shape.
3. Null-guard the turn-end reason: writer side turnEnds ?? { kind: "completed" }, plus a reader-side null check in the frontend. Prevents reason:null from reaching session files (this null was also the trigger of a separate blank-screen incident on 2026-11-05).
4. Raise the global undici Agent timeout 300s → 900s.
5. Fix abortedCancelCause in dsh-agent-loop: the default: return assertNever(cause) branch throws when the abort signal carries a cancel cause outside the {kind:"user"|"parent"|"disposed"|"hook"} set (observed: bare Error / DOMException from internal teardown paths). The throw escapes the catch block before turnEnds is assigned, so finally writes turn/end {kind:"completed"} for a turn whose message is flagged interrupted:true — this is the pseudo-completed ("completed + stopped") shape above. Local patch: default: return { kind: "aborted" }; (equal-length byte rewrite, asar offset 17313979). Suggest upstream classify unknown cancel causes as aborted instead of asserting.

Notes

• The "45 seconds" users report is a UI-perceived coincidence; no component in the event layer acts at 45s.
• cancel() calling inbox.clear() silently drops queued user messages. Suggest not clearing the inbox on cancel, or making the behavior configurable — turning a hang into silent data loss is worse than the hang itself.
• The idle watchdog does not guard TTFT (see turn 106 finding). Suggest also arming the watchdog from request/header time, or adding a separate TTFT timeout, so a 500s pre-first-byte stall can't pass unguarded.
• Documented behavior request: concurrent subagents inside one harness process can trip upstream account-level concurrency limits (429 user concurrency limit exceeded) whose retries burn through mode:"always" backoff for minutes; this interacts badly with the watchdog loop above.
• Non-stream error responses also go through the stream-wait path. A probe receiving an immediate 401 (application/json) is delivered cleanly by the logging proxy in 364ms, but the harness treats it as a stream and waits the full 300s idle timeout before surfacing pi-ai stream idle timeout after 300000ms. Suggest classifying non-200 application/json responses as immediate errors instead of entering the stream-consumption path.
• Watchdog-killed turns overwrite already-received thinking content. When the watchdog fires, the thinking tokens delivered before the stall vanish from the UI and are replaced by the timeout error. Suggest preserving partial stream content when a stall is terminated.
• No liveness indicator during a stalled-but-open stream. With a 900s watchdog the stream survives longer (good), but the user stares at a motionless turn for minutes. A "waiting on upstream" hint (e.g. elapsed time since last byte) would turn guesswork into information.
• 239 console errors during one working day (observed 2026-11-08, DevTools): the frontend logs hundreds of Failed to load resource: 500 (Service Unavailable) against relative dsh-api/... endpoints (i.e. UI → harness's own local web server, not the model channels) — dsc-v2/messages/..., dsc/session-registry/..., plus 404s on input/history and 5xx on llm/request/error|cancel, input/detach. User-confirmed timeline: these errors started appearing together with the streaming stalls and disappear/abate when stalls abate — they are downstream symptoms of stalled turns, not an independent defect. The likely mechanism: while a turn is stuck, the host process's internal API degrades (event-loop saturation or a broken status-sync path) and UI status-sync calls fail en masse with 5xx. Separately, a frontend slot error (LinearCache!: slot "settings.section" is not declared as a parent entry's children table must declare it!, stack through coordinators) appears at the top of the console — a plugin/skin registering against a slot the host's children table doesn't declare; possible contributor to the degraded state, cause unverified. One pre-existing symptom NOT tied to stalls — assistant replies themselves get silently swallowed (user-confirmed, observed since before the stall era, still ongoing): my turns complete ("stopped"/spinner resolves) but the reply text never reaches the user's UI. This is a loss on the assistant side, distinct from the queued-user-message loss above. Candidate mechanisms: (a) a turn whose reply was fully generated but which got classified onto the abort path (assistant/message with interrupted:true) before delivery; (b) frontend render/SSE loss where the message lands in the session file but the UI never displays it. Verification pending: check whether swallowed turns have an assistant/message event on disk (discriminates (a) from (b)).

Update (user-confirmed details): swallowed turns show a "completed" toast with zero body text and zero thinking panel — the turn is archived as normally finished with nothing delivered. Combined with the turn-106 finding (TTFT silence is unguarded; a stream cut before any finish event is classified as a normal end), this identifies the swallowed-reply shape as the mild form of the same stall death chain:

symptom	form	mechanism
"completed" + empty body + no thinking	mild	stream dies during TTFT silence → no finish → pseudo-completed
endless spinner	moderate	mid-stream silence → client waits for the 300s watchdog
"stopped" + thinking content wiped	severe	watchdog kills stream + error overwrites already-received content

One death chain, three skins — all rooted in upstream random stalls + pi-ai classifying finish-less stream ends as normal completion + no TTFT guard. (Distinct from channel-side true-empty responses, which do deliver a thinking panel.)

Fourth skin (2026-11-08, reproduced live): a turn shows neither "completed" nor "stopped" — blank turn, no status at all. Cause: turn/end never written (observed directly: the turn's last event in the session file was job/registered, no turn/end). Same turn-end bookkeeping family as pseudo-completed, one step further: the archive lacks the closing event entirely, so the UI has nothing to render.

Attribution update (2026-11-08 evening): four channels, cross-family simultaneity — root suspect moved onshore

The "upstream random stall" hypothesis above was overturned the same day by a fourth channel. Stalls were observed on four channels in one evening: freesharenow (reseller), qnaigc/Qiniu conatus-free (Qiniu's own fused model — verified by 1684 usage-block echoes of its own name, not a DeepSeek/Zhipu reseller), zcode official (first-party), and Qoder 8.3-Flash (Qwen family — no relationship to the DeepSeek or Zhipu serving pools). Decisive observation: any two of them stall in the same minute (user-controlled A/B: one window on the official vendor lane, another on the paid reseller — both froze simultaneously). Two independent random stall pools cannot explain minute-level simultaneity.

Revised attribution: primary suspect = a shared per-request path inside the single harness process (all sessions run in one host process; the only commonality across the four channels). Upstream LB random stalls are real (the A/B and TTFT data above stand) but are reclassified as parallel noise — they widen the exposure window, they don't explain the simultaneity. The local patches (45s fail-fast, 900s watchdog, abortfix) reduce the blast radius regardless of which layer is ultimately at fault; a third-level in-process request tracer is planned to close the last hop.