Streaming-turn stalls: report v2 — attribution rewritten to the harness's own request path (always-retry loop, watchdog design, finish-less stream classified as normal end)

[Dhuahai]

[Draft v2 — attribution rewritten] Streaming-turn stalls are amplified (and largely produced) by the harness's own request path: always-retry loop, keep-alive reuse, finish-less stream classified as normal end, pseudo-completed turn-end

What changed from v1 (2026-11-08): v1 concluded "primary suspect = shared per-request path inside the harness process, upstream LB stalls as parallel noise". Between 11-09 and 11-11 we completed the differential test v1 could only hypothesize: the same channels, from the same machine and network, driven by other agent products, do not storm — including the vendor's own first-party client (Zcode) on the same Zhipu models. The upstream-stall hypothesis is now demoted from "parallel noise" to "background fact": vendor gateways have ordinary timeout lines, but the storm phenomenon is a property of the harness's request path. Evidence chain below; v1 archived as _upstream-report-draft-v1-20261108.md.

Summary

DeepSeek Harness intermittently "storms": a channel that is fine externally starts timing out every request inside the harness for tens of minutes (observed chains: 30 consecutive retries on nvidia official, 16 consecutive on iFlytek Spark official, plus earlier episodes on qnaigc/freesharenow/zcode/Qoder — six vendor families, including three first-party endpoints and the vendor's own subscription models). The same channels served from the same IP by other clients during the same windows showed no such pattern.

What the harness contributes, each item backed by a capture in this process:

1. retryPolicy mode:"always" converts a few bad minutes into a half-hour storm — every retry re-sends the same ~1 MB, 100+-tool payload; the retry chain itself is the storm. (nvidia: 30 retries spanning 00:37–01:24; spark: 16 retries in 15 minutes.)
2. Stream-idle watchdog design: armed only from the first byte (TTFT stalls unguarded — turn 106 waited 507 s before the first byte with no timer firing), and its default (300 s, later 45 s via config, now per-provider) is the wrong knob for "vendor gateway recycles long-idle streams". A stalled-but-recovering stream (7 chunks → 184 s silence → normal END at 239 s) gets killed at any cap below ~250 s, retried, and killed again — the watchdog manufactures the storm out of a slow stream.
3. A stream end with no finish event is classified as a normal completion → turnEnds=null → turn/end {kind:"completed"} while the message is flagged interrupted:true (97/102 interrupted turns recorded as "completed"). Records look done; nothing retries on the error path; the UI shows "completed + stopped" or, in the mild form, a completed turn with zero body and zero thinking (the long-standing "swallowed reply").
4. abortedCancelCause assertNever throw when the abort cause is a bare Error/DOMException — escapes the catch before turnEnds is assigned; local patch in place, upstream fix still recommended.
5. cancel() → inbox.clear() silently drops user messages queued during a hung turn.
6. Non-200 JSON error responses enter the stream-wait path (a 401 delivered in 364 ms by the proxy surfaced 300 s later as an idle timeout).
7. keep-alive connection reuse against a 1 MB/request, 100-tools, minute-scale-turn workload is our remaining un-narrowed suspect for why other clients with lighter shapes don't storm on the same vendors. The decisive single-request capture (proxy shows full body incl. END, harness reports idle-timeout on the same request) is still pending — stated honestly.

The differential tests (what finally moved attribution onshore)

• Same machine, same network, same channels, other agents: no storm. User-controlled comparison across products; the strongest instance: Zhipu's own Zcode client on the same zcode/GLM models never storms, while DSH's zcode line stalled repeatedly (11-11 morning). Two lanes do exist for Zhipu (subscription vs API gateway) — but see next bullet, which prices the lane.
• Bare probes, same lane, same request shape, from this process's own machine: sizes 20 K–360 K tokens, dirty structures, 102-tool payloads, reasoning params, protocol-pairing violations — all complete normally; occasional stalls are an order of magnitude rarer than inside the harness process. Same lane + same shape + only-difference = "inside DSH" ⇒ the amplification is process-side.
• Minute-level cross-channel simultaneity inside the harness only (11-08: official lane + paid reseller + free fused-model lane all froze in the same minute; other windows/clients kept working on the same channels at the same minutes on 11-11). Shared component = the single harness process.
• Upstream side is not innocent but is ordinary: vendor gateways do recycle idle streams (transport captures in v1 stand: response-tail loss, 31 % TTFT>30 s on a reseller, 8 % on first-party). The point is that these ordinary lines are only ever hit by this client's request path — lighter clients don't touch them, which is exactly what a client-side amplification finding looks like.

Per-provider watchdog economics measured live (config layer, no asar needed)

streamIdleTimeoutMs is settable per provider in the patch YAML (zod .default() resolves first; the in-asar ?? fallback is dead code) and is hot-read per request (verified with a fake upstream: closed at 44.8 s, no restart). Current fleet: 45 s default, raised to 75 s on nvidia/nvidia-direct (11-11 01:xx, storm-era decision) and spark (11-11 07:0x). The 45 s cap demonstrably clips slow-but-alive streams (the 239 s self-healing sample above; during the nvidia storm, requests at 30–34 s TTFT died at 45 s while 15 s ones lived). We are no longer asking upstream to pick a better number — we are asking for a different mechanism: see Proposed fixes #2.

Evidence carried over from v1 (unchanged, still load-bearing)

Event-layer death chain; turn-106 TTFT sample; abort/error/silent-stall taxonomy table; 102-interrupted-turns statistics (46 land at 45.0–45.8 s = stacked 45 s watchdog caps, now explained by our own config, not a hidden layer); 8402 tail-loss capture (218 requests / 6 no-END, three timestamp-matched to user-visible stalls — "last open hop" now narrowed by the differential tests above to the harness process); fake-model-rewrite confounder on resellers; regional-routing incident separator; four symptom skins mapped to one chain; 239 internal-API 5xx as downstream symptom; slot-declaration console error as separate unverified contributor. The full text of these sections is preserved verbatim in discussion #12655 (the original body of this investigation).

Proposed fixes (rewritten around client-side ownership)

1. Classify finish-less stream ends as errors, not completions. A stream that closes without finish/[DONE] must not write turn/end {kind:"completed"}; today it both fakes success and bypasses the error path. (Local patches: null-guard + abortfix; the assertNever branch in abortedCancelCause should return {kind:"aborted"} for unknown causes.)
2. Replace the fixed idle cap with a two-tier timer: TTFT timer armed from request send (turn 106's 507 s pre-first-byte wait was unguarded), plus an inter-chunk timer — and when either fires, preserve received partial content (today the timeout error overwrites the thinking chain) and surface "waiting on upstream, last byte N s ago" instead of a motionless turn.
3. Make mode:"always" storm-aware: after K consecutive same-shape failures, back off the payload (trigger context compaction / drop-and-retry-smaller) or fail loudly to the user, instead of re-firing a 1 MB request every 60 s for 30 rounds. The retry loop is the storm's engine; unlimited retry was a deliberate user choice, so at minimum it should be visibly counted.
4. Stop calling inbox.clear() on cancel (queued user messages vanish — turns a hang into silent data loss).
5. Non-200 JSON responses should error immediately, never enter the stream-wait path (401-in-364 ms surfaced 300 s later).
6. Consider disabling keep-alive reuse (or shortening pool TTL) for streaming chat requests while the differential finding (last unreduced suspect, see Summary #7) is being chased — or ship a per-provider connectionReuse: false so users can A/B it without patching.
7. Write swallowed replies to a visible record: the "completed toast + zero body + zero thinking" shape (mild stall) has no artifact the user can point at; at least log the stall classification in the turn record (endClass: "stream-close-without-finish").

What would close the case

One captured request where the local logging proxy shows a complete body including the terminating [DONE]/END, while the harness reports stream idle timeout for that same request — that pins consumption-side loss and would justify fix #1/#2 as THE fix. Capture rig is in place (three logging proxies + event-triggered sampler); it has not yet produced the paired sample. Conversely, if a lighter client given the same 1 MB / 100-tool / minute-turn workload also storms, part of the blame returns upstream — we consider that unlikely given the Zcode comparison but state it as the falsifier.