Bug: LLM requests die at ~302s with terminated when a provider needs >5min silent prefill

[kesslerio]

TL;DR

Some model requests that should only take a minute silently fail after exactly ~302 seconds with a bare terminated transport error, and retries don't help. This happens whenever a provider needs more than five minutes of silent prefill before its first token — notably self-hosted vLLM/EXL3 clusters serving very large context windows. This PR removes the hidden five-minute kill switch that caused it.

Root cause

Node's global fetch is its built-in undici, which arms two defaults no slow LLM request can satisfy: headersTimeout and bodyTimeout, both 300,000 ms. A streaming endpoint emits response headers immediately and then sends no body bytes until prefill completes, so any request whose provider prefills longer than five minutes is aborted mid-flight by the client's own HTTP stack — surfacing as a bare TypeError: terminated (cause UND_ERR_BODY_TIMEOUT), flattened by pi-ai's error handling into a retryable TRANSPORT error, then retried five identically doomed times.

This is not a server-side or harness-watchdog issue: the deployment already raised its engine timeout (VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800) and its adapter firstEventTimeoutMs to 900s+, but undici's client-side body default wins the race against both and kills the request first. A production session deadlocked this way: context grew to ~494k of a 500k window while all compaction attempts (which must send the whole conversation to be summarized) died the same way.

Change

dsh-llm-pi-ai now installs the process-wide fetch dispatcher itself (src/egress.ts), configured by two new top-level settings fields:

• httpBodyTimeoutMs — max ms between response-body chunks before the dispatcher aborts; default 0 (disabled).
• httpHeadersTimeoutMs — max ms to wait for headers; default 0 (disabled).

Both default to disabled, which is deliberate: it returns timeout ownership to the adapter's own firstEventTimeoutMs/streamIdleTimeoutMs watchdogs (the design those fields already encode) instead of introducing a second, hidden timeout owner. A finite value re-installs a dispatcher-level floor below them. The guards install at mount and re-install through the settings seam on change; the install is idempotent.

The scope is process-wide by necessity: pi-ai constructs its OpenAI clients as new OpenAI({ apiKey, baseURL }) with no fetch seam, so there is no per-request place to attach a dispatcher. The npm undici setGlobalDispatcher governs Node's built-in fetch through the shared global-dispatcher symbol. The guards yield to @deepseek-ai/dsh-http-proxy when one of its policies is active, so they never displace a proxy-configured dispatcher (added as a runtime dependency).

Tests and verification

• Behavioral tests pin both directions: a stalled body survives under the disabled default and aborts with UND_ERR_BODY_TIMEOUT at a finite bound; missing headers abort with UND_ERR_HEADERS_TIMEOUT; the install is idempotent.
• The real-composition suite proves the guards install at mount and that a provider request through pi-ai's own client honors the configured bound (an end-to-end regression test for the reported incident).
• Proxy-coordination tests cover yielding to an active proxy policy and falling back when routing throws.
• Gates run locally: package tests (297 passing), per-file coverage 100% on the changed src files, tsc clean, oxlint clean, and the doc gates (translation pairing, agent note format, README known-limitations) all pass.

Known limitation

When a proxy policy is active, dsh-http-proxy owns undici's global dispatcher and the guards install nothing, so the proxy dispatcher keeps undici's default timeouts. Eliminating that gap would require threading both concerns into a single dispatcher; recorded in the package README under Known Limitations.

[argszero]

Confirmed the root cause and verified the fix isn't in master yet — but I want to flag one architectural tension in the proposed approach.

The root cause is right, and it's a known one. undici's headersTimeout/bodyTimeout both default to 300_000 ms, so any provider needing >5min of silent prefill gets client-side-killed at exactly ~302s, surfaced by pi-ai as a bare TypeError: terminated and retried identically. I've seen this same 300s coincidence before: packages/llm/llm-pi-ai/src/config.ts sets DEFAULT_STREAM_IDLE_TIMEOUT_MS to 300_000 too, so the pi-ai layer's watchdog and undici's body timeout are both 5:00 — which is why the symptom pins at 302s rather than at some other bound. Your diagnosis of "the adapter's own firstEventTimeoutMs/streamIdleTimeoutMs is the intended timeout owner, and undici's default wins the race" is exactly correct.

The tension I'd push back on: dsh-http-proxy is explicitly documented as the ONLY process-wide transport owner. From packages/util/http-proxy/src/index.md:

"This is a library, not a plugin: transport policy has one answer per process, so there is nothing for a composition to mount, swap, or scope."

installProxyFromEnvironment installs undici's global dispatcher (setGlobalDispatcher at install.ts:213). Adding a second process-wide dispatcher installer in llm-pi-ai/src/egress.ts that "yields to the proxy when one is active" recreates the single-global-slot problem this package was built to centralize. Specifically:

1. The yield gap you already acknowledge is real and is the worse half: when a proxy policy is active, the guards install nothing, so the proxy dispatcher keeps undici's 300s default. A user who needs BOTH (long-prefill into a self-hosted cluster AND a corporate proxy) gets the timeout still. That's the exact scenario that would have prevented the original deadlock in a proxied environment.

2. Install-order race: setGlobalDispatcher is a single global symbol shared between Node's built-in fetch and npm undici (via Symbol.for('undici.globalDispatcher.…')). await import('undici') itself can clobber it (this bit a real report — web_fetch requestPinned's import('undici') overwrote the proxy dispatcher mid-run). If llm-pi-ai's guard and http-proxy's installer both import undici and call setGlobalDispatcher at different lifecycle times, whichever runs last wins, and the "detect active proxy" check is a snapshot, not a lock.

Better shape — compose through the existing owner rather than beside it. Rather than llm-pi-ai installing its own dispatcher and detecting proxy activity, thread the timeout through dsh-http-proxy's single installer so one owner composes both concerns (proxy + timeouts on the same dispatcher). That eliminates the yield gap by construction and keeps the "one answer per process" invariant that index.md states. Since llm-pi-ai already declares @deepseek-ai/dsh-http-proxy as a dependency (package.json:57), this is a natural place to converge.

This is fundamentally the same class as the dsh-fetch-timeouts stopgap (#5124) — both paper over undici's inherent default by racing a second dispatcher installer against the canonical one. If the maintainers prefer to keep the timeout concern inside pi-ai, the cleanest seam is giving the pi-ai adapter a per-request fetch override (it already accepts a custom fetch) or a dispatcher option, rather than touching the process-global.

[xoykor]

I hit a matching five-minute failure on my locally patched DSH 0.1.5-rc.2 installation with LM Studio/Qwen, and tried the per-request fetch approach mentioned above.

My profile already had timeoutMs and streamIdleTimeoutMs set to 900000. A compaction request nevertheless ended with terminated after approximately 300.3 seconds. LM Studio was still advancing prefill (80.3% immediately before the disconnect), so this was not evidence that the model had stopped working.

My experimental adapter patch supplies a custom fetch through pi-ai's existing options. It delegates to the existing dispatcher and overrides headersTimeout / bodyTimeout only for requests matching the configured loopback provider origin. It does not install a new global dispatcher, and preserves the request's abort signal. This implementation is intentionally restricted to localhost/loopback, not arbitrary remote providers.

After applying it, local compactions completed in approximately 537, 544 and 631 seconds, followed by continuation. A separate custom tool-budget guard later stopped the task; that is independent of the transport timeout.

Implementation and three transport tests: version-pinned patch directory. Tests cover scaled HTTP timeout/completion and cancellation behavior, with origin-scoping checks; they are not a corporate-proxy end-to-end test. The backup contains a patched compiled adapter artifact, not an upstream-ready source PR, and I have not established applicability to current master.

Sharing this as additional local evidence for the per-request seam, rather than proposing another process-global transport owner. The existing workaround in #5124 is also relevant for people comparing approaches.

[PerryLink]

Verified on master (c291e79): the egress-timeout fix you linked has not been merged. There is no egress.ts under packages/llm, and httpBodyTimeoutMs/httpHeadersTimeoutMs appear nowhere in the tree.

Current state on master:

• The adapter only arms an idle watchdog: DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000 (packages/llm/llm-pi-ai/src/config.ts:46), armed per request at llm-pi-ai/src/adapter.ts:355. It fires on 300s of silence, not on total request duration — so tuning streamIdleTimeoutMs alone cannot fix your case (the watchdog never fires while prefill is streaming).
• The ~302s terminated comes from undici's body timeout on the global dispatcher: the http-proxy plugin installs the process-global dispatcher (packages/util/http-proxy/src/install.ts:209-217, setGlobalDispatcher at line 213) and llm-pi-ai depends on it (llm-pi-ai/package.json:57). argszero's "one owner per process" observation matches this design.
• Because the dispatcher is process-global, a per-request long-prefill override has to flow through that plugin (or a per-request dispatcher option on the underlying fetch).

Workarounds today: what xoykor described (a fresh fetch per request, so each gets its own socket and timeout accounting) is the pragmatic one. For the fix, your PR's approach (explicit egress timeouts wired through the proxy/undici options) is the right shape — I'd suggest rebasing it on master and calling out the http-proxy dispatcher interplay. (unverified: pi-ai's internal request construction lives in an external dependency; I read only the harness side.)