fix(downloader): bound the wait for response headers so a wedged origin cannot hang an install forever

[localai-bot]

What happened

A gallery model install (materializing a HuggingFace repo) hung for 94 minutes with zero bytes transferred, no error, no retry and no abort. The partial tree stayed frozen at 18G. The last log line was the download starting, then complete silence:

14:06:19 INFO Downloading url=".../LongCat-Video-Avatar-1.5/resolve/<rev>/base_model/diffusion_pytorch_model-000..."
(then nothing; partial tree still 18G when checked again at 15:40)

The retry machinery from #10985 was working - two retries fired just before, at 14:05:01 and 14:06:14 ("download failed, retrying"). The third attempt started at 14:06:19 and wedged. The install never completed, the model config was never written, and the operator sees only an install that silently never finishes.

Root cause

The stall watchdog added earlier wraps the response body:

if DownloadStallTimeout > 0 {
    source = newIdleTimeoutReader(resp.Body, DownloadStallTimeout)
}

That only starts guarding once downloadClient.Do(req) has returned. The transport built by httpclient.New bounds dial (30s), TLS handshake (10s), idle pooled connections (90s) and expect-continue (1s) - but had no ResponseHeaderTimeout. So a peer that completes the dial and TLS handshake, reads the request, and then never sends a status line parks Do() for the process lifetime. IdleConnTimeout governs pooled idle connections, not an in-flight request.

Two call sites were unguarded, not one: the body request, and the HEAD that probes for Range support before a resume.

The fix

http.Transport.ResponseHeaderTimeout, set at the transport rather than as a client-level Timeout - a client Timeout also bounds the body and would truncate multi-tens-of-GB downloads, reintroducing a worse bug. The deliberate "no body deadline" property is preserved and covered by a test.

120 seconds. It has to tolerate an origin legitimately slow to start answering (a CDN under load, cold object storage, a redirect chain) while still bounding a wedge. Two minutes is far above any plausible legitimate time-to-first-byte and twice the existing 60s stall window. Worst case becomes roughly 6 minutes across the 3-attempt budget instead of unbounded. Overridable via DownloadResponseHeaderTimeout, mirroring DownloadStallTimeout.

The knob is opt-in (httpclient.WithResponseHeaderTimeout) rather than a new default in HardenedTransport. httpclient.New is a shared constructor with 28 call sites, several of which carry streaming traffic - the cloud-proxy backend proxying chat completions, and the MCP SSE clients. A streaming endpoint may legitimately withhold headers until it has something to say (a queued or slow-to-first-token completion), so capping that by default would risk breaking exactly the traffic the constructor's doc comment says it protects. Only the downloader opts in; no other caller's behaviour changes.

A classification trap this exposed

net/http reports a ResponseHeaderTimeout as an error satisfying errors.Is(err, context.DeadlineExceeded) (verified empirically, not assumed). IsRetryable excluded context.DeadlineExceeded outright, so it read our own transport guard firing on a wedged peer as "the caller gave up" and refused to retry - the guard would have fired and then aborted the install permanently.

An explicit ErrTransientDownload marking now outranks the cancellation sentinels. A caller who genuinely gave up is still caught by the ctx.Err() check above it, so nothing that should stop is let through. The resume probe's error is likewise marked transient (it only ever fails on transport trouble - the status is not consulted), so a momentarily wedged origin no longer turns a resumable download into a hard install failure.

Tests

TDD, Ginkgo/Gomega. The new downloader specs use a server that accepts the connection, reads the request, and never writes a response. Before the fix all three hung the full Eventually window; the fourth spec (body not truncated) passed throughout, confirming the suite was not trivially red.

[FAILED] Timed out after 5.000s.   aborts a request whose response headers never arrive instead of hanging forever
[FAILED] Timed out after 5.000s.   classifies a response header timeout as retryable so the plan resumes
[FAILED] Timed out after 5.001s.   aborts a resume probe whose response headers never arrive
FAIL! -- 1 Passed | 3 Failed | 0 Pending | 49 Skipped

After the fix the wedge is detected in ~0.3s. Coverage added:

• request whose headers never arrive aborts in bounded time
• the error is classified retryable, so the plan resumes from the .partial
• the resume HEAD probe is guarded too
• a slow body after prompt headers is not truncated
• httpclient: opt-in bounds headers, does not bound the body, and the default transport stays unbounded so streaming is unaffected
• IsRetryable: transient outranks a deadline-reporting error, still refuses a done context and a deliberate user abort

Verification

command	exit
go build ./core/... ./pkg/...	0
go test ./pkg/downloader/ ./pkg/httpclient/	0
make lint	0

Related

Third defect found in this download path, and they form a set:

• fix(downloader): distinguish read from write failures and retry transient ones #10985 - read vs write errors conflated; no retry within the plan
• fix(distributed): count staging verification as progress, not as a stall #11026 - hash verification did real work while emitting no progress signal, and an expired deadline returned success
• this one - a request that never receives headers hangs unguarded, so the install silently never finishes

🤖 Generated with Claude Code

https://claude.ai/code/session_0156CbKpDh8qbAYzJZJDZ2yk