fix(downloader): hash the partial file before issuing the resume request - #11099
Merged
Conversation
The stall watchdog arms as soon as the response body exists, but the downloader then re-hashed the entire existing .partial before reading a single byte from the network. On slow models storage (a CIFS share reading at ~117MB/s) hashing a multi-GB partial outlasts the 60s stall window, so the watchdog aborted every healthy resume with 'download stalled: no data received for 1m0s'. The partial never grew, so every retry re-paid the same hash and failed identically, wedging the install permanently (any partial over ~7GB on such storage). Open the partial and hash it before the HTTP request instead: the watchdog now only measures actual network idle time, and the origin no longer sits on an idle connection while the hash runs. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
On a resumed download, the stall watchdog arms as soon as the HTTP response body exists, but the downloader then re-hashed the entire existing
.partialbefore reading a single byte from the network. Nothing resets the watchdog while hashing, so if the hash outlastsDownloadStallTimeout(60s), the watchdog closes the body and the resume dies with:Since the partial never grows, every retry re-pays the same hash and fails identically: the install wedges permanently.
This is not theoretical: on a k8s deployment with
/modelson a CIFS/SMB share reading at ~117MB/s, a 7.9GB partial takes ~67s to hash, and every HF import retry died at exactly the 60s window whilecurlfrom the same pod streamed from the HF CDN at full speed. Any partial over ~7GB on such storage hits this; fast local NVMe hashes in seconds, which is why it never showed up in dev.The fix
Open the partial and hash it before issuing the HTTP request. The watchdog now measures only actual network idle time, and the origin no longer sits on an idle connection while the hash runs.
Test
TDD: new spec in
stall_test.goresumes from a 2GiB sparse partial (hashing reliably outlasts a 150ms stall window) against a server that answers the Range request promptly. It failed with the exact production error before the reorder and passes after. Full downloader suite (59 specs) green, also under-race.🤖 Generated with Claude Code