No resuming of downloads

[TurtleTango42]

Describe the bug

When a large downloag gets interrupted, and the download command is run again, it restarts, discarding already downloaded data.

Reproduction

gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  17%|█████████████████████████████                                                          gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  17%|█████████████████████████████                                                          gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  18%|█████████████████████████████▊                                                         gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  18%|██████████████████████████████▌                                                        gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  19%|███████████████████████████████▏                                                       gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  20%|█████████████████████████████████▋                                                     gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:  20%|██████████████████████████████████▍                                                                                                                                      | 3.46G/16.9G [12:37<26:59, 8.33MB/s]
Broadcast message from ...@... (Wed 2026-05-06 14:26:43 GMT):

The system will suspend now!


Broadcast message from ...@... (Wed 2026-05-06 14:26:43 GMT):

The system will suspend now!

Shared connection to ... closed.
$ ssh ...
$ hf download unsloth/gemma-4-26B-A4B-it-GGUF gemma-4-26B-A4B-it-UD-Q4_K_M.gguf
gemma-4-26B-A4B-it-UD-Q4_K_M.gguf:   2%|█▏                                               | 404M/16.9G [01:50<1:04:33, 4.27MB/s]



### Logs

```shell

System info

$ hf version
1.5.0
$ cat /proc/version 
Linux version 6.18...

[jshaofa-ui]

🔧 Solution Proposal: Download Resumption for hf download

Root Cause

When hf download is interrupted (e.g., network disconnect, system suspend), the partially downloaded file is not recognized on subsequent runs. The download restarts from 0% instead of resuming from where it left off.

The issue is in src/huggingface_hub/_download_folder.py (for folder downloads) and src/huggingface_hub/utils/_http.py (for single file downloads). The download logic:

1. Checks if the final destination file exists → if yes, skips download
2. Does NOT check for partial/incomplete downloads (.partial or .tmp files)
3. When the partial file exists, it either overwrites it or creates a new file, discarding progress

The Bug

In src/huggingface_hub/utils/_http.py, the http_get function:

def http_get(
    url: str,
    temp_file: BinaryIO,
    ...
):
    headers = {}
    # BUG: No Range header is set based on existing partial file size
    r = _request("GET", url, stream=True, headers=headers)
    r.raise_for_status()
    for chunk in r.iter_content(chunk_size=64 * 1024 * 1024):
        temp_file.write(chunk)

The function always starts from byte 0, ignoring any existing partial download.

Fix: Add Resume Support via HTTP Range Header

File: src/huggingface_hub/utils/_http.py

def http_get(
    url: str,
    temp_file: BinaryIO,
    ...
):
    headers = {}
    
    # Check for existing partial download and resume from where it left off
    temp_file.seek(0, 2)  # Seek to end
    existing_size = temp_file.tell()
    
    if existing_size > 0:
        headers["Range"] = f"bytes={existing_size}-"
        r = _request("GET", url, stream=True, headers=headers)
        
        # Server must support Range requests (206 Partial Content)
        if r.status_code == 206:
            # Resume: continue writing from current position
            for chunk in r.iter_content(chunk_size=64 * 1024 * 1024):
                temp_file.write(chunk)
            return
        else:
            # Server doesn't support Range — restart from beginning
            temp_file.seek(0)
            temp_file.truncate()
            headers.pop("Range", None)
    
    # Full download from start
    r = _request("GET", url, stream=True, headers=headers)
    r.raise_for_status()
    for chunk in r.iter_content(chunk_size=64 * 1024 * 1024):
        temp_file.write(chunk)

File: src/huggingface_hub/_download_folder.py

Additionally, for folder downloads, we need to handle the .partial file tracking:

def _download_file(
    ...
    local_dir: str | None = None,
):
    # Check for existing partial download
    partial_path = Path(local_path) if local_path else Path(download_path)
    partial_path = partial_path.with_suffix(partial_path.suffix + ".partial")
    
    if partial_path.exists():
        # Move partial file to temp location for resume
        temp_file = partial_path.open("ab")  # Append mode
        should_resume = True
    else:
        temp_file = tempfile.NamedTemporaryFile(delete=False)
        should_resume = False
    
    try:
        http_get(url, temp_file, ...)
        
        # On success, rename to final path
        if local_path:
            shutil.move(temp_file.name, local_path)
    except Exception:
        # Keep .partial file for resume on next attempt
        if should_resume:
            pass  # Keep partial file
        raise
    finally:
        temp_file.close()

Alternative: Use requests Session with Retry + Range

A more robust approach using requests library's built-in retry mechanism:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def _create_session_with_resume():
    session = Session()
    retry = Retry(
        total=5,
        backoff_factor=1,
        status_forcelist=[500, 502, 503, 504],
        allowed_methods=["GET"],
    )
    session.mount("https://", HTTPAdapter(max_retries=retry))
    return session

Combined with the Range header approach above, this provides both automatic retry and resume support.

Files to Modify

1. src/huggingface_hub/utils/_http.py — Add Range header support in http_get()
2. src/huggingface_hub/_download_folder.py — Track partial downloads with .partial extension
3. src/huggingface_hub/cli/download.py — Add --resume flag (optional, auto-detect by default)

Regression Test

def test_download_resume():
    import tempfile
    import os
    from huggingface_hub.utils._http import http_get
    
    # Create a temp file with partial content
    with tempfile.NamedTemporaryFile(delete=False, mode="wb") as f:
        f.write(b"partial content")
        partial_path = f.name
    
    try:
        # Download should resume from partial content
        with open(partial_path, "ab") as temp_file:
            http_get("https://httpbin.org/range/1024", temp_file)
        
        # Verify content was appended, not replaced
        with open(partial_path, "rb") as f:
            content = f.read()
            assert content.startswith(b"partial content")
            assert len(content) > len(b"partial content")
    finally:
        os.unlink(partial_path)

Impact

• Fixes download resumption for large model files (16.9GB+ in the reported case)
• Saves bandwidth and time for interrupted downloads
• No breaking changes — purely additive feature
• Works with servers that support HTTP Range requests (most CDNs and object storage do)

[ishan-1010]

Dug into this on main (a861924). The restart-from-zero is real on both download paths, and it recently got broader than this report.

What happens now. _download_to_tmp_and_move writes every download to a process-unique temp file: {etag}.{uuid4}.incomplete, opened in "wb". A rerun can't find the previous run's partial by construction, and the orphan is never cleaned up either. http_get has full resume machinery (resume_size + Range), but this caller never passes it.

Reproduced with SmolLM2-135M/model.safetensors (~270 MB), kill -9 at ~100 MB, rerun:

• HTTP path (HF_HUB_DISABLE_XET=1): rerun starts at 0 and the old partial stays on disk. Total .incomplete bytes grew 100 MB -> 320 MB for a 270 MB file: full re-download plus a permanent 100 MB orphan per interrupted run.
• Xet path: same orphaned .incomplete accumulation. Any recovery there depends on the xet chunk cache rather than the partial file, so for files larger than the chunk cache's budget (like the 17 GB GGUF in this report) an interrupted download effectively restarts too.

Why it's set up this way. #4306 (merged Jun 5) deliberately moved to unique temp files because a shared {etag}.incomplete corrupts the cache when flock silently fails (Lustre/GPFS/some NFS). The tradeoff was explicit: "a broken lock costs only duplicated bandwidth". This issue is the cost showing up for everyone, not just broken-flock filesystems, and interrupted runs now also leak disk.

(#4143 proposed resume hardening with a sidecar, but it targets the pre-#4306 shared-incomplete design and has been quiet since April; it doesn't apply cleanly to the current code.)

Possible directions

• (a) Cheap, uncontroversial: clean up stale {etag}.*.incomplete orphans for the same etag once a download completes (or on startup). Fixes the disk leak regardless of resume.
• (b) Restore cross-process resume safely: on start, pick the largest existing partial for the same etag, validate with If-Range/Content-Range (the validation http_get already half-has), copy/rename it into the new unique temp, and pass resume_size. Keeps [Fix] Make concurrent downloads safe even when file locking is broken #4306's no-shared-writes property: each process still writes only to its own file.

Happy to PR either or both. @Wauplin since #4306 touched this exact tradeoff.

[NancySadkov]

I can confirm the script is sloppy, and on bad public wifi connection it is impossible to complete something like

uvx --from huggingface_hub hf download shieldstar/Qwen3.5-122B-A10B-int4-AutoRound-EC 

is it possible to use curl instead?

[Wauplin]

Yes you can use curl (or any other download method) using the download URL (e.g. https://huggingface.co/shieldstar/Qwen3.5-122B-A10B-int4-AutoRound-EC/resolve/main/model-00001-of-00013.safetensors). This will likely be slower and more manual but totally possible.

For the "resume on failing download" we will try to figure out something but the solutions suggested above are not correct / considered. We need a solution happening in hf_xet instead.

[galmok]

Has a solution been found? Just spent an hour downloading 360GB and it stopped at 98% (just froze, no download happening, no cpu usage).

Update: I let it be and it is now at 99%. Seems to be some long retry timeout.

[mkultra333]

No update on this? It makes the Hugging Face downloader more or less useless for large downloads. If the safetensor or gguf part files are all 2GB files or something it's tolerable, but when they're 15GB+ this turns into a nightmare.

The program never resumes anyway, it just stalls at some point and you have to restart, and it'll continue from the beginning of the first file not completely downloaded. So you have to baby sit the thing even with the smaller files.

[snailwei]

same issue here with latest version, each time download was timeout, run again, it starts over, ... no resuming.

1. Downloaded 65.98GB, connection broken
2. Restart the download, it starts from scratch

[feffy380]

No update on this? It makes the Hugging Face downloader more or less useless for large downloads. If the safetensor or gguf part files are all 2GB files or something it's tolerable, but when they're 15GB+ this turns into a nightmare.

The program never resumes anyway, it just stalls at some point and you have to restart, and it'll continue from the beginning of the first file not completely downloaded. So you have to baby sit the thing even with the smaller files.

Yeah at this point just use wget/curl. hf download seems to be written with enterprise internet connections in mind. On a slow consumer connection it and xet have given me none of the promised benefits but plenty of headaches with downloads failing at 99%, resumes not working, and silent corruption.

[MohammedAlkindi]

Some measured data points for the HTTP (non-xet) path, plus a scoping question, since the thread's last state leaves that part ambiguous.

At current main (72b5d66) the resume machinery is intact but unreachable: _download_to_tmp_and_move writes to {etag}.{uuid4:8}.incomplete (file_download.py:1948) and calls http_get without resume_size, so the Range logic at file_download.py:367 never triggers — and a rerun cannot even construct the previous attempt's filename. The finally: tmp_path.unlink() covers catchable failures, but not kill -9 / power loss, which is the scenario this issue is about, so the orphan also stays on disk forever.

Reproduced end-to-end on Windows 11 / Python 3.13 with the real _download_to_tmp_and_move against a local Range-supporting HTTP server (25 MB synthetic file; child process hard-killed mid-download at 41.9%):

• current main: the rerun sends no Range header, transfers 25,000,000 / 25,000,000 bytes again, and run 1's orphan .incomplete file is still there afterwards.
• with a scoped patch: the rerun sends Range: bytes=10485760-, transfers only the missing 14,514,240 bytes, final sha256 matches, no orphan left. 3/3 runs each way.

On the #4306 constraint (broken flock on Lustre/GPFS/NFS — the reason the uuid suffix exists): resume can be restored without reopening that hazard by claiming the largest same-stem orphan via Path.replace() before opening — rename atomicity, not advisory locking, decides the winner, and the loser falls through to a fresh uuid path exactly as today. I raced two simultaneous processes at one planted orphan, four times: exactly one resumes (206), the other does a fresh full download (200), byte-correct output every time, never a shared write. That keeps #4306's actual invariant — and holds precisely on the filesystems where flock is the broken part.

The question: when you wrote that a solution "needs to happen in hf_xet instead" — does that cover the plain-HTTP path too (xet unavailable, or HF_HUB_DISABLE_XET=1), or was it about the xet transfer's own recovery? If a scoped HTTP-path-only resume (claim-orphan + resume_size, ~60 lines including the adjusted cleanup semantics) would be considered, I have the patch and the test harness ready and would be happy to open a PR. If the call is that this waits for hf_xet-side work regardless, that's just as useful to know — this comment at least pins the current mechanics with line references.

What I did not verify: behavior against the live CDN (my server is localhost, so real-world etag/Range semantics on huggingface.co are unverified by me); the xet path (untouched); and the production-marked download tests, which need Hub access — the offline TestHttpGet suite passes 9/9.