`hf download` exits 0 while leaving `.incomplete` blobs after xet 416 CAS errors

[cipherfoxie]

Describe the bug

hf download of large xet-backed models can exit with status 0 even when downloads did not actually complete. The cache blobs/ directory contains *.incomplete files at exit time, but the CLI reports success and subsequent from_pretrained() calls fail with cryptic errors because the snapshot points at incomplete blobs.

Distinct from #3960 (process killed mid-download): in our case the process exits normally with status 0 but the data is incomplete. The exit code lies; only filesystem state is the ground truth.

Reproduction

NVIDIA DGX Spark (GB10 Blackwell SM12.1, ARM v9.2-A aarch64), Debian 13, default xet enabled.

# Large model with multiple xet-backed shards (3.5 GB each)
hf download mistralai/Voxtral-4B-TTS-2603 --revision main
echo "exit: $?"   # → 0 (claims success)

find ~/.cache/huggingface/hub -name '*.incomplete' | wc -l
# → 3 (silent failure)

Symptoms surface during retry runs on flaky upstream connectivity. Xet endpoints return HTTP 416 "Requested Range Not Satisfiable" on partial-blob CAS lookups, which hf_transfer 0.1.9 treats as fatal but the CLI exit code does not reflect this.

Expected behavior

If any blob remains .incomplete at end of run, exit should be non-zero. Filesystem state is the source of truth.

Workarounds (each independently effective on this hardware)

1. `HF_HUB_DISABLE_XET=1` — falls back to non-xet path. Slower but stable.
2. `HF_HUB_DOWNLOAD_TIMEOUT=300` — extends httpx timeout for >3 GB shards (separate `httpx.ReadTimeout` failure mode without it).
3. Filesystem-level `find blobs -name '*.incomplete' | wc -l == 0` check as exit-code substitute.

Full write-up with the debugging path and a 347-LOC wrapper (`hf-pull`) that combines all three with exponential-backoff retries and progress diagnostics: https://sovgrid.org/blog/fixes-hf-download-lies-at-22gb/

System info

```

• huggingface_hub version: 1.8.0
• Platform: Linux 6.17.0-1014-nvidia aarch64
• Hardware: NVIDIA DGX Spark (GB10, SM12.1, 128 GB unified memory, CUDA 13)
• Python version: 3.12.3
• httpx: 0.28.1
• hf_transfer: 0.1.9
• HF_HUB_DOWNLOAD_TIMEOUT: not set (default)
• HF_HUB_DISABLE_XET: not set (xet enabled)
  ```

Additional context

Reproducible across three different large model downloads (Voxtral-4B-TTS-2603, VibeVoice-Large, Qwen3.6-35B-A3B-PrismaQuant). All exit-zero with incomplete blobs without workarounds applied.

[hanouticelina]

Hi @cipherfoxie,
could you upgrade your huggingface_hub version to the latest and well as hf_xet and see if you still have the same issue? if yes, could you share your log file from ~/.cache/huggingface/xet/logs?

[cipherfoxie]

Thanks for taking a look, @hanouticelina.

Version context from when this occurred: huggingface_hub 1.8.0 + hf_xet 1.4.2. I'll upgrade to the current huggingface_hub 1.15.0 / hf_xet 1.5.0 in the next large model-pull cycle and report back, with the ~/.cache/huggingface/xet/logs file attached if it recurs.

One thing worth flagging regardless of version: the core of this report is not "the process died" but that hf download exited 0 while leaving .incomplete blobs on disk after the xet 416 CAS errors. Even once the xet-side fetch is fixed, the exit-code-vs-filesystem-state mismatch is the part that silently corrupts downstream automation (a wrapper that trusts exit 0 ships a truncated model). The failure was intermittent around ~22 GB, so a deterministic repro is hard; the filesystem ground truth (exit 0 + residual .incomplete) is the reliable signal.

As a stopgap on our side we added a filesystem-level validation wrapper that re-checks blob completeness instead of trusting the exit code, but the exit-zero behaviour itself still seems worth hardening upstream. Will follow up with the upgrade result.

[Wauplin]

TL;DR: please upgrade hf_xet to >= 1.5.1 and the exit-0-on-incomplete behaviour should be gone. Let us know if you still manage to hit it after upgrading!

---

Hi @cipherfoxie, you are correct that the scary part is not the leftover file but the fact that the process returned exit 0 while the download was actually incomplete. I dug into this and I can confirm it, including reproducing the exact "4MB out of 125MB, exit 0" behaviour.

Root cause (on hf_xet 1.4.2, your version): when a CAS reconstruction request gets a spurious HTTP 416 on a range that is still inside the file (exactly what flaky connectivity produces), hf_xet 1.4.2 treats that 416 as end-of-file and stops reconstruction early. It then returned the partial byte count without comparing it to the expected file size, so a truncated download was reported as a success. huggingface_hub trusted that result and committed the short file into the cache behind a valid pointer, hence the exit 0.

Note this is different from a 416 on a beyond-EOF probe, which is normal and correctly handled (the CAS server returns 416 there on purpose). Only the spurious within-file 416 truncates.

Good news: this is already fixed in hf_xet >= 1.5.1 by two changes in xet-core:

• fix: bound file reconstruction range using file_size to prevent 416 errors xet-core#716 bounds the reconstruction range to the known file size, so the client no longer issues the requests that triggered the spurious 416s in the first place.
• Implement optional range specification for downloads. xet-core#735 adds a final downloaded_bytes != expected_size check that turns any short reconstruction into a hard error instead of a silent success. I verified this directly: same scenario on 1.4.2 exits 0, on 1.5.1 it raises File size mismatch: expected ... but downloaded ... and exits non-zero. So exit 0 is a real success again. 🎉

So upgrading to hf_xet >= 1.5.1 (any recent huggingface_hub pulls it in) resolves the exit-0-but-incomplete issue. As a side note, #4306 also reworked the cache write path so partial files can never be committed in place, which hardens this further.

I reproduced the original behaviour deterministically with the script below (pin huggingface_hub==1.8.0 + hf_xet==1.4.2 to see it). It puts a tiny proxy in front of the CAS endpoint and injects a single 416 on a within-file range, then downloads a 125MB xet-backed file:

repro_416.py (run: HF_HOME=/tmp/repro HF_XET_RECONSTRUCTION_MIN_RECONSTRUCTION_FETCH_SIZE=4mb python repro_416.py)

"""Repro for #4223: a spurious HTTP 416 on a CAS reconstruction call silently
truncates a xet download (success reported, exit 0, file incomplete).

Pin the affected versions to see it:
    pip install "huggingface_hub==1.8.0" "hf_xet==1.4.2"

A local proxy forwards every CAS request verbatim EXCEPT it injects a single 416
on the first reconstruction range that starts *inside* the file (a spurious 416,
as caused by flaky connectivity). Run with a small min fetch size so several
reconstruction calls are issued:
    HF_HOME=/tmp/repro HF_XET_RECONSTRUCTION_MIN_RECONSTRUCTION_FETCH_SIZE=4mb python repro_416.py
"""

import os
import sys
import threading
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

REPO_ID = "openai-community/gpt2"
FILENAME = "64-8bits.tflite"
EXPECTED_SIZE = 125162496
INJECT_RANGE_START = 4000000  # a range that is still WITHIN the 125MB file

REAL_ENDPOINT = None
injected = []


class ProxyHandler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self):
        range_header = self.headers.get("Range", "")
        is_reconstruction = "/reconstructions/" in self.path

        if is_reconstruction and range_header.startswith(f"bytes={INJECT_RANGE_START}-") and not injected:
            injected.append((self.path, range_header))
            print(f"[proxy] >>> INJECTING 416 for {self.path} (Range: {range_header})", flush=True)
            self.send_response(416)
            self.send_header("Content-Length", "0")
            self.end_headers()
            return

        url = REAL_ENDPOINT + self.path
        req = urllib.request.Request(url)
        for h in ("Authorization", "Range", "User-Agent", "xet-version"):
            if self.headers.get(h):
                req.add_header(h, self.headers[h])
        try:
            with urllib.request.urlopen(req) as resp:
                body, status = resp.read(), resp.status
        except urllib.error.HTTPError as e:
            body, status = e.read(), e.code
        print(f"[proxy] {status} {self.path} (Range: {range_header or '-'}) -> {len(body)} bytes", flush=True)
        self.send_response(status)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


def main():
    import huggingface_hub.file_download as fd
    from huggingface_hub import hf_hub_download

    server = ThreadingHTTPServer(("127.0.0.1", 0), ProxyHandler)
    port = server.server_address[1]
    threading.Thread(target=server.serve_forever, daemon=True).start()

    real_refresh = fd.refresh_xet_connection_info

    def patched_refresh(**kwargs):
        global REAL_ENDPOINT
        info = real_refresh(**kwargs)
        REAL_ENDPOINT = info.endpoint
        return type(info)(
            endpoint=f"http://127.0.0.1:{port}",
            access_token=info.access_token,
            expiration_unix_epoch=info.expiration_unix_epoch,
        )

    fd.refresh_xet_connection_info = patched_refresh

    print(f"[repro] downloading {REPO_ID}/{FILENAME} (expected {EXPECTED_SIZE:,} bytes)")
    try:
        path = hf_hub_download(REPO_ID, FILENAME)
    except Exception as e:
        print(f"[repro] download RAISED: {type(e).__name__}: {e}")
        sys.exit(2)

    actual = os.path.getsize(os.path.realpath(path))
    print(f"[repro] hf_hub_download returned successfully: {path}")
    print(f"[repro] blob size: {actual:,} (expected {EXPECTED_SIZE:,})")
    if actual != EXPECTED_SIZE:
        print("\n*** SILENT TRUNCATION REPRODUCED: success reported, file is incomplete ***")
    else:
        print("\nfile intact (injection did not trigger; rerun)")


if __name__ == "__main__":
    main()

Output on huggingface_hub==1.8.0 + hf_xet==1.4.2:

[proxy] >>> INJECTING 416 for /v1/reconstructions/f39b58dc... (Range: bytes=4000000-11999999)
[repro] hf_hub_download returned successfully: .../snapshots/.../64-8bits.tflite
[repro] blob size: 4,000,000 (expected 125,162,496)

*** SILENT TRUNCATION REPRODUCED: success reported, file is incomplete ***
exit code: 0

[Wauplin]

Hi @cipherfoxie , above comment is AI-generated but accurate. I tested locally and confirmed the repro + that it's fixed in hf_xet 1.5.1. I'll therefore close this issue. Thanks again for flagging 🙏

[cipherfoxie]

Upgraded to hf_xet 1.5.1 on the affected box (DGX Spark, was on 1.4.2). I cannot force the flaky-connectivity repro on demand, so the honest status is: the buggy version is gone from the machine, the filesystem .incomplete check stays in the download wrapper as a cheap belt-and-suspenders, and if the exit-0-on-incomplete behaviour ever resurfaces on >= 1.5.1 I will reopen with logs. Thanks for digging out the 416-as-EOF root cause and confirming the repro, that matches exactly what the flaky path here produced.