`hf download` routinely crashes with `HF_HUB_DISABLE_XET` (no retry)

[gohai]

Describe the bug

XET unfortunately doesn't work on my network. Downloading with HF_HUB_DISABLE_XET=1 will randomly fail with a similar protocol error httpx.RemoteProtocolError: peer closed connection without sending complete message body (received 17281988305 bytes, expected 42520394144). When this happens, there is no automatic retry, and no files left in the cache to resume the download.

Reproduction

Unable to reproduce constantly, but this is happening ca 2/3rds of tries when downloading a 40 GB model from the hub.

Logs

", line 897, in iter_bytes
    for raw_bytes in self.iter_raw():
                     ~~~~~~~~~~~~~^^
  File "/home/gohai/.hf-cli/venv/lib64/python3.13/site-packages/httpx/_models.py", line 951, in iter_raw
    for raw_stream_bytes in self.stream:
                            ^^^^^^^^^^^
  File "/home/gohai/.hf-cli/venv/lib64/python3.13/site-packages/httpx/_client.py", line 153, in __iter__
    for chunk in self._stream:
                 ^^^^^^^^^^^^
  File "/home/gohai/.hf-cli/venv/lib64/python3.13/site-packages/httpx/_transports/default.py", line 126, in __iter__
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/gohai/.hf-cli/venv/lib64/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: peer closed connection without sending complete message body (received 17281988305 bytes, expected 42520394144)
Segmentation fault (core dumped)

System info

- huggingface_hub version: 1.18.0
- Platform: Linux-6.17.10-100.fc41.x86_64-x86_64-with-glibc2.40
- Python version: 3.13.9
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/gohai/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: gohai
- Configured git credential helpers:
- Run by AI agent ?: No
- Installation method: hf_installer
- httpx: 0.28.1
- hf_xet: 1.5.1
- gradio: N/A
- tensorboard: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/gohai/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/gohai/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/gohai/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/gohai/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS: False
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: True
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
- HF_XET_HIGH_PERFORMANCE: False

[Wauplin]

Disclaimer: AI-generated comment, reviewed manually and accurate

---

Thanks for the report! Two things:

1. The RemoteProtocolError: peer closed connection without sending complete message body was slipping past our auto-resume logic — the legacy HTTP download path only retried ConnectError/TimeoutException, not this protocol error, so the whole download aborted instead of resuming. I've opened [Download] Retry on RemoteProtocolError in http_get #4351 to fix exactly that: it'll now resume on this error too, and since the retry budget resets every time a chunk arrives, a download that keeps making progress (your 17 GB → 42 GB case) should keep going to completion. That'll ship in the next release.

2. That said, for 40 GB files on a flaky network, hf_xet is really the path you want — it chunks and resumes far more robustly than plain HTTP. You mentioned "XET doesn't work on my network" — could you share the exact error you get when xet is enabled (i.e. HF_HUB_DISABLE_XET unset, with hf_xet installed)? It connects to a dedicated host (cas-bridge.xethub.hf.co), and "doesn't work" is often a firewall/proxy blocking that domain, which is usually fixable by allowlisting it.

   The hf_xet logs are very helpful for diagnosing this — you'll find them under ~/.cache/huggingface/xet/logs. If you can share the relevant lines from a failed xet download, we can likely pinpoint what's being blocked.

Also, the segfault is surprising given xet is disabled in your setup — if you can reproduce it, a backtrace (and whether the machine is under memory pressure / OOM) would help us understand it.

[gohai]

Many thanks @Wauplin for making the http path more resilient! I'll try to do some testing with hf_xet as soon as possible.

[Wauplin]

Hope it'll help yes 😃 But as mentioned, fixing the hf_xet issue on your side would be more future-proof. If you have any specific errors (especially in the logs), please open a separate issue as this one will be closed by #4351