Downloads Stall when hf-xet is Enabled

[ConnoisseurProtege]

Describe the bug

My downloads using hf download seem to stall when I enable hf-xet regardless of what I'm downloading. When I use HF_HUB_DISABLE_XET=1 the download works fine. Log is too long so I had to cut off the beginning.

Reproduction

Just run hf download to try to download anything. More specifically for the logs I was downloading https://huggingface.co/bartowski/Qwen_Qwen3.5-122B-A10B-GGUF

Logs

{"timestamp":"2026-05-23T01:23:27.533308Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 34.28s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:27.533319Z","level":"INFO","fields":{"message":"Concurrency control for download: Current concurrency = 64; predicted bandwidth = 1769489; success_ratio = 1.000; reference_size = 1.0MB; observed bytes sent so far = 2269162817; completed transmissions = 298"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":597}
{"timestamp":"2026-05-23T01:23:27.882048Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 62 to 63; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 12MB at new concurrency is 59.37s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:28.150946Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 34.28s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:28.789718Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 34.26s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:29.436170Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 34.20s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:30.063754Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 34.14s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:30.670690Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 33.91s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:30.747347Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.747347Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.747347Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.747349Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.747364Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.747425Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.747510Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:30.788425Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:30.788448Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:30.789348Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:30.789355Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:30.800431Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:30.800442Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:31.140116Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:31.140148Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:31.143283Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:31.143292Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:31.359544Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:31.359568Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:31.359593Z","level":"INFO","fields":{"message":"Concurrency control for download: Increased concurrency from 64 to 64; reason: success ratio 1.000 is above threshold 0.800 and predicted RTT for 0MB at new concurrency is 30.04s < target 60.0s"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":543}
{"timestamp":"2026-05-23T01:23:31.451242Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:31.451267Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:31.452734Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:31.452741Z","level":"INFO","fields":{"message":"Retry strategy","max_attempts":5,"base_delay":"3s","no_retry_on_429":false},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":235}
{"timestamp":"2026-05-23T01:23:31.488634Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:31.488660Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T01:23:31.788009Z","level":"INFO","fields":{"message":"Received CAS response","request_id":"","status_code":206},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":288}
{"timestamp":"2026-05-23T01:23:31.788033Z","level":"INFO","fields":{"message":"Request Success: s3::get_range api call succeeded (request id )."},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":196}
{"timestamp":"2026-05-23T02:40:26.764287Z","level":"WARN","fields":{"message":"Request error","err":"Reqwest(reqwest::Error { kind: Request, url: \"https://transfer.xethub.hf.co/xorbs/default/d494e4adf0123dc9cb4019d4db0e9435f0f4b9bf46707f25efdcdfb569cb6f7b?user_id=6903cd888b82cf1035f6e4d5&repo_id=699dd5e65e910ba1f3e7f6c3&X-Xet-Session-Id=019e5254-f30b-73e0-a6c6-5b750a6cb290&X-Xet-Signed-Range=bytes%3D9629625-9642927%2C26021219-26038688%2C30728648-30735056%2C36153821-36172687%2C41800156-41808654%2C43776615-43784744%2C43813616-43828764%2C43886469-43897827%2C43984107-43989077%2C47127931-47133759%2C50741838-50753080%2C50765703-50775081%2C56781176-56784253&Expires=1779501360&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly90cmFuc2Zlci54ZXRodWIuaGYuY28veG9yYnMvZGVmYXVsdC9kNDk0ZTRhZGYwMTIzZGM5Y2I0MDE5ZDRkYjBlOTQzNWYwZjRiOWJmNDY3MDdmMjVlZmRjZGZiNTY5Y2I2ZjdiP3VzZXJfaWQ9NjkwM2NkODg4YjgyY2YxMDM1ZjZlNGQ1JnJlcG9faWQ9Njk5ZGQ1ZTY1ZTkxMGJhMWYzZTdmNmMzJlgtWGV0LVNlc3Npb24tSWQ9MDE5ZTUyNTQtZjMwYi03M2UwLWE2YzYtNWI3NTBhNmNiMjkwJlgtWGV0LVNpZ25lZC1SYW5nZT1ieXRlcyUzRDk2Mjk2MjUtOTY0MjkyNyUyQzI2MDIxMjE5LTI2MDM4Njg4JTJDMzA3Mjg2NDgtMzA3MzUwNTYlMkMzNjE1MzgyMS0zNjE3MjY4NyUyQzQxODAwMTU2LTQxODA4NjU0JTJDNDM3NzY2MTUtNDM3ODQ3NDQlMkM0MzgxMzYxNi00MzgyODc2NCUyQzQzODg2NDY5LTQzODk3ODI3JTJDNDM5ODQxMDctNDM5ODkwNzclMkM0NzEyNzkzMS00NzEzMzc1OSUyQzUwNzQxODM4LTUwNzUzMDgwJTJDNTA3NjU3MDMtNTA3NzUwODElMkM1Njc4MTE3Ni01Njc4NDI1MyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc3OTUwMTM2MH19fV19&Signature=MEQCIHIhOLBHLmY7AEK1ozIFQRJvHK-Tbt3HzPE8kG9a%7EaC1AiAEmAWuDVxAgflJYSgzYwm43KhnPK9ngb8KCL0ykWBLgA__&Key-Pair-Id=K3TGA0E1JYVXF7\", source: hyper_util::client::legacy::Error(Connect, Custom { kind: Other, error: Os { code: 10054, kind: ConnectionReset, message: \"An existing connection was forcibly closed by the remote host.\" } }) })"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\common\\http_client.rs","line_number":291}
{"timestamp":"2026-05-23T02:40:26.764360Z","level":"INFO","fields":{"message":"Connection attempt 3/5","api":"s3::get_range"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":105}
{"timestamp":"2026-05-23T02:40:26.764363Z","level":"INFO","fields":{"message":"Retryable Client Error: s3::get_range api call failed (retry 2): error sending request for url (https://transfer.xethub.hf.co/xorbs/default/d494e4adf0123dc9cb4019d4db0e9435f0f4b9bf46707f25efdcdfb569cb6f7b?user_id=6903cd888b82cf1035f6e4d5&repo_id=699dd5e65e910ba1f3e7f6c3&X-Xet-Session-Id=019e5254-f30b-73e0-a6c6-5b750a6cb290&X-Xet-Signed-Range=bytes%3D9629625-9642927%2C26021219-26038688%2C30728648-30735056%2C36153821-36172687%2C41800156-41808654%2C43776615-43784744%2C43813616-43828764%2C43886469-43897827%2C43984107-43989077%2C47127931-47133759%2C50741838-50753080%2C50765703-50775081%2C56781176-56784253&Expires=1779501360&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly90cmFuc2Zlci54ZXRodWIuaGYuY28veG9yYnMvZGVmYXVsdC9kNDk0ZTRhZGYwMTIzZGM5Y2I0MDE5ZDRkYjBlOTQzNWYwZjRiOWJmNDY3MDdmMjVlZmRjZGZiNTY5Y2I2ZjdiP3VzZXJfaWQ9NjkwM2NkODg4YjgyY2YxMDM1ZjZlNGQ1JnJlcG9faWQ9Njk5ZGQ1ZTY1ZTkxMGJhMWYzZTdmNmMzJlgtWGV0LVNlc3Npb24tSWQ9MDE5ZTUyNTQtZjMwYi03M2UwLWE2YzYtNWI3NTBhNmNiMjkwJlgtWGV0LVNpZ25lZC1SYW5nZT1ieXRlcyUzRDk2Mjk2MjUtOTY0MjkyNyUyQzI2MDIxMjE5LTI2MDM4Njg4JTJDMzA3Mjg2NDgtMzA3MzUwNTYlMkMzNjE1MzgyMS0zNjE3MjY4NyUyQzQxODAwMTU2LTQxODA4NjU0JTJDNDM3NzY2MTUtNDM3ODQ3NDQlMkM0MzgxMzYxNi00MzgyODc2NCUyQzQzODg2NDY5LTQzODk3ODI3JTJDNDM5ODQxMDctNDM5ODkwNzclMkM0NzEyNzkzMS00NzEzMzc1OSUyQzUwNzQxODM4LTUwNzUzMDgwJTJDNTA3NjU3MDMtNTA3NzUwODElMkM1Njc4MTE3Ni01Njc4NDI1MyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc3OTUwMTM2MH19fV19&Signature=MEQCIHIhOLBHLmY7AEK1ozIFQRJvHK-Tbt3HzPE8kG9a%7EaC1AiAEmAWuDVxAgflJYSgzYwm43KhnPK9ngb8KCL0ykWBLgA__&Key-Pair-Id=K3TGA0E1JYVXF7)"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\retry_wrapper.rs","line_number":108}
{"timestamp":"2026-05-23T02:40:26.764374Z","level":"INFO","fields":{"message":"Concurrency control for download: Decreased concurrency from 63 to 62; reason: transfer failed (success_ratio = 1.000, threshold = 0.500)"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":581}
{"timestamp":"2026-05-23T02:40:26.764382Z","level":"INFO","fields":{"message":"Concurrency control for download: Current concurrency = 62; predicted bandwidth = 7720088; success_ratio = 1.000; reference_size = 12.7MB; observed bytes sent so far = 4101162270; completed transmissions = 574"},"filename":"D:\\a\\xet-core\\xet-core\\xet_client\\src\\cas_client\\adaptive_concurrency\\controller.rs","line_number":597}

System info

- huggingface_hub version: 1.16.1
- Platform: Windows-11-10.0.26200-SP0
- Python version: 3.13.13
- Has saved token ?: True
- Installation method: pip
- httpx: 0.28.1
- hf_xet: 1.5.0

[seanses]

According to your log it seems the connection to https://transfer.xethub.hf.co kept getting closed by the remote. Is there any proxy or firewall sitting between you and the Internet?

[ConnoisseurProtege]

According to your log it seems the connection to https://transfer.xethub.hf.co kept getting closed by the remote. Is there any proxy or firewall sitting between you and the Internet?

Yea I'm on a VPN, does that affect it? Disabling xet or downloading thru browser seem to work fine.

[seanses]

It's very likely due to a network issue caused by the VPN (or failure to route through the VPN). Downloading without hf-xet or through browser get URLs under different domain (historically under AWS Cloudfront, now a mixture of hf.co and Cloudfront)

[cinit]

Thanks. Same problem. Solved with HF_HUB_DISABLE_XET=1.

[kanishkg]

+1 also echoed by #883

[seanses]

@cinit @kanishkg We just put up a tentative fix. Can you try hf-xet==1.5.2rc0 and see if that resolves the issue? I'd also recommend you to upgrade huggingface_hub to 1.23.0. We made an UI change to better report download progress.

[seanses]

hf-xet 1.5.2 is released which should fix download hangs under poor network condition.