[BUG] HF Hub is DOWN on AWS in Xet mode

[michaelroyzen]

System Info

Xet-backed model download hangs midway for Qwen/Qwen3.6-35B-A3B

Summary

Downloading Qwen/Qwen3.6-35B-A3B on a p5en.48xlarge in us-east-2 on AWS hangs partway through when using the default Hugging Face download path. Clearing ~/.cache does not fix it.

The evidence suggests the hang is in the Hugging Face Xet-backed download path (hf-xet / CAS), not local disk, cache corruption, or general connectivity. Disabling Xet with HF_HUB_DISABLE_XET=1 works around the issue.

Reproduction

uv run hf download Qwen/Qwen3.6-35B-A3B

The dry run succeeds immediately:

uv run hf download Qwen/Qwen3.6-35B-A3B --dry-run

Output:

[dry-run] Will download 40 files (out of 40) totalling 71.9G.

Observed Behavior

The real download starts, writes several GB to the Hugging Face cache, and then stalls indefinitely with no visible progress.

In one reproduction:

• The cache grew to about 4.6G.
• The Python process stayed alive.
• CPU usage continued, but disk writes stopped.
• TCP sockets to Hugging Face / Xet endpoints stayed open but idle.
• strace showed the process mostly waking on timers, with no meaningful network reads/writes.
• The Xet log showed repeated 403 Forbidden responses from signed us.aws.cdn.hf.co/xorbs/... range URLs, followed by retrieval URL refresh/retry loops.

Representative Xet log lines:

Received CAS response ... status_code=403
Refreshing expired retrieval URLs
Retry on 403 (Forbidden) enabled): "s3::get_range" api call failed ... HTTP status client error (403 Forbidden) for url (https://us.aws.cdn.hf.co/xorbs/...)
Retrieval URLs refreshed successfully

Expected Behavior

The model download should either complete successfully or fail with a clear error. It should not stall indefinitely without surfacing the repeated 403 Forbidden failures to the CLI.

Workaround

Disabling Xet allows the download path to work:

HF_HUB_DISABLE_XET=1 uv run hf download Qwen/Qwen3.6-35B-A3B

A smaller shard test completed successfully with Xet disabled:

HF_HUB_DISABLE_XET=1 uv run hf download Qwen/Qwen3.6-35B-A3B \
  --include model-00007-of-00026.safetensors \
  --local-dir /tmp/qwen-no-xet-test

That downloaded a 1.1G safetensors shard successfully in a few seconds.

Environment

All tests run on a p5en.48xlarge instance in us-east-2 on AWS.

OS: Linux 6.8.0-1050-aws
Python: 3.12.13
huggingface_hub: 1.13.0
hf-xet: 1.5.0rc0
requests: 2.34.0.dev1
urllib3: 2.6.3
CLI: hf 1.13.0

No relevant HF_*, HUGGINGFACE_*, HTTP_PROXY, or HTTPS_PROXY environment variables were set during the test.

Disk space was not the issue:

Filesystem: /dev/root
Size: 1.5T
Available: ~1.5T
Use: 3%
Inodes used: 1%

Additional Context

The CLI warns that the request is unauthenticated:

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

UPDATE: logging in did not fix it, this was not the issue.

However, the observed behavior looks like a Xet/CAS transfer stall rather than ordinary rate limiting:

• Metadata/dry-run requests succeed quickly.
• Downloads proceed partially before stalling.
• The Xet backend logs repeated 403 Forbidden responses from signed range URLs.
• Disabling Xet makes the same repository downloadable.

This may belong in huggingface_hub or xet-core rather than transformers, but it affects model loading/downloading workflows that Transformers users will hit by default.

Who can help?

@ArthurZucker @SunMarc

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

uv run hf download Qwen/Qwen3.6-35B-A3B

on a p5en.48xlarge instance in us-east-2 on AWS.

Expected behavior

Xet download works in us-east-2.

[Rocketknight1]

Is anyone else encountering this..? Xet downloads are working fine for me

[michaelroyzen]

Hi @Rocketknight1 this seems to be an AWS-specific issue and it's possibly related to how Cloudfront is configured for HF Hub downloads with Xet on AWS. Can you spin up an EC2 instance in us-east-2 and try from there to reproduce?

I tried again today on multiple distinct p5en.48xlarge in us-east-2 and each time it makes it part way through the download until it tries fetching the remaining files and hangs on infinite 403s. It is a real issue, it seems. I'm not familiar with the internals of the HF Hub CDN but it seems that it has a Cloudfront CDN for downloads to EC2 instances on AWS and that seems to be having problems here.

[ThomasTosik]

We got the same issue on eu-central-1

[Rocketknight1]

Hi @michaelroyzen @ThomasTosik this should now be fixed! The error was indeed real but was internal to the HF Xet systems. If you retry now it should work.

Closing the issue since I think it should be fixed, but if you still encounter the same error, please ping me and I'll reopen