Constant `CAS service error` on XET enabled repos

[danielhanchen]

People who download from our repo https://huggingface.co/unsloth unfortunately keep getting RuntimeError: Data processing error: CAS service error : IO Error: Too many open files (os error 24)

Longer error:

"timestamp":"2025-05-12T02:12:04.887262Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind:
 Request, url: \"https://transfer.xethub.hf.co/xorbs/default/34b2f024c1843d3c470c2fab0c55ccfbc86fc8f207ae08e
cc1070d1080734b5e?X-Xet-Signed-Range=bytes%3D0-57223773&Expires=1747019506&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc2
91cmNlIjoiaHR0cHM6Ly90cmFuc2Zlci54ZXRodWIuaGYuY28veG9yYnMvZGVmYXVsdC8zNGIyZjAyNGMxODQzZDNjNDcwYzJmYWIwYzU1Y2
NmYmM4NmZjOGYyMDdhZTA4ZWNjMTA3MGQxMDgwNzM0YjVlP1gtWGV0LVNpZ25lZC1SYW5nZT1ieXRlcyUzRDAtNTcyMjM3NzMiLCJDb25ka$
Rpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE3NDcwMTk1MDZ9fX1dfQ__&Signature=vpEdXcP0~0xiOxt-l8DuG-k
buNPbA72k5DPR~2t8NOrytM~f77IqEiHuSFDHIYQIlcxkrICvkcjBujiCygzUNkFHXG35Y7i5wdsQP6imOKJtAxKNyGd1kGuvQfW2SEYl9o5
~6nRtx47apnjBTC1TMzApBle-wZIk8iSC7U-CGBmEmto3cZcbmKZR-QOHr8WMtgVqvsESf0e9FmjmubVGNkxifEuwhFkgUZnM1G2NReut5Ly
BFe~VZBOh~c-08is87Gkgm5rawut93q8zRrPsJfL6Dhh~F6XGTV-muliFcFAmphYeN9359~nutzvV7Q4z80Xw3m8SzOm6pOTwlBfucg__&Ke
y-Pair-Id=K2L8F4GPSG1IFC\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"tcp open error
\", Os { code: 24, kind: Uncategorized, message: \"Too many open files\" })) }). Retrying..."},"filename":"/
home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":164}

1. Disabling XET and using HF_TRANSFER has the same issue.
2. Disabling XET and disabling HF_TRANSFER solves the issue.

The issue seems to be prevalent on a lot of our uploads at https://huggingface.co/unsloth

[jsulz]

Thanks for filing this issues @danielhanchen we'll take a look at this.

I was able to reproduce this with https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF using:

huggingface-cli download --repo-type model unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-BF16.gguf

With hf-xet v1.1.0 and huggingface-hub v0.31.1.

In my case, the file Qwen2.5-VL-7B-Instruct-BF16.gguf eventually successfully downloads; is this similar to what you are experiencing?

When you encounter this:

• What repo are you using?
• Do you have any HF_XET_* environment variables set (env | grep HF_XET)?
• Do you have any details about the machine specs (most notably disk and network)?
• Are you using the huggingface-cli or another method?

[danielhanchen]

Thanks @jsulz !

Env vars for XET:

HF_XET_CACHE=/.cache/huggingface/xet
HF_XET_HIGH_PERFORMANCE=1

Repo: Tbh nearly all Unsloth repos - but yes the Qwen2.5 7B GGUF, https://huggingface.co/unsloth/Qwen2.5-VL-72B-Instruct-GGUF is more obvious since it fails downloading even 1 file

I'm using snapshot_download ie:

import os
from huggingface_hub import snapshot_download
snapshot_download(
	"unsloth/Qwen2.5-VL-72B-Instruct-GGUF",
	local_dir = "Qwen2.5-VL-72B-Instruct-GGUF",
	allow_patterns = ["*BF16*",],
)

[ghchris2021]

I may have encountered this issue or something related to it FWIW; details at:
#322 (comment)

[ghchris2021]

IDK how your system(s) are set up where the original code error in the OP was produced, but the OP said: "...RuntimeError: Data processing error: CAS service error : IO Error: Too many open files (os error 24)" was a listed symptom in the case they discussed.

FWIW I'll mention what may be obvious or irrlevant -- in LINUX and typical UNIX-like environments there's a user level open file maximum 'ulimit':

% ulimit -a
...
open files (-n) 1024
...

And there's a file system relevant 'sysctl' limit also:
% sysctl -a | grep _open
fs.nr_open = 1048576

So typically one must adjust those suitable to the demands of the use case if an number exceeding the configured limits is needed.

[jsulz]

Thanks for following up with more details @danielhanchen - @assafvayner has been looking into this, so assigning the issue to him.

@ghchris2021 I added a note to #322 which some quick questions for you.

[danielhanchen]

@assafvayner If you have any questions, ask away! I also sometimes see similar messages for uploading as well

@ghchris2021 It might be something to do with the end user's machine, but most likely not, since normal HF without hf_transfer or xet seems to be fine - my guess is maybe some hash collision maybe, or maybe some bad mapping / something broke, but I'm just speculating

[danielhanchen]

@assafvayner Oh fantastic work on the fix - do you know approx when there might be a new pypi release? :) I'll test it out and update everyone who had the issue - appreciate it!

[assafvayner]

@danielhanchen Thanks for giving it a test. I suspect we will make a release tomorrow (May 15 PST), at least that is what I discussed briefly with @jsulz

[jsulz]

Hey @danielhanchen we're resolving a few issues with our next release (including this one). We should have a release ready either tomorrow or early next week; we'll follow up here the moment it's out.

[rajatarya]

@danielhanchen fix is being deployed to PyPI in a few minutes! hf-xet 1.1.2.