DNS error while download file with xet storage backend

[iNARI777]

Describe the bug

I encountered a DNS error when using the API from huggingface_hub to download model files from HuggingFace. This issue only occurs with files stored using xet storage—files stored with LFS or regular git can be downloaded without any problems.

Here is the core code snippet I used to download the model files. Since my machine cannot connect to the public internet directly, I have set up an HTTP proxy in the code.

Could you please advise me on how to resolve this issue? Thank you!

Reproduction

import requests

from huggingface_hub import hf_hub_download, configure_http_backend


def set_hf_http_proxy(http_proxy):
    proxies = {
        "http": http_proxy,
        "https": http_proxy
    }
    def session_factory():
        session = requests.Session()
        session.proxies = proxies
        return session
    configure_http_backend(session_factory)


http_proxy = "http://xx.xxx.xx.xxx:xxxx"
set_hf_http_proxy(http_proxy)

repo_id = "Qwen/Qwen3-Reranker-0.6B"
filename = "tokenizer.json"
repo_type = "model"
revision = "main"
local_dir = "./"
cache_dir = "./"
local_dir_use_symlinks = False

hf_hub_download(
    repo_id=repo_id,
    filename=filename,
    repo_type=repo_type,
    revision=revision,
    local_dir=local_dir,
    cache_dir=cache_dir,
    local_dir_use_symlinks=local_dir_use_symlinks,
    resume_download=True
)

Logs

$ python file_download.py
tokenizer.json:   0%|                                                                                                                                                        | 0.00/11.4M [00:00<?, ?B/s]{"timestamp":"2025-06-09T12:32:41.363157Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-09T12:32:41.363213Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.053998972s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-09T12:32:42.435602Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-09T12:32:42.435636Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 1.606925917s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-09T12:32:44.061634Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-09T12:32:44.061664Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 10.395844705s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-09T12:32:54.482074Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-09T12:32:54.482106Z","level":"WARN","fields":{"message":"Retry attempt #3. Sleeping 23.1864274s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-09T12:33:17.684544Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-09T12:33:17.684571Z","level":"WARN","fields":{"message":"Retry attempt #4. Sleeping 38.695224788s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-09T12:33:56.402793Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-09T12:33:56.402865Z","level":"ERROR","fields":{"message":"error invoking get_reconstruction api, error: ReqwestMiddlewareError(Middleware(Request failed after 5 retries\n\nCaused by:\n    0: error sending request for url (https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf)\n    1: client error (Connect)\n    2: dns error: failed to lookup address information: Temporary failure in name resolution\n    3: failed to lookup address information: Temporary failure in name resolution))","caller":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs:286"},"filename":"/home/runner/work/xet-core/xet-core/error_printer/src/lib.rs","line_number":28}
tokenizer.json:   0%|
Traceback (most recent call last):
  File "/home/hadoop-aipnlp/projects/nlp-llm-tools/file_download.py", line 29, in <module>
    hf_hub_download(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 988, in hf_hub_download
    return _hf_hub_download_to_local_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1290, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1710, in _download_to_tmp_and_move
    xet_get(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 627, in xet_get
    download_files(
RuntimeError: Data processing error: CAS service error : ReqwestMiddleware Error: Request failed after 5 retries

System info

- huggingface_hub version: 0.32.4
- Platform: Linux-4.18.0-147.mt20200626.413.el8_1.x86_64-x86_64-with-glibc2.17
- Python version: 3.12.0
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/hadoop-aipnlp/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.2.2
- Jinja2: 3.1.3
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 10.3.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.26.4
- pydantic: 2.6.4
- aiohttp: 3.11.18
- hf_xet: 1.1.3
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/hadoop-aipnlp/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/hadoop-aipnlp/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/hadoop-aipnlp/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/hadoop-aipnlp/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[Wauplin]

any idea @bpronan @rajatarya ?

[bpronan]

@iNARI777: sorry to hear you are having this issue. We are seeing similar reports from other users while downloading on home networks.

While we work on a fix, there are a few things to try in the meantime to unblock yourself.

Other users have been successful with usually one of the following:

• downgrade hf_xet with pip install --force-reinstall -v "hf_xet==1.1.2"
• increase your file descriptor limit (run ulimit -n to see the previous value to reset to): ulimit -Sn 4096
• try reducing your concurrent xet downloads export HF_XET_MAX_CONCURRENT_DOWNLOADS=2. Note: this may result in a slower download.

[jsulz]

@iNARI777 after looking into this today, we believe we were able to determine the root cause.

As a test, could you pip install hf_xet==1.1.4-rc1 and see if this helps?

[iNARI777]

@jsulz After upgrading hf_xet to version 1.1.4-rc1, the error log seems different when running the previous code. In the error log, I noticed that the domain being resolved is cas-server.xethub.hf.co.gh.sankuai.com, and dns1.sankuai.com is our private DNS server. I have no expertise in this field, but this DNS resolution issue seems quite difficult to solve in my environment?

$ python file_download.py 
tokenizer.json:   0%|                                                                                                                                                  | 0.00/11.4M [00:00<?, ?B/s]{"timestamp":"2025-06-11T06:29:15.752378Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", ResolveError { kind: NoRecordsFound { query: Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }, soa: Some(Record { name_labels: Name(\"gh.sankuai.com.\"), rr_type: SOA, dns_class: IN, ttl: 60, rdata: Some(SOA { mname: Name(\"dns1.sankuai.com.\"), rname: Name(\"hostmaster.sankuai.com.\"), serial: 2016116155, refresh: 18000, retry: 400, expire: 72000, minimum: 1800 }) }), negative_ttl: Some(60), response_code: NXDomain, trusted: true } })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-11T06:29:15.752438Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 2.856961206s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-11T06:29:18.616208Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", ResolveError { kind: NoRecordsFound { query: Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }, soa: Some(Record { name_labels: Name(\"gh.sankuai.com.\"), rr_type: SOA, dns_class: IN, ttl: 60, rdata: Some(SOA { mname: Name(\"dns1.sankuai.com.\"), rname: Name(\"hostmaster.sankuai.com.\"), serial: 2016116155, refresh: 18000, retry: 400, expire: 72000, minimum: 1800 }) }), negative_ttl: Some(57), response_code: NXDomain, trusted: true } })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-11T06:29:18.616359Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 250.695743ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-11T06:29:18.872946Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", ResolveError { kind: NoRecordsFound { query: Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }, soa: Some(Record { name_labels: Name(\"gh.sankuai.com.\"), rr_type: SOA, dns_class: IN, ttl: 60, rdata: Some(SOA { mname: Name(\"dns1.sankuai.com.\"), rname: Name(\"hostmaster.sankuai.com.\"), serial: 2016116155, refresh: 18000, retry: 400, expire: 72000, minimum: 1800 }) }), negative_ttl: Some(56), response_code: NXDomain, trusted: true } })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-11T06:29:18.872988Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 8.466658754s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-11T06:29:27.343418Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", ResolveError { kind: NoRecordsFound { query: Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }, soa: Some(Record { name_labels: Name(\"gh.sankuai.com.\"), rr_type: SOA, dns_class: IN, ttl: 60, rdata: Some(SOA { mname: Name(\"dns1.sankuai.com.\"), rname: Name(\"hostmaster.sankuai.com.\"), serial: 2016116155, refresh: 18000, retry: 400, expire: 72000, minimum: 1800 }) }), negative_ttl: Some(48), response_code: NXDomain, trusted: true } })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-11T06:29:27.343459Z","level":"WARN","fields":{"message":"Retry attempt #3. Sleeping 712.948908ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-11T06:29:28.060341Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", ResolveError { kind: NoRecordsFound { query: Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }, soa: Some(Record { name_labels: Name(\"gh.sankuai.com.\"), rr_type: SOA, dns_class: IN, ttl: 60, rdata: Some(SOA { mname: Name(\"dns1.sankuai.com.\"), rname: Name(\"hostmaster.sankuai.com.\"), serial: 2016116155, refresh: 18000, retry: 400, expire: 72000, minimum: 1800 }) }), negative_ttl: Some(47), response_code: NXDomain, trusted: true } })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-11T06:29:28.060395Z","level":"WARN","fields":{"message":"Retry attempt #4. Sleeping 6.343477176s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-11T06:29:34.408660Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", ResolveError { kind: NoRecordsFound { query: Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }, soa: Some(Record { name_labels: Name(\"gh.sankuai.com.\"), rr_type: SOA, dns_class: IN, ttl: 60, rdata: Some(SOA { mname: Name(\"dns1.sankuai.com.\"), rname: Name(\"hostmaster.sankuai.com.\"), serial: 2016116155, refresh: 18000, retry: 400, expire: 72000, minimum: 1800 }) }), negative_ttl: Some(41), response_code: NXDomain, trusted: true } })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-11T06:29:34.408704Z","level":"ERROR","fields":{"message":"error invoking get_reconstruction api, error: ReqwestMiddlewareError(Middleware(Request failed after 5 retries\n\nCaused by:\n    0: error sending request for url (https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf)\n    1: client error (Connect)\n    2: dns error: no record found for Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }\n    3: no record found for Query { name: Name(\"cas-server.xethub.hf.co.gh.sankuai.com.\"), query_type: A, query_class: IN }))","caller":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs:286"},"filename":"/home/runner/work/xet-core/xet-core/error_printer/src/lib.rs","line_number":28}
tokenizer.json:   0%|                                                                                                                                                  | 0.00/11.4M [00:18<?, ?B/s]
Traceback (most recent call last):
  File "/home/hadoop-aipnlp/projects/nlp-llm-tools/file_download.py", line 29, in <module>
    hf_hub_download(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 988, in hf_hub_download
    return _hf_hub_download_to_local_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1290, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1710, in _download_to_tmp_and_move
    xet_get(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 627, in xet_get
    download_files(
RuntimeError: Data processing error: CAS service error : ReqwestMiddleware Error: Request failed after 5 retries

I also tried uninstalling hf_xet using pip uninstall hf_xet, and it works. From the logs, it looks like the download falls back to using HTTP, which works for me at the moment.

$ python file_download.py 
Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.4M/11.4M [00:02<00:00, 5.58MB/s]

[hoytak]

Thank you; I think I see the issue. Let me fix it and cut another rc candidate for you.

[hoytak]

@iNARI777 : Updated and cut another RC wheel for you. Could you try again with pip install hf_xet==1.1.4-rc2? Thank you!

I'm hopeful this fixes the issue; if not, there is one more thing to try but that will be a bit more complicated...

[iNARI777]

@hoytak Thank you! After update hf_xet to 1.1.4-rc2, I found the dns error still exists. It seems this is indeed a rather complex issue, haha!

$ python file_download.py 
tokenizer.json:   0%|                                                                                                                                                  | 0.00/11.4M [00:00<?, ?B/s]{"timestamp":"2025-06-12T02:42:15.347239Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-12T02:42:15.347276Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.737626166s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-12T02:42:17.104261Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-12T02:42:17.104294Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 2.171708714s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-12T02:42:19.295227Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-12T02:42:19.295294Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 3.237373301s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-12T02:42:22.556249Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-12T02:42:22.556293Z","level":"WARN","fields":{"message":"Retry attempt #3. Sleeping 6.591442449s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-12T02:42:29.160227Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-12T02:42:29.160279Z","level":"WARN","fields":{"message":"Retry attempt #4. Sleeping 3.366053355s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-12T02:42:32.543333Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":200}
{"timestamp":"2025-06-12T02:42:32.543371Z","level":"ERROR","fields":{"message":"error invoking get_reconstruction api, error: ReqwestMiddlewareError(Middleware(Request failed after 5 retries\n\nCaused by:\n    0: error sending request for url (https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf)\n    1: client error (Connect)\n    2: dns error: failed to lookup address information: Temporary failure in name resolution\n    3: failed to lookup address information: Temporary failure in name resolution))","caller":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs:286"},"filename":"/home/runner/work/xet-core/xet-core/error_printer/src/lib.rs","line_number":28}
tokenizer.json:   0%|                                                                                                                                                  | 0.00/11.4M [00:17<?, ?B/s]
Traceback (most recent call last):
  File "/home/hadoop-aipnlp/projects/nlp-llm-tools/file_download.py", line 29, in <module>
    hf_hub_download(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 988, in hf_hub_download
    return _hf_hub_download_to_local_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1290, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1710, in _download_to_tmp_and_move
    xet_get(
  File "/home/hadoop-aipnlp/conda/envs/llm/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 627, in xet_get
    download_files(
RuntimeError: Data processing error: CAS service error : ReqwestMiddleware Error: Request failed after 5 retries

[hoytak]

Okay, we hope we've fixed the issue (thanks to @Hugoch), and built another RC wheel to try. Could you do pip install hf_xet==1.1.4-rc3 now and see if that fixes it?

Thanks so much for working with us on this.

[nynj]

Upgraded to rc3, dns error still appears. Workaround mentioned previously pip uninstall hf_xet didn't work for me, still see the dns errors. Workaround mentioned previously pip uninstall hf_xet didn't work for me either.

[iNARI777]

@hoytak @Hugoch Thank you for your efforts! But the dns error still exists 😂

{"timestamp":"2025-06-13T02:46:12.535558Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":242}
{"timestamp":"2025-06-13T02:46:12.535590Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 450.929808ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-13T02:46:12.994026Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":242}
{"timestamp":"2025-06-13T02:46:12.994079Z","level":"WARN","fields":{"message":"Retry attempt #1. Sleeping 3.453365107s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-13T02:46:16.456574Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":242}
{"timestamp":"2025-06-13T02:46:16.456604Z","level":"WARN","fields":{"message":"Retry attempt #2. Sleeping 7.994401483s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-13T02:46:24.456806Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":242}
{"timestamp":"2025-06-13T02:46:24.456832Z","level":"WARN","fields":{"message":"Retry attempt #3. Sleeping 18.85937704s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-13T02:46:43.322890Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":242}
{"timestamp":"2025-06-13T02:46:43.322923Z","level":"WARN","fields":{"message":"Retry attempt #4. Sleeping 39.871424037s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.7.0/src/middleware.rs","line_number":171}
{"timestamp":"2025-06-13T02:47:23.200811Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf\", source: hyper_util::client::legacy::Error(Connect, ConnectError(\"dns error\", Custom { kind: Uncategorized, error: \"failed to lookup address information: Temporary failure in name resolution\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":242}
{"timestamp":"2025-06-13T02:47:23.200846Z","level":"ERROR","fields":{"message":"error invoking get_reconstruction api, error: ReqwestMiddlewareError(Middleware(Request failed after 5 retries\n\nCaused by:\n    0: error sending request for url (https://cas-server.xethub.hf.co/reconstruction/6aec39639a0a2d1ca966356b8c2b8426a484f80ff80731f44fa8482040713bdf)\n    1: client error (Connect)\n    2: dns error: failed to lookup address information: Temporary failure in name resolution\n    3: failed to lookup address information: Temporary failure in name resolution))","caller":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs:328"},"filename":"/home/runner/work/xet-core/xet-core/error_printer/src/lib.rs","line_number":28}

Is it possible that there is some misconfiguration in our private DNS?

[Hugoch]

Hey @iNARI777

Is it possible that there is some misconfiguration in our private DNS?

Yes, the fact that your DNS appends .gh.sankuai.com means that it does not recognize the domain hf.co as being absolute, so it adds the search domain.

Can you try to run a dig cas-server.xethub.hf.co on the machine on which you see the problem? Can you check that it gives the same result as dig cas-server.xethub.hf.co. ?

[iNARI777]

@Hugoch I found that the DNS issue was actually caused by the way I configured the HTTP proxy, but not the method of DNS parsing used by xet-core's http_client. Previously, I only set the proxy for the requests (as I shown in this issue's description), but it turns out that hf_xet does not use requests to send HTTP requests.

After I reconfigured the HTTP proxy using system environment variables HTTP_PROXY and HTTPS_PROXY, the downloads worked correctly. Now, everything works as expected with both hf_xet version 1.1.3 and 1.1.4-rc3.