Certificate Verify Failed cas-server vs. cas-bridge

[marked23]

When using:

model_name = "Qwen/Qwen2.5-Coder-7B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cpu"
)

It tries to download from:

https://cas-server.xethub.hf.co/reconstruction/ead...

But the certificate that the server responds with, is for:

https://cas-bridge.xethub.hf.co

I was able to see this by running this command:

$ openssl s_client -connect cas-server.xethub.hf.co:443 -servername cas-server. xethub.hf.co

Connecting to 52.71.209.178
CONNECTED(00000003)
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M03
verify return:1
depth=0 CN=cas-bridge.xethub.hf.co
verify return:1

Certificate chain
0 s:CN=cas-bridge.xethub.hf.co
i:C=US, O=Amazon, CN=Amazon RSA 2048 M03
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 29 00:00:00 2025 GMT; NotAfter: Feb 27 23:59:59 2026 GMT

Here are some of the error messages:

"timestamp":"2025-05-26T21:08:23.139684Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/ead555725919f18abfb5b674d5438dee531369f4e77db078ffb3a50dec0d3f12\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: \"SSL routines\", function: \"tls_post_process_server_certificate\", reason: \"certificate verify failed\", file: \"ssl/statem/statem_clnt.c\", line: 2092 }]))) }, X509VerifyResult { code: 20, error: \"unable to get local issuer certificate\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":175}
{"timestamp":"2025-05-26T21:08:23.139733Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.407907729s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.6.1/src/middleware.rs","line_number":166}
                                                                                                                                                          {"timestamp":"2025-05-26T21:08:23.142706Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/1626213712be57d779fda6b59cca00091f1fb2664416a49c01d0264a3f5462e0\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: \"SSL routines\", function: \"tls_post_process_server_certificate\", reason: \"certificate verify failed\", file: \"ssl/statem/statem_clnt.c\", line: 2092 }]))) }, X509VerifyResult { code: 20, error: \"unable to get local issuer certificate\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":175}
{"timestamp":"2025-05-26T21:08:23.142772Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.218760805s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.6.1/src/middleware.rs","line_number":166}
{"timestamp":"2025-05-26T21:08:23.150914Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/a50ecfc9f1368880975b7cc77bb53a6d27746bc651c742251787ad6cbac0a926\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: \"SSL routines\", function: \"tls_post_process_server_certificate\", reason: \"certificate verify failed\", file: \"ssl/statem/statem_clnt.c\", line: 2092 }]))) }, X509VerifyResult { code: 20, error: \"unable to get local issuer certificate\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":175}
{"timestamp":"2025-05-26T21:08:23.150940Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.592819081s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.6.1/src/middleware.rs","line_number":166}

[rajatarya]

What is the version of hf-xet in your Python environment? Can you share the output of pip freeze? What version of Python are you using? What platform is this running on? Are you building the hf-xet whl from source?

[bpronan]

@marked23: Thanks for catching that. I've adjusted the certificate for cas-server.xethub.hf.co. Could you please give it a another try?

[marked23]

It's still happening for me. Although I see the certificate it's offering me is is a different CN.
CN=hf.co If this is meant to be a wildcard cert, I would expect to see
CN=*.hf.co

$ openssl s_client -connect cas-server.xethub.hf.co:443 -servername cas-server.xethub.hf.co returns:

Connecting to 18.172.170.39
CONNECTED(00000003)
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M03
verify return:1
depth=0 CN=hf.co
verify return:1

Certificate chain
0 s:CN=hf.co
i:C=US, O=Amazon, CN=Amazon RSA 2048 M03
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 25 00:00:00 2024 GMT; NotAfter: Oct 24 23:59:59 2025 GMT
1 s:C=US, O=Amazon, CN=Amazon RSA 2048 M03
i:C=US, O=Amazon, CN=Amazon Root CA 1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 23 22:26:04 2022 GMT; NotAfter: Aug 23 22:26:04 2030 GMT
2 s:C=US, O=Amazon, CN=Amazon Root CA 1
i:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT

I still see the same behavior.

$ python ./test_qwen.py 
Testing Qwen/Qwen2.5-Coder-7B...
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.31k/7.31k [00:00<00:00, 25.2MB/s]
vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.78M/2.78M [00:00<00:00, 27.7MB/s]
merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.67M/1.67M [00:00<00:00, 19.5MB/s]
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.03M/7.03M [00:00<00:00, 25.9MB/s]
Tokenizer loaded
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 668/668 [00:00<00:00, 11.0MB/s]
model.safetensors.index.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 27.8k/27.8k [00:00<00:00, 115MB/s]
Fetching 4 files:   0%|                                                                                                                                              | 0/4 [00:00<?, ?it/s{"timestamp":"2025-05-29T00:28:00.334359Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/cd9b3569e15af48b5338d1f02bf99476542809310dde89f1a4301215b1a8a81d\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: \"SSL routines\", function: \"tls_post_process_server_certificate\", reason: \"certificate verify failed\", file: \"ssl/statem/statem_clnt.c\", line: 2092 }]))) }, X509VerifyResult { code: 20, error: \"unable to get local issuer certificate\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":175}
{"timestamp":"2025-05-29T00:28:00.334421Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 594.452111ms before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.6.1/src/middleware.rs","line_number":166}
{"timestamp":"2025-05-29T00:28:00.341040Z","level":"WARN","fields":{"message":"Reqwest(reqwest::Error { kind: Request, url: \"https://cas-server.xethub.hf.co/reconstruction/1626213712be57d779fda6b59cca00091f1fb2664416a49c01d0264a3f5462e0\", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: \"SSL routines\", function: \"tls_post_process_server_certificate\", reason: \"certificate verify failed\", file: \"ssl/statem/statem_clnt.c\", line: 2092 }]))) }, X509VerifyResult { code: 20, error: \"unable to get local issuer certificate\" })) }). Retrying..."},"filename":"/home/runner/work/xet-core/xet-core/cas_client/src/http_client.rs","line_number":175}
{"timestamp":"2025-05-29T00:28:00.341080Z","level":"WARN","fields":{"message":"Retry attempt #0. Sleeping 1.995050362s before the next attempt"},"filename":"/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-retry-0.6.1/src/middleware.rs","line_number":166}

This is how I'm testing it. I tried it using Qwen2 and also Auto, just in case that matters.

import os
# os.environ['HF_HUB_DISABLE_XET'] = '1'
# os.environ['HF_XET_DISABLED'] = '1'

from transformers import Qwen2Tokenizer, Qwen2ForCausalLM, AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "Qwen/Qwen2.5-Coder-7B"
print(f"Testing {model_name}...")

try:
    # tokenizer = Qwen2Tokenizer.from_pretrained(model_name, trust_remote_code=True)
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
    print("Tokenizer loaded")
    
    # model = Qwen2ForCausalLM.from_pretrained(
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        trust_remote_code=True,
        torch_dtype=torch.bfloat16,
        device_map="cpu"
    )
    print("Model loaded")
    
except Exception as e:
    print(f"Error: {e}")

@rajatarya
Python 3.12.7
Ubuntu 24.10
hf-xet not built from source.

accelerate==1.7.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.1
aiosignal==1.3.2
attrs==25.3.0
bitsandbytes==0.43.3.dev0
certifi==2025.4.26
charset-normalizer==3.4.2
contourpy==1.3.2
cycler==0.12.1
datasets==3.6.0
dill==0.3.8
einops==0.8.1
filelock==3.18.0
fonttools==4.58.0
frozenlist==1.6.0
fsspec==2025.3.0
GPUtil==1.4.0
hf-xet==1.1.2
huggingface-hub==0.32.1
idna==3.10
iniconfig==2.1.0
inquirerpy==0.3.4
Jinja2==3.1.6
kiwisolver==1.4.8
lion-pytorch==0.2.3
MarkupSafe==3.0.2
matplotlib==3.9.4
mpmath==1.3.0
multidict==6.4.4
multiprocess==0.70.16
networkx==3.4.2
numpy==2.2.6
packaging==25.0
pandas==2.2.3
peft==0.15.2
pfzy==0.3.4
pillow==11.2.1
pluggy==1.6.0
prompt_toolkit==3.0.51
propcache==0.3.1
psutil==7.0.0
pyarrow==20.0.0
pyparsing==3.2.3
pytest==8.3.5
python-dateutil==2.9.0.post0
pytorch-triton-rocm @ https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4.1/pytorch_triton_rocm-3.2.0%2Brocm6.4.1.git6da9e660-cp312-cp312-linux_x86_64.whl#sha256=1d97c15798bf178299328032141a21d9777e7cdef59d5a7e3ac74e297c17198e
pytz==2025.2
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.3
safetensors==0.5.3
scipy==1.14.1
sentencepiece==0.2.0
setuptools==80.8.0
six==1.17.0
sympy==1.13.1
tokenizers==0.21.1
torch @ https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4.1/torch-2.6.0%2Brocm6.4.1.git1ded221d-cp312-cp312-linux_x86_64.whl#sha256=6b141e1a03148b007c6217519cd9947d760123ded5caebadffec22cba7358d2d
torchaudio @ https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4.1/torchaudio-2.6.0%2Brocm6.4.1.gitd8831425-cp312-cp312-linux_x86_64.whl#sha256=d93faa440d5e9bb80146a33d85e163946c8006ed25db7e0bfcd8ea5284ea35ce
torchvision @ https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4.1/torchvision-0.21.0%2Brocm6.4.1.git4040d51f-cp312-cp312-linux_x86_64.whl#sha256=a532d46e26341741ed9929ab3e4d763597d2bfec62ae0b9a046d18bae0f68696
tqdm==4.67.1
transformers==4.52.3
typing_extensions==4.13.2
tzdata==2025.2
urllib3==2.4.0
wcwidth==0.2.13
wheel==0.43.0
xxhash==3.5.0
yarl==1.20.0

[bpronan]

@marked23: sorry for the slow response. We had some bizarre issues on this side preventing us from consistently returning the correct cert for cas-server.xethub.hf.co. These issues have now been resolved.

We should now always return the a certificate that has *.xethub.hf.co as the primary domain name for the cert. This should match cas-server on even the most strict SSL cert checks.

I've confirmed on my side that the call to openssl s_client -connect cas-server.xethub.hf.co:443 -servername cas-server.xethub.hf.co is consistently returning:

...
-----END CERTIFICATE-----
subject=CN=*.xethub.hf.co
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M02
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: rsa_pkcs1_sha256
Peer Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 4472 bytes and writt
...

Could you give please give it another try?

Thanks for your patience. My apologies for the slow turnaround time.

[marked23]

Yes, the openssl certificate check now responds with the expected wildcard cert. But there's more:

mark@four:~$ openssl s_client -connect cas-server.xethub.hf.co:443 -servername cas-server.xethub.hf.co
Connecting to 54.166.72.233
CONNECTED(00000003)
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M02
verify return:1
depth=0 CN=*.xethub.hf.co
verify return:1

I needed to correct my hosts file, which was:

mark@wide:~$ cat /etc/hosts
# Standard host addresses
127.0.0.1  localhost
::1        localhost ip6-localhost ip6-loopback
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
# This host address
127.0.1.1  wide
18.172.170.80
18.172.170.55
18.172.170.94
18.172.170.39 cas-server.xethub.hf.co

mark@wide:~$ ll /etc | grep -i hosts
drwxr-xr-x 4 root root 4096 Oct 7 2024 ghostscript/
-rw-r--r-- 1 root root 261 May 26 14:04 hosts
-rw-r--r-- 1 root root 261 Jun 6 13:19 hosts.0.1
-rw-r--r-- 1 root root 411 Oct 7 2024 hosts.allow
-rw-r--r-- 1 root root 711 Oct 7 2024 hosts.deny

I don't think I human-edited my hosts file on May 26th. Perhaps an installer did. Anyway, once I removed those Amazon IPs from my hosts file, the openssl check started returning the expected response.

However, in spite of the fact that I'm getting a good response to this openssl check. I still get the same Reqwest certificate error as seen in my first message.

[hoytak]

Implemented several fixes based on this investigation (thanks @marked23). Could you try pip install hf_xet==1.1.4-rc1 and see if that fixes the issue?

[marked23]

1.1.4-rc1 works for me. Thank you.