TEI 1.8.1 report "ERROR" when the "Qwen3-Embedding-0.6B-ONNX" model is used

[quaggalinux]

I already tried:
ghcr.io/huggingface/text-embeddings-inference:cpu-ipex-latest

and

ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.1

/# docker run --name tei-embedding -p 8080:80 -v /:/data -e MAX_CLIENT_BATCH_SIZE=4096 ghcr.io/huggingface/text-embeddings-inference:cpu-ipex-latest --pooling last-token --model-id /data/Qwen3-Embedding-0.6B-ONNX
2025-09-08T00:38:24.100856Z  INFO text_embeddings_router: router/src/main.rs:203: Args { model_id: "/dat*/*****-*********-*.**-*NNX", revision: None, tokenization_workers: None, dtype: None, pooling: Some(LastToken), max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 4096, auto_truncate: false, default_prompt_name: None, default_prompt: None, dense_path: None, hf_api_token: None, hf_token: None, hostname: "993cfff8d5ab", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }
2025-09-08T00:38:24.444390Z  WARN text_embeddings_router: router/src/lib.rs:190: Could not find a Sentence Transformers config
2025-09-08T00:38:24.444410Z  INFO text_embeddings_router: router/src/lib.rs:194: Maximum number of tokens per request: 32768
2025-09-08T00:38:24.444529Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:38: Starting 8 tokenization workers
2025-09-08T00:38:24.771051Z  INFO text_embeddings_router: router/src/lib.rs:242: Starting model backend
2025-09-08T00:38:24.773832Z  INFO text_embeddings_backend_python::management: backends/python/src/management.rs:68: Starting Python backend
2025-09-08T00:38:31.867929Z  INFO python-backend: text_embeddings_backend_python::logging: backends/python/src/logging.rs:37: backend device: cpu


  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.11/dist-packages/typer/core.py", line 716, in main
    return _main(
  File "/usr/local/lib/python3.11/dist-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.11/dist-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/src/backends/python/server/text_embeddings_server/cli.py", line 51, in serve
    server.serve(model_path, dtype, uds_path, pool)
  File "/usr/src/backends/python/server/text_embeddings_server/server.py", line 92, in serve
    asyncio.run(serve_inner(model_path, dtype))
  File "/usr/lib/python3.11/asyncio/runners.py", line 188, in run
    return runner.run(main)
  File "/usr/lib/python3.11/asyncio/runners.py", line 120, in run
    return self._loop.run_until_complete(task)
  File "/usr/lib/python3.11/asyncio/base_events.py", line 637, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 604, in run_forever
    self._run_once()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
> File "/usr/src/backends/python/server/text_embeddings_server/server.py", line 61, in serve_inner
    model = get_model(model_path, dtype, pool)
  File "/usr/src/backends/python/server/text_embeddings_server/models/__init__.py", line 137, in get_model
    return create_model(DefaultModel, model_path, device, datatype, pool)
  File "/usr/src/backends/python/server/text_embeddings_server/models/__init__.py", line 53, in create_model
    model_handle = model_class(
  File "/usr/src/backends/python/server/text_embeddings_server/models/default_model.py", line 26, in __init__
    AutoModel.from_pretrained(model_path, trust_remote_code=trust_remote)
  File "/usr/local/lib/python3.11/dist-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 279, in _wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 4260, in from_pretrained
    checkpoint_files, sharded_metadata = _get_resolved_checkpoint_files(
  File "/usr/local/lib/python3.11/dist-packages/transformers/modeling_utils.py", line 952, in _get_resolved_checkpoint_files
    raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory /data/Qwen3-Embedding-0.6B-ONNX.

2025-09-08T00:38:32.767525Z ERROR text_embeddings_backend: backends/src/lib.rs:474: Could not start Python backend: Could not start backend: Python backend failed to start
Error: Could not create backend

Caused by:
    Could not start backend: Could not start a suitable backend

/# docker run --name tei-embedding -p 8080:80 -v /:/data -e MAX_CLIENT_BATCH_SIZE=4096 ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.1 --pooling last-token --model-id /data/Qwen3-Embedding-0.6B-ONNX
2025-09-08T00:37:11.549751Z  INFO text_embeddings_router: router/src/main.rs:203: Args { model_id: "/dat*/*****-*********-*.**-*NNX", revision: None, tokenization_workers: None, dtype: None, pooling: Some(LastToken), max_concurrent_requests: 512, max_batch_tokens: 16384, max_batch_requests: None, max_client_batch_size: 4096, auto_truncate: false, default_prompt_name: None, default_prompt: None, dense_path: None, hf_api_token: None, hf_token: None, hostname: "1f80d7fce660", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }
2025-09-08T00:37:11.898502Z  WARN text_embeddings_router: router/src/lib.rs:190: Could not find a Sentence Transformers config
2025-09-08T00:37:11.898517Z  INFO text_embeddings_router: router/src/lib.rs:194: Maximum number of tokens per request: 32768
2025-09-08T00:37:11.898638Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:38: Starting 8 tokenization workers
2025-09-08T00:37:12.222672Z  INFO text_embeddings_router: router/src/lib.rs:242: Starting model backend
2025-09-08T00:37:12.696602Z ERROR text_embeddings_backend: backends/src/lib.rs:402: Could not start ORT backend: Could not start backend: Deserialize tensor model.layers.14.attn.v_proj.MatMul.weight failed.tensorprotoutils.cc:1091 GetExtDataFromTensorProto External initializer: model.layers.14.attn.v_proj.MatMul.weight offset: 1531400192 size to read: 4194304 given file_length: 135 are out of bounds or can not be read in full.
2025-09-08T00:37:12.698175Z ERROR text_embeddings_backend: backends/src/lib.rs:448: Could not start Candle backend: Could not start backend: No such file or directory (os error 2)
Error: Could not create backend

Caused by:
    Could not start backend: Could not start a suitable backend

[alvarobartt]

Hey @quaggalinux, thanks for reporting!

The issue seems to be with the model files rather than the container, as per the commands above I guess that you're using the ONNX export from https://huggingface.co/onnx-community/Qwen3-Embedding-0.6B-ONNX, which is downloaded in advance with hf download and mounted in /data/Qwen3-Embedding-0.6B-ONNX, right?

I just tried the following and it worked just fine:

$ docker run --platform linux/amd64 -p 3000:80 ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.1 --model-id onnx-community/Qwen3-Embedding-0.6B-ONNX --dtype float32 --pooling last-token --max-batch-tokens 1024

2025-09-09T08:59:19.405095Z  INFO text_embeddings_router: router/src/main.rs:203: Args { model_id: "onn*-*********/*****-*********-*.**-*NNX", revision: None, tokenization_workers: None, dtype: Some(Float32), pooling: Some(LastToken), max_concurrent_requests: 512, max_batch_tokens: 1024, max_batch_requests: None, max_client_batch_size: 32, auto_truncate: false, default_prompt_name: None, default_prompt: None, dense_path: None, hf_api_token: None, hf_token: None, hostname: "a4714ad0ec67", port: 80, uds_path: "/tmp/text-embeddings-inference-server", huggingface_hub_cache: Some("/data"), payload_limit: 2000000, api_key: None, json_output: false, disable_spans: false, otlp_endpoint: None, otlp_service_name: "text-embeddings-inference.server", prometheus_port: 9000, cors_allow_origin: None }
2025-09-09T08:59:19.547872Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:42: Starting download
2025-09-09T08:59:19.548384Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_bert_config.json`
2025-09-09T08:59:19.841392Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_roberta_config.json`
2025-09-09T08:59:20.002047Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_distilbert_config.json`
2025-09-09T08:59:20.149762Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_camembert_config.json`
2025-09-09T08:59:20.284605Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_albert_config.json`
2025-09-09T08:59:20.415351Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_xlm-roberta_config.json`
2025-09-09T08:59:20.542892Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `sentence_xlnet_config.json`
2025-09-09T08:59:20.671193Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `config_sentence_transformers.json`
2025-09-09T08:59:20.814183Z  WARN download_artifacts: text_embeddings_core::download: core/src/download.rs:65: Download failed: request error: HTTP status client error (404 Not Found) for url (https://huggingface.co/onnx-community/Qwen3-Embedding-0.6B-ONNX/resolve/main/config_sentence_transformers.json)
2025-09-09T08:59:20.814826Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `config.json`
2025-09-09T08:59:21.229447Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:18: Downloading `tokenizer.json`
2025-09-09T08:59:22.090201Z  INFO download_artifacts: text_embeddings_core::download: core/src/download.rs:72: Model artifacts downloaded in 2.542203543s
2025-09-09T08:59:22.359213Z  WARN text_embeddings_router: router/src/lib.rs:190: Could not find a Sentence Transformers config
2025-09-09T08:59:22.359266Z  INFO text_embeddings_router: router/src/lib.rs:194: Maximum number of tokens per request: 32768
2025-09-09T08:59:22.359983Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:38: Starting 12 tokenization workers
2025-09-09T08:59:22.492484Z  INFO text_embeddings_router: router/src/lib.rs:242: Starting model backend
2025-09-09T08:59:22.494134Z  INFO text_embeddings_backend: backends/src/lib.rs:594: Downloading `model.onnx`
2025-09-09T08:59:22.624971Z  WARN text_embeddings_backend: backends/src/lib.rs:598: Could not download `model.onnx`: request error: HTTP status client error (404 Not Found) for url (https://huggingface.co/onnx-community/Qwen3-Embedding-0.6B-ONNX/resolve/main/model.onnx)
2025-09-09T08:59:22.625319Z  INFO text_embeddings_backend: backends/src/lib.rs:599: Downloading `onnx/model.onnx`
2025-09-09T08:59:33.133952Z  INFO text_embeddings_backend: backends/src/lib.rs:608: Downloading `model.onnx_data`
2025-09-09T08:59:33.261556Z  WARN text_embeddings_backend: backends/src/lib.rs:612: Could not download `model.onnx_data`: request error: HTTP status client error (404 Not Found) for url (https://huggingface.co/onnx-community/Qwen3-Embedding-0.6B-ONNX/resolve/main/model.onnx_data)
2025-09-09T08:59:33.261840Z  INFO text_embeddings_backend: backends/src/lib.rs:613: Downloading `onnx/model.onnx_data`
2025-09-09T09:00:46.999196Z  INFO text_embeddings_backend: backends/src/lib.rs:381: Model ONNX weights downloaded in 84.504929996s
2025-09-09T09:00:46.999300Z  INFO text_embeddings_backend: backends/src/lib.rs:389: Downloading `tokenizer_config.json`
2025-09-09T09:00:48.469176Z  INFO text_embeddings_router: router/src/lib.rs:260: Warming up model
2025-09-09T09:00:52.980357Z  WARN text_embeddings_router: router/src/lib.rs:269: Backend does not support a batch size > 8
2025-09-09T09:00:52.980424Z  WARN text_embeddings_router: router/src/lib.rs:270: forcing `max_batch_requests=8`
2025-09-09T09:00:52.980924Z  WARN text_embeddings_router: router/src/lib.rs:319: Invalid hostname, defaulting to 0.0.0.0
2025-09-09T09:00:53.214119Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1852: Starting HTTP server: 0.0.0.0:80
2025-09-09T09:00:53.214147Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1853: Ready
2025-09-09T09:01:20.286462Z  INFO embed{total_time="76.728459ms" tokenization_time="17.343208ms" queue_time="5.980333ms" inference_time="51.0985ms"}: text_embeddings_router::http::server: router/src/http/server.rs:733: Success

Note

Note that I included the --max-batch-tokens 1024 to reduce the number of tokens that can fit in a batch, otherwise, given that Qwen3 has max_position_embeddings = 32768 it means that each sequence can have up to 32768 tokens. Meaning that 32768 tokens with a batch size of 8 it's probable that you run OOM, so I decreased it to prevent the warmup from leading to an OOM. But if you have enough memory it should work out of the box without the --max-batch-tokens override.

Also you shouldn't use the CPU IPEX image for ONNX exports, but rather the default CPU image, as CPU IPEX will use PyTorch + IPEX for the inference with Python i.e. not the ONNX Runtime (ORT), so it will fail with ONNX weights, but should work just fine with the default weights at https://huggingface.co/Qwen/Qwen3-Embedding-0.6B.

Finally, note that as per #715 the intel-mkl-src support for CPU has been fixed, as it was preventing users from running non-ONNX models on CPU, but will now be fixed in v1.8.2 meaning you should be able to also run https://huggingface.co/Qwen/Qwen3-Embedding-0.6B with the ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.2 image.

Please let me know if the above command works for you and, close the issue if so, otherwise I'd suggest re-downloading the ONNX weights to be sure those are correct and not-corrupted, or rather provide more information to reproduce 🤗 Thanks in advance!

[quaggalinux]

It worked perfectly when I used the command line you suggested.

I also used version 1.8.2 for both the ONNX and non-ONNX models.

Thank you very much for your help!

[alvarobartt]

Hey @quaggalinux that's awesome news, glad it worked just fine, and apologies it didn't in the first place 🤗