[Bug]: Docker image CUDA error on RTX 2080 Ti

[arkadijs]

Your current environment

ψ python3 collect_env.py 
Traceback (most recent call last):
  File "/home/arkadi/collect_env.py", line 17, in <module>
    import regex as re
ModuleNotFoundError: No module named 'regex'

🐛 Describe the bug

The project-provided Docker vllm/vllm-openai 0.6.4.post1, v0.8.4, v0.8.5.post1, v0.9.0.1, v0.9.1 images doesn't work on a system with RTX 2080 Ti

NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8

The container runtime is configured correctly. Ollama, Whisper, Riva works no problem in Containerd environment.

time=2025-06-10T18:53:03.106Z level=INFO source=types.go:130 msg="inference compute" id=GPU-30de6d51-b12f-ca41-1081-26dbd109b1c2 library=cuda variant=v12 compute=7.5 driver=12.8 name="NVIDIA GeForce RTX 2080 Ti" total="10.6 GiB" available="10.4 GiB"

INFO 06-10 11:43:59 [__init__.py:244] Automatically detected platform cuda.                                                                                                                                    
WARNING 06-10 11:44:03 [utils.py:1416] argument 'device' is deprecated   
INFO 06-10 11:44:03 [api_server.py:1287] vLLM API server version 0.9.1rc1                                                                                                                                      
INFO 06-10 11:44:03 [cli_args.py:309] non-default args: {'model': 'Qwen/Qwen2.5-7B-Instruct-AWQ', 'max_model_len': 16000, 'max_seq_len_to_capture': 16000, 'served_model_name': ['qwen-2.5'], 'gpu_memory_utili
zation': 0.95, 'enable_prefix_caching': True, 'device': 'cuda', 'disable_log_stats': True}                                                                                                                     
INFO 06-10 11:44:12 [config.py:823] This model supports multiple tasks: {'generate', 'embed', 'score', 'reward', 'classify'}. Defaulting to 'generate'.
WARNING 06-10 11:44:13 [config.py:931] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
WARNING 06-10 11:44:13 [arg_utils.py:1635] Compute Capability < 8.0 is not supported by the V1 Engine. Falling back to V0.          
INFO 06-10 11:44:13 [api_server.py:265] Started engine process with PID 161
WARNING 06-10 11:44:15 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: https://github.com/NVIDIA/nccl/issues/1234           
INFO 06-10 11:44:17 [__init__.py:244] Automatically detected platform cuda.                                                                                                                                    
INFO 06-10 11:44:19 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.1rc1) with config: model='Qwen/Qwen2.5-7B-Instruct-AWQ', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct-AWQ', skip_tok
enizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=16000, download_dir=None, load_format=Load
Format.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig
(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=Non
e, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwen-2.5, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=Fal
se, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"i
nductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,1
68,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":256,"local_cache_dir":null}, use_cached_outputs=True,    
INFO 06-10 11:44:21 [cuda.py:279] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.                                                                                                               
INFO 06-10 11:44:21 [cuda.py:328] Using XFormers backend.     
Process SpawnProcess-1:                                                                                                                                                                                        
ERROR 06-10 11:44:22 [engine.py:458] Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[polarathene]

Is your nvidia-smi output from within the container or outside of it? Can you confirm it's the same in both cases?

Can you also run ldconfig -p | grep -F 'libcuda.so' in the container to see which path the CUDA driver is being loaded from?

Do note in your logs that it says The V1 engine does not support a Compute Capability below 8.0 (Ampere), and your GPU is Turing (CC 7.5). It likewise recognized FlashAttention-2 couldn't be used 🤔

Looking over the Dockerfile at least it seems like it was setup to build CUDA kernels for sm_75, but the CUDA libs would need to have sm_70 / sm_75 support as well for anything used there.

---

While this hasn't worked well for me in other projects when I tried it, you could try running the container with this ENV set LD_LIBRARY_PATH=/usr/local/cuda/compat, which is meant to fix the CUDA version and driver issue in some cases. For me though when I tried this it would fail to detect my GPU.

If the ldconfig command above instead showed that path (without setting the mentioned ENV), then you want to run just ldconfig in the container before it starts vllm, since the image build AFAIK unintentionally cached this compat path.

[polarathene]

Related: #18571

[arkadijs]

nvidia-smi doesn't work in the container and here is the ldconfig output:

root@vllm-7cc8c497b7-jc729:/vllm-workspace# nvidia-smi
Failed to initialize NVML: Unknown Error
root@vllm-7cc8c497b7-jc729:/vllm-workspace# ldconfig
root@vllm-7cc8c497b7-jc729:/vllm-workspace# nvidia-smi
Failed to initialize NVML: Unknown Error
root@vllm-7cc8c497b7-jc729:/vllm-workspace# ldconfig -p | grep -F 'libcuda.so'
        libcuda.so.1 (libc6,x86-64) => /usr/local/cuda-12.8/compat/libcuda.so.1
        libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so (libc6,x86-64) => /usr/local/cuda-12.8/compat/libcuda.so
        libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so

For comparison, here is also ldconfig output from Ollama container:

root@ollama-5f87996f4d-9szq6:/# ldconfig -p | grep -F 'libcuda.so'
        libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so

[polarathene]

OK thanks, try delete the ld cache in the container:

rm /etc/ld.so.cache
unset LD_LIBRARY_PATH

The last command might also help. Then try nvidia-smi again.

[arkadijs]

vLLM is able to start if ld.so.cache is removed.

vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:02:50 [__init__.py:244] Automatically detected platform cuda.
vllm-54c87b4cf9-ntqhx vllm WARNING 06-25 12:02:54 [utils.py:1416] argument 'device' is deprecated
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:02:54 [api_server.py:1287] vLLM API server version 0.9.1
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:02:54 [cli_args.py:309] non-default args: {'model': 'Qwen/Qwen2.5-7B-Instruct-AWQ', 'max_model_len': 16000, 'max_seq_len_to_capture': 16000, 'served_model_name': ['qwen-2.5'], 'gpu_memory_utilization': 0.95, 'enable_prefix_caching': True, 'device': 'cuda', 'disable_log_stats': True}
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:03 [config.py:823] This model supports multiple tasks: {'reward', 'score', 'generate', 'embed', 'classify'}. Defaulting to 'generate'.
vllm-54c87b4cf9-ntqhx vllm WARNING 06-25 12:03:05 [config.py:931] awq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
vllm-54c87b4cf9-ntqhx vllm WARNING 06-25 12:03:05 [arg_utils.py:1642] Compute Capability < 8.0 is not supported by the V1 Engine. Falling back to V0. 
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:05 [api_server.py:265] Started engine process with PID 163
vllm-54c87b4cf9-ntqhx vllm WARNING 06-25 12:03:06 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: https://github.com/NVIDIA/nccl/issues/1234
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:09 [__init__.py:244] Automatically detected platform cuda.
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:10 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.1) with config: model='Qwen/Qwen2.5-7B-Instruct-AWQ', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=16000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwen-2.5, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":256,"local_cache_dir":null}, use_cached_outputs=True, 
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:12 [cuda.py:275] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:12 [cuda.py:324] Using XFormers backend.
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:14 [parallel_state.py:1065] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:14 [model_runner.py:1171] Starting to load model Qwen/Qwen2.5-7B-Instruct-AWQ...
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:15 [weight_utils.py:292] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:09<00:09,  9.54s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:13<00:00,  6.24s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:13<00:00,  6.74s/it]
vllm-54c87b4cf9-ntqhx vllm 
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:29 [default_loader.py:272] Loading weights took 13.52 seconds
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:29 [model_runner.py:1203] Model loading took 5.2036 GiB and 14.356908 seconds
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:34 [worker.py:294] Memory profiling takes 4.30 seconds
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:34 [worker.py:294] the current vLLM instance can use total_gpu_memory (10.57GiB) x gpu_memory_utilization (0.95) = 10.04GiB
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:34 [worker.py:294] model weights take 5.20GiB; non_torch_memory takes 0.05GiB; PyTorch activation peak memory takes 2.26GiB; the rest of the memory reserved for KV Cache is 2.52GiB.
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:34 [executor_base.py:113] # cuda blocks: 2954, # CPU blocks: 4681
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:34 [executor_base.py:118] Maximum concurrency for 16000 tokens per request: 2.95x
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:03:39 [model_runner.py:1513] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:28<00:00,  1.21it/s]
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [model_runner.py:1671] Graph capturing finished in 29 secs, took 0.56 GiB
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [llm_engine.py:428] init engine (profile, create kv cache, warmup model) took 38.57 seconds
vllm-54c87b4cf9-ntqhx vllm WARNING 06-25 12:04:08 [config.py:1363] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [serving_chat.py:118] Using default chat sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [serving_completion.py:66] Using default completion sampling params from model: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [api_server.py:1349] Starting vLLM API server 0 on http://0.0.0.0:8000
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:29] Available routes are:
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /docs, Methods: GET, HEAD
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /redoc, Methods: GET, HEAD
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /health, Methods: GET
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /load, Methods: GET
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /ping, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /ping, Methods: GET
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /tokenize, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /detokenize, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/models, Methods: GET
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /version, Methods: GET
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/completions, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/embeddings, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /pooling, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /classify, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /score, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/score, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /rerank, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v1/rerank, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /v2/rerank, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /invocations, Methods: POST
vllm-54c87b4cf9-ntqhx vllm INFO 06-25 12:04:08 [launcher.py:37] Route: /metrics, Methods: GET
vllm-54c87b4cf9-ntqhx vllm INFO:     Started server process [21]
vllm-54c87b4cf9-ntqhx vllm INFO:     Waiting for application startup.
vllm-54c87b4cf9-ntqhx vllm INFO:     Application startup complete.

We're using NVIDIA_VISIBLE_DEVICES, btw.
nvidia-smi output (before vLLM):

vllm-7c9846d665-sh2ph vllm Wed Jun 25 12:07:31 2025       
vllm-7c9846d665-sh2ph vllm +-----------------------------------------------------------------------------------------+
vllm-7c9846d665-sh2ph vllm | NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
vllm-7c9846d665-sh2ph vllm |-----------------------------------------+------------------------+----------------------+
vllm-7c9846d665-sh2ph vllm | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
vllm-7c9846d665-sh2ph vllm | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
vllm-7c9846d665-sh2ph vllm |                                         |                        |               MIG M. |
vllm-7c9846d665-sh2ph vllm |=========================================+========================+======================|
vllm-7c9846d665-sh2ph vllm |   0  NVIDIA GeForce RTX 2080 Ti     Off |   00000000:3E:00.0 Off |                  N/A |
vllm-7c9846d665-sh2ph vllm | 29%   29C    P8              9W /  250W |       1MiB /  11264MiB |      0%      Default |
vllm-7c9846d665-sh2ph vllm |                                         |                        |                  N/A |
vllm-7c9846d665-sh2ph vllm +-----------------------------------------+------------------------+----------------------+
vllm-7c9846d665-sh2ph vllm                                                                                          
vllm-7c9846d665-sh2ph vllm +-----------------------------------------------------------------------------------------+
vllm-7c9846d665-sh2ph vllm | Processes:                                                                              |
vllm-7c9846d665-sh2ph vllm |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
vllm-7c9846d665-sh2ph vllm |        ID   ID                                                               Usage      |
vllm-7c9846d665-sh2ph vllm |=========================================================================================|
vllm-7c9846d665-sh2ph vllm |  No running processes found                                                             |
vllm-7c9846d665-sh2ph vllm +-----------------------------------------------------------------------------------------+

[arkadijs]

Sidenote, the v0.8.5.post1 image is the last version that actually works on RTX 2080.
v0.9.x fails on second request, something to do with the prefix cache.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[arkadijs]

Let's keep this open or there is no actionables from the project to fix the issue?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!