[Bug]: Fail to load vLLM on new NVIDIA driver

[huydhn]

Your current environment

I can't run collect_env.py because importing PyTorch itself is failing.

🐛 Describe the bug

After #30784, I start seeing vLLM failing to load on its benchmark job. I suspect that the change doesn't work with newer NVIDIA driver that the job is using to be compatible with CUDA 13.0

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA B200                    Off |   00000000:D1:00.0 Off |                    0 |
| N/A   32C    P0            141W /  750W |       0MiB / 183359MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

With #30784, importing PyTorch fails right away:

python3 -c 'import torch; torch.cuda.is_available()'
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:182: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.)
  return torch._C._cuda_getDeviceCount() > 0

Here is an example failure https://github.com/pytorch/pytorch-integration-testing/actions/runs/21017403967/job/60426060877#step:19:1452

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[wangshangsam]

Skipping import of cpp extensions due to incompatible torch version 2.9.1+cu129 for torchao version 0.14.1

Should the cu130 version of torch being used in this case?

[huydhn]

I'm using the Docker image from vLLM public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:2a60ac91d0f5c24cdb2863b178d2f5405fae50b8 and run it on a server using 580.105.08 NVIDIA driver which is compatible with CUDA 13.0. The system CUDA runtime is fine, but #30784 now tries to load CUDA libraries under /usr/local/cuda-12.9/compat first, which in this case, is older than what the system provides. And this causes the mismatch.

[wangshangsam]

Ah ... damn ...

I'm wondering then, why (before that PR)

# Workaround for triton/pytorch issues
RUN ldconfig /usr/local/cuda-$(echo $CUDA_VERSION | cut -d. -f1,2)/compat/

exists in the Dockerfile in the first place? Seems like it's not really doing anything?

(The reason why I'm saying this is, before that PR, cu130 was working, so I can only assume that was because the ldconfig didn't take effect?)

[huydhn]

Yeah, before that change, I don't think I see it shows up, i.e. no /usr/local/cuda-12.9/compat showing up here:

root@429748c09fce:/tmp/workspace# ldconfig -p | grep cuda
	libnvrtc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so.12
	libnvrtc.alt.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.alt.so.12
	libnvrtc-builtins.so.12.9 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so.12.9
	libnvrtc-builtins.alt.so.12.9 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.alt.so.12.9
	libnvblas.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvblas.so.12
	libicudata.so.70 (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so.70
	libcurand.so.10 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so.10
	libcurand.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so
	libcudart.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12
	libcudart.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
	libcudadebugger.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcudadebugger.so.1
	libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
	libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so
	libcublasLt.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.12
	libcublas.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.12

https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html mentions the compatibility list, which explains the mismatch

[wangshangsam]

I'm still a bit bizarred by the original problem that @emricksini-h (sorry!) was facing - like, then what would be the proper way to enable CUDA compatibility mode? What @emricksini-h observed seem to indicate that CUDA compatibility mode wasn't working for the 12.x in the first place?

[emricksini-h]

To answer your question @wangshangsam

I'm wondering then, why (before that PR)

# Workaround for triton/pytorch issues
RUN ldconfig /usr/local/cuda-$(echo $CUDA_VERSION | cut -d. -f1,2)/compat/

I've checked, and it was working precisely because of the issue I was originally trying to fix. This particular workflow launches the base vLLM image with docker exec -t 6fcd86412e2eafc1ffc437e8b0429f47ffe3eecee928b0057a5cbd868f88f398 bash -c 'cd vllm-benchmarks/vllm && bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh.
Within that script, apt-get gets called:

main() {
  ...

  # dependencies
  (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
  (which jq) || (apt-get update && apt-get -y install jq)
  (which lsof) || (apt-get update && apt-get install -y lsof)
}

This triggers an ldconfig update which resets the cache (wiping the ephemeral compat entry). This allowed the system to fall back to LD_LIBRARY_PATH, letting the host's CUDA 13.0 drivers take precedence as intended.

Sorry for introducing this regression, and thanks @huydhn for the notice! I hadn't considered the case where the host CUDA version is newer than the container's (where the compatibility layer isn't required).

I agree we should revert the change to ld.so.conf.d. To solve the original persistence issue without breaking newer drivers, I will instead try to append the compat path to LD_LIBRARY_PATH in the Dockerfile.

[ehfd]

Because v0.14.0 is already out now, is this considered a regression in v0.14.0?

[wangshangsam]

Because v0.14.0 is already out now, is this considered a regression in v0.14.0?

I actually couldn't reproduce this issue on GB200: #32474 (comment)