Describe the Bug
When attempting to launch dynamo serve using the vLLM backend on a machine with 2×H100 GPUs, CUDA 12.8, and NVIDIA driver version 570.133.20, the system fails at runtime with the following error:
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination
The error occurs during GPU initialization by the VllmWorker component, even though nvidia-smi confirms that CUDA and the H100 GPUs are visible both on the host and inside the container. As a result, the model engine fails to start and inference cannot proceed.
Additionally, the Planner component later fails due to a missing dynamo.json state file under /root/.dynamo/state.
This seems to be a runtime compatibility issue specific to this driver/CUDA combination or Docker setup.
Steps to Reproduce
1. Create a compatible Python environment
The python3 -m venv method installs Python 3.13.2 by default on Ubuntu 24.04, which is not supported by ai-dynamo-runtime. To avoid dependency resolution issues, I used Conda to create a Python 3.12 environment, which is supported by ai-dynamo-runtime:
conda create --name dynamo-test2 python=3.12
conda activate dynamo-test2
2. Install the ai-dynamo[all] package
pip install "ai-dynamo[all]"
Running a model
3. Fix shared library loading issue
When attempting to run a model with dynamo run (more specifically : dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B) I initially hit a libpython3.12.so.1.0 error. To resolve it, I updated LD_LIBRARY_PATH:
export LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"
Then I confirmed model loading works with:
dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B. I was able to chat with the model.
Serving a model
I then wanted to serve the model specifically using --framework vllm so I followed the official instructions in https://github.com/ai-dynamo/dynamo/blob/main/examples/llm/README.md
4. Start infrastructure services
Note: You must clone the repo first (git clone https://github.com/ai-dynamo/dynamo.git) to access the Docker configs.
cd dynamo
docker compose -f deploy/metrics/docker-compose.yml up -d
5. Launch the containerized vLLM runtime
According to the docs, the next step is to run the container:
./container/run.sh -it --framework vllm
Immediately upon container launch, I see this error in the logs:
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available. [[ System has unsupported display driver / cuda driver combination (error 803) ]]
Despite the error, I'm dropped into a shell at /workspace.
6. Attempt to serve the model
Navigate to the LLM example and run:
cd $DYNAMO_HOME/examples/llm dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml
This fails with the following critical error during cudaGetDeviceCount(): RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination.
I will attach the full output when running dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml to the in Actual Behavior Section in case it is useful:
7. Confirm GPUs are visible
I ran nvidia-smi inside the container immediately after the failure:
nvidia-smi
✅ Both H100 GPUs are detected and idle.
✅ CUDA 12.8 and Driver 570.133.20 appear to be installed correctly.
Expected Behavior
Expected Behavior
I expected the model to be successfully served using dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml, with vLLM initializing the GPU backend properly and no CUDA-related errors. Specifically:
- The
VllmWorker component should initialize without failure.
- CUDA should be recognized and initialized correctly inside the container.
- Inference requests should be accepted at the exposed endpoint (
localhost:8000/chat/completions).
Actual Behavior
Actual Behavior
When attempting to serve the model using dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml, the following occurred:
-
The container successfully started, and all components (Frontend, Planner, Processor, VllmWorker) began initialization.
-
However, during VllmWorker initialization, the following error occurred:
RuntimeError: Unexpected error from cudaGetDeviceCount(): Error 803: system has unsupported display driver / cuda driver combination
-
This caused the engine process to fail, and consequently:
- The
VllmWorker crashed.
- The planner later also failed due to a missing state file:
FileNotFoundError: State file not found: /root/.dynamo/state/dynamo.json
-
Notably, nvidia-smi inside the container does detect the H100 GPUs correctly and shows no running processes, so CUDA visibility itself appears to be working — but initialization fails.
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize.
[[ System has unsupported display driver / cuda driver combination (error 803) ]]
This blocks any ability to serve inference workloads using vLLM through Dynamo.
I have attached a file with the entire output of dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml here.
root@_MY_MACHINE___workspace_examples_llm# dynamo serve graphs.txt
Environment
From nvidia-smi
-
GPU Model: 2 × NVIDIA H100 NVL
-
GPU Memory: 95,830 MiB each (0 MiB in use at error time)
-
Driver Version: 570.133.20
-
CUDA Version (reported by nvidia-smi): 12.8
From dynamo env:
System Information:
OS: Ubuntu 24.04
Glibc Version: 2.39
GCC Version: 13.3.0
Cmake Version: 3.28.3
Rust Version: 1.86.0
Docker Version: N/A
CPU Information:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Vendor ID: AuthenticAMD
Model name: AMD EPYC 9334 32-Core Processor
CPU family: 25
Model: 17
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
Stepping: 1
Frequency boost: enabled
CPU(s) scaling MHz: 60%
CPU max MHz: 3910.2529
CPU min MHz: 1500.0000
BogoMIPS: 5391.93
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization: AMD-V
L1d cache: 2 MiB (64 instances)
L1i cache: 2 MiB (64 instances)
L2 cache: 64 MiB (64 instances)
L3 cache: 256 MiB (8 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Python Version: 3.12.3
Python Platform: Linux-5.15.0-134-generic-x86_64-with-glibc2.39
Python Packages:
ai-dynamo: 0.2.1
ai-dynamo-runtime: 0.2.1
ai-dynamo-vllm: 0.8.4.post1
genai-perf: 0.0.12
nixl: 0.2.1
numpy: 1.26.4
nvidia-cublas-cu12: 12.4.5.8
nvidia-cuda-cupti-cu12: 12.4.127
nvidia-cuda-nvrtc-cu12: 12.4.127
nvidia-cuda-runtime-cu12: 12.4.127
nvidia-cudnn-cu12: 9.1.0.70
nvidia-cufft-cu12: 11.2.1.3
nvidia-curand-cu12: 10.3.5.147
nvidia-cusolver-cu12: 11.6.1.9
nvidia-cusparse-cu12: 12.3.1.170
nvidia-ml-py: 12.575.51
nvidia-nccl-cu12: 2.21.5
nvidia-nvjitlink-cu12: 12.4.127
nvidia-nvtx-cu12: 12.4.127
pyzmq: 26.4.0
tensorrt_llm: Not installed
torch: 2.6.0
torchaudio: 2.6.0
transformers: 4.51.3
tritonclient: Not installed
Additional Context
No response
Screenshots
No response
Describe the Bug
When attempting to launch dynamo serve using the vLLM backend on a machine with 2×H100 GPUs, CUDA 12.8, and NVIDIA driver version 570.133.20, the system fails at runtime with the following error:
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combinationThe error occurs during GPU initialization by the VllmWorker component, even though nvidia-smi confirms that CUDA and the H100 GPUs are visible both on the host and inside the container. As a result, the model engine fails to start and inference cannot proceed.
Additionally, the Planner component later fails due to a missing dynamo.json state file under /root/.dynamo/state.
This seems to be a runtime compatibility issue specific to this driver/CUDA combination or Docker setup.
Steps to Reproduce
1. Create a compatible Python environment
The python3 -m venv method installs Python 3.13.2 by default on Ubuntu 24.04, which is not supported by ai-dynamo-runtime. To avoid dependency resolution issues, I used Conda to create a Python 3.12 environment, which is supported by ai-dynamo-runtime:
conda create --name dynamo-test2 python=3.12conda activate dynamo-test22. Install the ai-dynamo[all] package
pip install "ai-dynamo[all]"Running a model
3. Fix shared library loading issue
When attempting to run a model with dynamo run (more specifically :
dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B) I initially hit a libpython3.12.so.1.0 error. To resolve it, I updated LD_LIBRARY_PATH:export LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"Then I confirmed model loading works with:
dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B. I was able to chat with the model.Serving a model
I then wanted to serve the model specifically using
--framework vllmso I followed the official instructions in https://github.com/ai-dynamo/dynamo/blob/main/examples/llm/README.md4. Start infrastructure services
cd dynamodocker compose -f deploy/metrics/docker-compose.yml up -d5. Launch the containerized vLLM runtime
According to the docs, the next step is to run the container:
./container/run.sh -it --framework vllm6. Attempt to serve the model
Navigate to the LLM example and run:
cd $DYNAMO_HOME/examples/llm dynamo serve graphs.agg:Frontend -f ./configs/agg.yamlI will attach the full output when running
dynamo serve graphs.agg:Frontend -f ./configs/agg.yamlto the in Actual Behavior Section in case it is useful:7. Confirm GPUs are visible
I ran nvidia-smi inside the container immediately after the failure:
nvidia-smi✅ Both H100 GPUs are detected and idle.
✅ CUDA 12.8 and Driver 570.133.20 appear to be installed correctly.
Expected Behavior
Expected Behavior
I expected the model to be successfully served using
dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml, with vLLM initializing the GPU backend properly and no CUDA-related errors. Specifically:VllmWorkercomponent should initialize without failure.localhost:8000/chat/completions).Actual Behavior
Actual Behavior
When attempting to serve the model using
dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml, the following occurred:The container successfully started, and all components (
Frontend,Planner,Processor,VllmWorker) began initialization.However, during
VllmWorkerinitialization, the following error occurred:RuntimeError: Unexpected error from cudaGetDeviceCount(): Error 803: system has unsupported display driver / cuda driver combinationThis caused the engine process to fail, and consequently:
VllmWorkercrashed.FileNotFoundError: State file not found: /root/.dynamo/state/dynamo.jsonNotably,
nvidia-smiinside the container does detect the H100 GPUs correctly and shows no running processes, so CUDA visibility itself appears to be working — but initialization fails.This blocks any ability to serve inference workloads using vLLM through Dynamo.
I have attached a file with the entire output of
dynamo serve graphs.agg:Frontend -f ./configs/agg.yamlhere.root@_MY_MACHINE___workspace_examples_llm# dynamo serve graphs.txt
Environment
From nvidia-smi
GPU Model: 2 × NVIDIA H100 NVL
GPU Memory: 95,830 MiB each (0 MiB in use at error time)
Driver Version: 570.133.20
CUDA Version (reported by nvidia-smi): 12.8
From dynamo env:
System Information:
OS: Ubuntu 24.04
Glibc Version: 2.39
GCC Version: 13.3.0
Cmake Version: 3.28.3
Rust Version: 1.86.0
Docker Version: N/A
CPU Information:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Vendor ID: AuthenticAMD
Model name: AMD EPYC 9334 32-Core Processor
CPU family: 25
Model: 17
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
Stepping: 1
Frequency boost: enabled
CPU(s) scaling MHz: 60%
CPU max MHz: 3910.2529
CPU min MHz: 1500.0000
BogoMIPS: 5391.93
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization: AMD-V
L1d cache: 2 MiB (64 instances)
L1i cache: 2 MiB (64 instances)
L2 cache: 64 MiB (64 instances)
L3 cache: 256 MiB (8 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Python Version: 3.12.3
Python Platform: Linux-5.15.0-134-generic-x86_64-with-glibc2.39
Python Packages:
ai-dynamo: 0.2.1
ai-dynamo-runtime: 0.2.1
ai-dynamo-vllm: 0.8.4.post1
genai-perf: 0.0.12
nixl: 0.2.1
numpy: 1.26.4
nvidia-cublas-cu12: 12.4.5.8
nvidia-cuda-cupti-cu12: 12.4.127
nvidia-cuda-nvrtc-cu12: 12.4.127
nvidia-cuda-runtime-cu12: 12.4.127
nvidia-cudnn-cu12: 9.1.0.70
nvidia-cufft-cu12: 11.2.1.3
nvidia-curand-cu12: 10.3.5.147
nvidia-cusolver-cu12: 11.6.1.9
nvidia-cusparse-cu12: 12.3.1.170
nvidia-ml-py: 12.575.51
nvidia-nccl-cu12: 2.21.5
nvidia-nvjitlink-cu12: 12.4.127
nvidia-nvtx-cu12: 12.4.127
pyzmq: 26.4.0
tensorrt_llm: Not installed
torch: 2.6.0
torchaudio: 2.6.0
transformers: 4.51.3
tritonclient: Not installed
Additional Context
No response
Screenshots
No response