Cuda execution provider is not available

[thewh1teagle]

Cuda execution provider is not available

root@ubuntu:~# mkdir proj
root@ubuntu:~# cd proj
root@ubuntu:~/proj# uv init
Initialized project `proj`
root@ubuntu:~/proj# uv add onnxruntime-gpu onnxruntime
Using CPython 3.12.9
Creating virtual environment at: .venv
Resolved 12 packages in 293ms
Prepared 4 packages in 2.97s
Installed 10 packages in 34ms
 + coloredlogs==15.0.1
 + flatbuffers==25.2.10
 + humanfriendly==10.0
 + mpmath==1.3.0
 + numpy==2.2.3
 + onnxruntime==1.20.1
 + onnxruntime-gpu==1.20.1
 + packaging==24.2
 + protobuf==5.29.3
 + sympy==1.13.3
root@ubuntu:~/proj# uv run python3
Python 3.12.9 (main, Feb 12 2025, 14:50:50) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime
>>> onnxruntime.get_available_providers()
['AzureExecutionProvider', 'CPUExecutionProvider']
>>> 
root@ubuntu:~/proj# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.5 LTS"
root@ubuntu:~/proj# lspci | grep -i nvidia
00:07.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
00:08.0 Audio device: NVIDIA Corporation Device 22ba (rev a1)
root@ubuntu:~/proj# nvidia-smi
Thu Feb 27 05:44:54 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:00:07.0 Off |                  Off |
| 31%   39C    P0              78W / 480W |      0MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
root@ubuntu:~/proj# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

Also, I would add warning log if onnxruntime-gpu installed but there's no any GPU found!
Maybe even GPU vendor detection with instructions for the most common distro (eg. Ubuntu) for how to fix the issue.

[snnn]

Can you try our nightly version?

[snnn]

Recently we added a feature that you can fetch CUDA/CUDNN libs from pip when installing onnxruntime. See #23659 .
It reduces the chance of CUDA version mismatch.

[thewh1teagle]

Recently we added a feature that you can fetch CUDA/CUDNN libs from pip when installing onnxruntime. See #23659 . It reduces the chance of CUDA version mismatch.

Still not working

root@ubuntu:~/kokoro-onnx# sudo apt install nvidia-cudnn nvidia-cuda-toolkit
root@ubuntu:~/kokoro-onnx# uv pip install --pre --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-gpu

Resolved 9 packages in 1m 13s
⠸ Preparing packages... (0/1)
onnxruntime-gpu ------------------------------ 57.56 MiB/267.80 MiB        
root@ubuntu:~/kokoro-onnx# find / -name "libcublas*.so"
/usr/lib/x86_64-linux-gnu/libcublasLt.so
/usr/lib/x86_64-linux-gnu/stubs/libcublasLt.so
/usr/lib/x86_64-linux-gnu/stubs/libcublas.so
/usr/lib/x86_64-linux-gnu/libcublas.so
root@ubuntu:~/kokoro-onnx# LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH" LOG_LEVEL=DEBUG uv run examples/with_session.py 
Available onnx runtime providers: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
Setting threads to CPU cores count: 30
2025-02-27 16:45:02.883638075 [E:onnxruntime:Default, provider_bridge_ort.cc:2022 TryGetProviderInfo_TensorRT] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1695 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_tensorrt.so with error: libcublas.so.12: cannot open shared object file: No such file or directory

*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/python/onnxruntime_pybind_state.cc:505 void onnxruntime::python::RegisterTensorRTPluginsAsCustomOps(PySessionOptions&, const onnxruntime::ProviderOptions&) Please install TensorRT libraries as mentioned in the GPU requirements page, make sure they're in the PATH or LD_LIBRARY_PATH, and that your GPU is supported.
 when using ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CUDAExecutionProvider', 'CPUExecutionProvider'] and retrying.
****************************************
2025-02-27 16:45:02.994227831 [E:onnxruntime:Default, provider_bridge_ort.cc:2036 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1695 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory

2025-02-27 16:45:02.994251395 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:994 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*. Please install all dependencies as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.
DEBUG    [__init__.py:169] Creating audio for 1 batches for 49 phonemes
DEBUG    [__init__.py:76] Phonemes: həlˈoʊ. ðɪs ˈɔːdɪˌoʊ dʒˈɛnɚɹˌeɪɾᵻd baɪ kəkˈɔːɹoʊ!
DEBUG    [__init__.py:100] Created audio in length of 3.70s for 49 phonemes in 0.81s (RTF: 0.22
DEBUG    [__init__.py:180] Created audio in 0.82s
Created audio.wav

[snnn]

It was because:

Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory

[thewh1teagle]

It was because:

Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory

I have no idea how to fix it, the docs of onnxruntime are too general.
Debian / Ubuntu is the most popular distro. why can't onnxruntime add example how to setup it with cuda?