Cant get GPU to work with ONNX Runtime 1.19 Cuda 12.6 CuDNN 9 RTXA4000

[sismith999]

Describe the issue

providers = ['CUDAExecutionProvider'] # Specify the GPU provider
session = ort.InferenceSession(model_path, providers=providers) # Create the ONNX Runtime InferenceSession with GPU

Reports error

['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
2024-08-22 13:01:04.2808367 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "c:\Simon\Timpi\test\myenv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

2024-08-22 13:01:04.2871671 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:965 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*, and the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.
ONNX Runtime is using the CPU (CPUExecutionProvider).

To reproduce

providers = ['CUDAExecutionProvider'] # Specify the GPU provider
session = ort.InferenceSession(model_path, providers=providers) # Create the ONNX Runtime InferenceSession with GPU

DUMPBIN on DLL

cublasLt64_12.dll          Directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin
cublas64_12.dll
cudnn64_9.dll
cufft64_11.dll
cudart64_12.dll
onnxruntime_providers_shared.dll
KERNEL32.dll
MSVCP140.dll
VCRUNTIME140.dll
VCRUNTIME140_1.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll

cublasLt64_12.dll:  Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin\cublasLt64_12.dll
cublas64_12.dll:    Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin\cublas64_12.dll
cudnn64_9.dll:      Found at C:\Program Files\NVIDIA\CUDNN\v9.3\bin\11.8\cudnn64_9.dll
cufft64_11.dll:     Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin\cufft64_11.dll
cudart64_12.dll:    Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin\cudart64_12.dll
onnxruntime_providers_shared.dll:   Not found
KERNEL32.dll: Not found
MSVCP140.dll:       Found at C:\Program Files (x86)\NortonInstaller\{0C55C096-0F1D-4F28-AAA2-85EF591126E7}\NGC\A5E82D02\22.23.1.21\MSVCP140.dll
VCRUNTIME140.dll:   Found at C:\Program Files (x86)\NortonInstaller\{0C55C096-0F1D-4F28-AAA2-85EF591126E7}\NGC\A5E82D02\22.23.1.21\VCRUNTIME140.dll
VCRUNTIME140_1.dll: Not found
api-ms-win-crt-heap-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-string-l1-1-0.dll

Urgency

Urgent

Platform

Windows

OS Version

11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

11.9

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

cuda 12.6

[tianleiwu]

Two are suspicious:

• C:\Program Files\NVIDIA\CUDNN\v9.3\bin\11.8\cudnn64_9.dll (Does 11.8 means it is cudnn for cuda 11.8)?
• VCRUNTIME140_1.dll: Not found

Please install latest MSVC runtime: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

Also install cudnn 9.3 for cuda 12 like https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/windows-x86_64/cudnn-windows-x86_64-9.3.0.75_cuda12-archive.zip. You can unzip the folder, and add the lib directory to PATH.

[sismith999]

THanks for your reply, i checked the items you mentioned and removed some old directories but i still cant get it to work... I downgraded to CUDA 12.4 and DNN 3.9  I tested Cuda with Pytorch below:- import torch # import sys# import os# # Paths to CUDA DLLs# cuda_bin_path = r"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin"# cuda_libnvvp_path = r"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\libnvvp"# cuda_dnn = r"C:\Program Files\NVIDIA\CUDNN\v9.3\bin\12.6"    ## r"C:\Program Files\NVIDIA\CUDNN\v9.3\bin"# venv = r"C:\Simon\Timpi\test\myenv\Lib\site-packages\onnxruntime\capi";# # Add them to sys.path# sys.path.append(cuda_bin_path)# sys.path.append(cuda_libnvvp_path)# sys.path.append(cuda_dnn)# sys.path.append(venv)# # Optional: Also set the PATH environment variable (for subprocesses)# os.environ["PATH"] = cuda_bin_path + os.pathsep + cuda_libnvvp_path + os.pathsep + os.environ["PATH"] + os.pathsep + cuda_dnn + os.pathsep + venv def test_cuda_and_cudnn():    # Check if CUDA is available    print(torch.version.cuda)    print(torch.cuda.get_device_name(0))     if torch.cuda.is_available():        print("CUDA is available.")                # Get the CUDA version        print("CUDA version:", torch.version.cuda)                # Get the cuDNN version        print("cuDNN version:", torch.backends.cudnn.version())                # Create a tensor on GPU        x = torch.randn(3, 3).cuda()        print("Tensor on GPU:", x)                # Perform a simple operation        y = x @ x.T        print("Result of a simple matrix multiplication on GPU:", y)    else:        print("CUDA is not available. Please check your installation.") if __name__ == "__main__":    test_cuda_and_cudnn() and this works...  NVIDIA RTX A4000CUDA is available.CUDA version: 12.4cuDNN version: 90100Tensor on GPU: tensor([[ 0.7406,  0.3850,  0.5371],        [-1.2015,  0.0742,  0.4274],        [ 0.6611,  0.1410, -1.6860]], device='cuda:0')Result of a simple matrix multiplication on GPU: tensor([[ 0.9851, -0.6317, -0.3616],        [-0.6317,  1.6317, -1.5044],        [-0.3616, -1.5044,  3.2995]], device='cuda:0') NOTE my dlls are now  cublasLt64_12.dll: Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cublasLt64_12.dllcublas64_12.dll: Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cublas64_12.dllcudnn64_9.dll: Found at C:\Program Files\NVIDIA\CUDNN\v9.3\bin\11.8\cudnn64_9.dllcufft64_11.dll: Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cufft64_11.dllcudart64_12.dll: Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cudart64_12.dllonnxruntime_providers_shared.dll: Found at \Simon\Timpi\test\myenv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_shared.dllKERNEL32.dll: Found at C:\WINDOWS\System32\kernel32.dllMSVCP140.dll: Found at C:\Program Files\Blender Foundation\Blender 3.1\blender.crt\msvcp140.dllVCRUNTIME140.dll: Found at C:\Program Files\Azure Data Studio\tools\vcruntime140.dllVCRUNTIME140_1.dll: Found at C:\Program Files\Blender Foundation\Blender 3.1\blender.crt\vcruntime140_1.dllapi-ms-win-crt-heap-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-heap-l1-1-0.dllapi-ms-win-crt-math-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-math-l1-1-0.dllapi-ms-win-crt-runtime-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-runtime-l1-1-0.dllapi-ms-win-crt-stdio-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-stdio-l1-1-0.dllapi-ms-win-crt-string-l1-1-0.dll: Found at C:\Program Files\Azure Data Studio\resources\app\extensions\mssql\sqltoolsservice\Windows\4.0.1.1\api-ms-win-crt-string-l1-1-0.dll note onnxruntime-gpu is 1.18:-   Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/7982ae20-ed19-4a35-a362-a96ac99897b7/pypi/download/onnxruntime-gpu/1.18.1/onnxruntime_gpu-1.18.1-cp312-cp312-win_amd64.whl error still ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']2024-08-22 18:50:44.3526054 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 onnxruntime::TryGetProviderInfo_CUDA] D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "c:\Simon\Timpi\test\myenv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll" 2024-08-22 18:50:44.3590433 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:895 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirementsto ensure all dependencies are met. any advice is appreciated... thanks Simon. On Thursday 22 August 2024 at 13:47:13 GMT+7, Tianlei Wu ***@***.***> wrote: Two are suspicious: - C:\Program Files\NVIDIA\CUDNN\v9.3\bin\11.8\cudnn64_9.dll (Does 11.8 means it is cudnn for cuda 11.8)? - VCRUNTIME140_1.dll: Not found Please install latest MSVC runtime: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 Also install cudnn 9.3 for cuda 12 like https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/windows-x86_64/cudnn-windows-x86_64-9.3.0.75_cuda12-archive.zip. You can unzip the folder, and add the bin directory to PATH. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[tianleiwu]

Here is one way to setup a python environment for onnxruntime-gpu of CUDA 12 and python 3.12 in Windows:

(1) Install Ana Conda: https://docs.anaconda.com/anaconda/install/windows/
(2) Run the following to create a conda environment:

conda create -n py312 python=3.12
conda activate py312
pip install msvc-runtime==14.40.33807
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
pip install nvidia-cudnn-cu12==9.3.0.75 nvidia-cuda-runtime-cu12==12.5.82 nvidia-cufft-cu12==11.2.3.61
pip install onnxruntime-gpu==1.19.0

Then you can test your environment like the following:

import os
import ctypes
import subprocess
import sys

def print_loaded_libraries():
    import psutil
    pid = os.getpid()
    process = psutil.Process(pid)
    for dll in process.memory_maps():
        print(dll.path)

def install_package(package_name, version):
    print(f"Installing {package_name} {version}...")
    subprocess.check_call([sys.executable, "-m", "pip", "install", f"{package_name}=={version}"])
    print(f"{package_name} {version} has been installed successfully.")

# Optional for AnaConda. It is recommended to install  explicitly if you are not sure system has latest VC Runtime.
install_package("msvc-runtime", "14.40.33807")

install_package("nvidia-cudnn-cu12", "9.3.0.75")
install_package("nvidia-cuda-runtime-cu12", "12.5.82")
install_package("nvidia-cufft-cu12", "11.2.3.61")
install_package("onnxruntime-gpu", "1.19.0")

# psutil is used in print_loaded_libraries()
install_package("psutil", "6.0.0")

# Get the path to the Conda environment
conda_env = os.environ.get("CONDA_PREFIX")
if conda_env is None:
    raise EnvironmentError("Conda environment is not active.")

# import sysconfig
# site_packages_directory = sysconfig.get_path('purelib')
site_packages_directory = os.path.join(conda_env, 'Lib', 'site-packages')

cudnn_dll_directory =  os.path.join(site_packages_directory, 'nvidia', 'cudnn', 'bin')
cuda_runtime_directory = os.path.join(site_packages_directory, 'nvidia', 'cuda_runtime', 'bin')
cufft_dll_directory = os.path.join(site_packages_directory, 'nvidia', 'cufft', 'bin')
cublas_dll_directory = os.path.join(site_packages_directory, 'nvidia', 'cublas', 'bin')
ort_dll_directory = os.path.join(site_packages_directory, 'onnxruntime', 'capi')
msvc_dll_directory = conda_env

dll_to_directory = {
    "cublasLt64_12.dll" : cublas_dll_directory,
    "cublas64_12.dll" : cublas_dll_directory,
    "cudnn64_9.dll" : cudnn_dll_directory,
    "cufft64_11.dll" : cufft_dll_directory,
    "cudart64_12.dll" : cuda_runtime_directory,
    "MSVCP140.dll" : conda_env,
    "VCRUNTIME140.dll" : conda_env,
    "VCRUNTIME140_1.dll" : conda_env,
    "onnxruntime_providers_cuda.dll" : ort_dll_directory,
}

unique_directories = set(dll_to_directory.values())
for directory in unique_directories:
    os.add_dll_directory(directory)

# Preload could avoid some issues related to PATH setting.
enable_preload = True

preloaded_dlls = []
if enable_preload:
    for filename, directory in dll_to_directory.items():
        dll_full_path = os.path.join(directory, filename)
        try:
            dll = ctypes.CDLL(dll_full_path)
            preloaded_dlls.append(dll)
            print(f"Successfully loaded DLL: {dll_full_path}")
        except OSError as e:
            print(f"Failed to load cuDNN DLL: {dll_full_path}")
else:
    # The following can be used to test whether you have set PATH environment variable correctly.
    dependent_dlls = ["cublasLt64_12.dll", "cublas64_12.dll", "cudnn64_9.dll", "cufft64_11.dll", "cudart64_12.dll", "onnxruntime_providers_shared.dll", "MSVCP140.dll", "VCRUNTIME140.dll", "VCRUNTIME140_1.dll"]
    for dll_name in dependent_dlls:
        try:
            my_dll = ctypes.CDLL(dll_name)
            print(f"Successfully loaded the {dll_name}.")
        except OSError as e:
            print(f"Failed to load the {dll_name}: {e}")    

# Test import onnxruntime.
import onnxruntime

# Verify that the path of DLLs to make sure the right DLLs are loaded.
print_loaded_libraries()

[yuslepukhin]

This is needed: onnxruntime_providers_shared.dll: Not found

[jywu-mysoft]

This is needed: onnxruntime_providers_shared.dll: Not found

the latest output has onnxruntime_providers_shared.dll: Found at \Simon\Timpi\test\myenv\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_shared.dll
so i don't think that is the issue.

i think the version of cudnn is probably still the issue,

cudnn64_9.dll: Found at C:\Program Files\NVIDIA\CUDNN\v9.3\bin\11.8\cudnn64_9.dll

it's finding a version of cudnn9 that is for cuda 11.8 , not cuda 12.x

[sismith999]

Thanks to everyone who commented, you can close this now i did finally get it to work...
a few notes

A) my environment was complicated with many python versions installed so be careful with testing ( my python3 was going to systemwide dll vs python going to my venv/conda)

B)
HAD TO USE THESE VERSION COMBINATIONS... CUDA 12.4 AND CUDNN 3.9 OK IF PATH SEE BELOW:-

cuda_bin_path = r"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin" (INSTALLED FROM Nvidia installer..)

cuda_dnn = "D:\DOWN\CUDNN_help\cudnn-windows-x86_64-9.3.0.75_cuda12-archive\cudnn-windows-x86_64-9.3.0.75_cuda12-archive\bin"

note ACTUALLY NVIDIA 3.9 INSTALLER FOR CUDNN 3.9 DOES SUFFICE IF YOU ADD THIS PATH CAREFULLY, NOTE THE INSTALL SEEMS TO CREATE SUBFOLDERS FROM BIN \ 11.7 AND 12.6, SO YOUHAVE TO SPECIFY LIKE BELOW...
(NOTE 12.6 DID SEEM OK AS PER NVIDIA COMPAT MATRIX EVEN THOUGH MY CUDA WAS 12.4)
C:\Program Files\NVIDIA\CUDNN\v9.3\bin\12.6 NOT C:\Program Files\NVIDIA\CUDNN\v9.3\bin

[deepconvai]

A quick remark because this problem resurfaces many times in many systems.
If your base system nvidia driver is compatible with your cudnn library, then you can install nvidia stuff from pypi, eg.

pip install nvidia-cudnn-cu12

This installs the .so files on linux, but it doesn't load them. (I assume on windows it will install dll files)

from ctypes import CDLL
import nvidia.cudnn.lib as cudnn
from pathlib import Path
import onnxruntime as ort

cudnn_dir = str(Path(cudnn.__file__).parent)
for dll_file in cudnn_dir.glob("*.so.*"):          # replace it with DLL pattern on windows
    CDLL(dll_file)

providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
model_path = "..."
ort_session = ort.InferenceSession(model_path, providers=providers)

This works because python has a caching mechanism, so if it once found the .so file, then next time it will use that. So even though the CDLL returns some DLL-like objects, you can discard them, we only use it for putting names into the lookup cache mechanism, and you don't need to add LD_LIBRARY_PATH or similar lookup path changes.

It's an ugly hack, but you need to add 6 lines at the beginning of your python code, and 1 dependency to the python package (pyproject.toml), and it should be good to go.
Why don't we load a single .so, like libcudnn.so.9? Depending on the onnxruntime, they might need other libs too, like cublas, etc. , and it's a generic pattern, so moving later version, like future cuda 13, you only need to update the library dependency.

This issue in various forms appear in many cases, like in #21684 or #22198.

[galbiati]

This works because python has a caching mechanism, so if it once found the .so file, then next time it will use that.

Thanks - I ran into #21684, and this helped me understand what importing torch had to do with the problem. I didn't look into how torch does it, but import torch does result in the libraries being cached. (The dllist package was also helpful for confirming that this was the relevant difference).

I went looking for how libraries are loaded in onnxruntime, and was surprised to discover there's actually already a function for this situation. While that function is in ORT's documentation, the documentation doesn't quite clarify why you might need this, as opposed to just using it to configure library location more explicitly.

So what I ended up with is

import onnxruntime as ort

ort.preload_dlls()

which solved the issue in my environment. (Ubuntu, onnxruntime-gpu[cuda,cudnn]==1.22.0)

[deepconvai]

So what I ended up with is

import onnxruntime as ort
ort.preload_dlls()
which solved the issue in my environment. (Ubuntu, onnxruntime-gpu[cuda,cudnn]==1.22.0)

Thanks! That certainly looks cleaner than my original hack, I should have found this function.
As far as I understood, it should work both with cudnn being installed or pytorch being installed (and bringing cudnn along)

As far as I understood, regardless of torch import, calling ort.preload_dlls() seems to be safe.
I don't quite understand why it is not called automatically when the package is imported...

I see that the same or a closely related issue appears in a few other threads too: #24719, #24266, although there seem to be partial importing issues too. So if someone gets here but doesn't find a solution, maybe continue on these tickets.

[galbiati]

FWIW, I don't see preload_dlls released until 1.21.0, which was March this year. It might not have been around for you to find!

I don't quite understand why it is not called automatically when the package is imported...

I'm a bit puzzled by this too.

Maybe there's an intention for LD_LIBRARY_PATH to be the default approach, unless the Python packages are explicitly requested? But from that perspective, the interaction with torch probably shouldn't be seen as a minor convenience the way the docstring seems to frame it. (Since import torch will implicitly violate the assumption that LD_LIBRARY_PATH is the default for onnxruntime.)

If "work seamlessly with pytorch" is a goal, then I guess maybe you do want this behavior, because otherwise there's a "seam"? (But I'm not sure why I'd want this. I doubt handing data directly from torch to ORT is all that common a need, but I've been surprised before 🤷)

So maybe this just got caught in conflicting or unclear design goals, and ended up in an awkward limbo.