Enabling CUDA execution provider

[GeorgeGaziev]

Hi, thanks for such a nice crate for the onnxruntime!

I'm trying to do inference with it and struggling with enabling CUDA execution provider. Here is how I set it up in the cargo.toml:
ort = {version = "=1.15.2", features = ["cuda", "tensorrt"]}, and no crate-specific system variables set. I also have a CUDA v12.1 installed. I've seen the question #14, but I suppose I have a different issue.

When I run the project by PowerShell $env:RUST_LOG = 'ort=debug'; cargo run I get following warnings:

2023-08-21T18:50:24.472279Z  WARN apply_execution_providers: ort::execution_providers: An error occurred when attempting to register `CUDAExecutionProvider`: D:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1131 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\git\onnxruntime-test\target\debug\onnxruntime_providers_cuda.dll"

2023-08-21T18:50:24.472394Z  WARN apply_execution_providers: ort::execution_providers: No execution providers registered successfully. Falling back to CPU.

My questions is, how can I use the cuda in this case?

Thanks

[decahedron1]

Hmm, it's hard to tell the exact error here. Do you also have cuDNN installed and added bin to the PATH? (or copy the bin files to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin).

[decahedron1]

Yes, cuDNN is separate from the CUDA Toolkit.
https://developer.nvidia.com/rdp/cudnn-download

[GeorgeGaziev]

Thanks! I'll try it and report back!

[GeorgeGaziev]

I've downloaded the cuDNN, and copied content from bin, lib and include folders to the respective folders in the C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\, did the cargo clean and getting the same warnings

[decahedron1]

Hmm, I've also seen this happen with a missing zlibwapi.dll (at least for CUDA 11). It seems NVIDIA removed the download from their docs so I don't know if CUDA 12 depends on it anymore, but you could try extracting dll_x64/zlibwapi.dll from http://www.winimage.com/zLibDll/zlib123dllx64.zip to the CUDA bin dir.

Additionally, ONNX Runtime 1.15.1 doesn't necessarily support CUDA 12 which could be causing this error, though some report it working, so I'm not sure. If nothing else works it might not hurt to uninstall CUDA 12 and try CUDA 11.8 (though note you'll also need to re-download cuDNN as there are separate versions for CUDA 11.x and 12.x)

[GeorgeGaziev]

Thanks, I'll try those options and will come back!

[GeorgeGaziev]

I've switched to CUDA 11.8, added zlibwapi.dll CUDA bin folder, and installed cuDNN for CUDA 11.8, and it successfully registered the CUDA execution provider!

Adding zlibwapi.dll and using CUDA 12 with ONNX Runtime 1.15.1 didn't work for me.

Thanks for the help!