How to get the hardware capability?

[idruker-cerence]

Describe the issue

onnxruntime can be built with a number of Execution Providers. Let's take CUDA as example. At runtime CUDA is appended as EP to the session options. But if the hardware does not have CUDA and cuDNN drivers installed, Ort::Session() will fail. Is there an elegant way to check the hardware capability before adding an EP and deal with error handling?

To reproduce

That's a query

Urgency

No response

Platform

Linux

OS Version

20.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.18

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

No response

[tianleiwu]

If you created a session with CUDA provider but CUDA is not available, it will fall back to CPU by default. Then you can use session.get_providers() to check whether CUDA provider is available for current session.

>>> import onnxruntime
>>> onnxruntime.get_available_providers()
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']
>>> ort_session = onnxruntime.InferenceSession("model.onnx", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
2024-07-19 17:43:44.2260019 [E:onnxruntime:Default, provider_bridge_ort.cc:1972 onnxruntime::TryGetProviderInfo_CUDA] D:\onnxruntime\onnxruntime\core\session\provider_bridge_ort.cc:1636 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\Users\.conda\envs\py310\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll"

2024-07-19 17:43:44.2312351 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:970 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*, and the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.
>>> ort_session.get_providers()
['CPUExecutionProvider']

[idruker-cerence]

Thank you Tianlei

Let me refine the question. I am trying to do it in C++. In C++ there is no way to explicitly add CPU execution provider. I assume that it's added implicitly and onnxruntime would fall back to CPU EP once fails to activate any accelerator like CUDA. In reality, as per my observation, the falling back does not happen.

[RyanUnderhill]

Yes, for C++ you'll have to catch the error and try not adding the cuda provider. The CPU provider is always implicitly added at the end of the list internally, you can't add it directly.

The list of available providers isn't the ones that will actually work, it's just the ones that onnxruntime was built with. Sadly the only way to detect if cuda will work is to try using it.

[idruker-cerence]

Thank you Ryan.

It seems like CUDA EP is a special case. Because when I tried QNN EP, the mechanism of falling back to CPU seemed to work. Could you confirm that?

[tianleiwu]

It seems like CUDA EP is a special case. Because when I tried QNN EP, the mechanism of falling back to CPU seemed to work. Could you confirm that?

In Linux, you can disable the fallback of QNN if you do not want the session to run in CPU EP. If you do not disable it, some node might fallback to CPU if QNN does not support the operator.

See example:

onnxruntime/onnxruntime/test/providers/qnn/qnn_basic_test.cc

Lines 40 to 45 in b14b4ec

	// Can only enforce that model runs on QNN in linux CI machines
	// because they support the CPU backend and emulate the HPT backend.
	// TODO: Remove #ifdef when Windows Arm64 machines support the CPU backend.
	#if defined(__linux__)
	so.AddConfigEntry(kOrtSessionOptionsDisableCPUEPFallback, "1"); // Disable fallback to the CPU EP.
	#endif

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.