Skip to content

Add NPU condition for "cmake/onnxruntime_providers_openvino.cmake" #24557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

mc-nv
Copy link
Contributor

@mc-nv mc-nv commented Apr 25, 2025

Description

Make NPU flag optional in cmake/onnxruntime_providers_openvino.cmake

Motivation and Context

Motivation for this issue is to make NPU flag optional
#22985

@snnn
Copy link
Member

snnn commented Apr 26, 2025

In https://github.com/microsoft/onnxruntime/blob/main/cmake/onnxruntime_providers.cmake we have

if (onnxruntime_USE_OPENVINO)
  include(onnxruntime_providers_openvino.cmake)
endif()

So how does it affect you?

@jywu-msft
Copy link
Member

@preetha-intel @sfatimar does this make sense?

@@ -17,7 +17,7 @@
message(FATAL_ERROR "OpenVINO 2024.5 and newer are supported. Please, use latest OpenVINO release")
endif()

if(OpenVINO_VERSION VERSION_GREATER_EQUAL 2024.4)
if(OpenVINO_VERSION VERSION_GREATER_EQUAL 2024.4 AND onnxruntime_USE_OPENVINO_NPU)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onnxruntime_USE_OPENVINO_NPU is only enabled when we build ORT with OpenVINO and default device type as NPU which is not always the case. Even when we build with CPU we should be able to use NPU and NPU Device memory as a runtime option

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mc-nv, I think what @sfatimar is saying is that -DUSE_OVEP_NPU_MEMORY=1 should always be set, whether or not the system has an NPU. onnxruntime_USE_OPENVINO_NPU just builds with NPU as default device type, but we want to allow NPU for even CPU default device type.
The issue you previously encountered should have been fixed. you shouldn't be seeing any runtime errors on systems that don't have an NPU. Are you seeing differently?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the explanation.
Let me move the PR in draft till I finish my internal testing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mc-nv The NPU allocator will be used only when NPU HW is selected. Refer here. The flag is basically set to ensure that from the specified version of OV the NPU allocators are available.

@jywu-msft
Copy link
Member

In https://github.com/microsoft/onnxruntime/blob/main/cmake/onnxruntime_providers.cmake we have

if (onnxruntime_USE_OPENVINO)
  include(onnxruntime_providers_openvino.cmake)
endif()

So how does it affect you?

I think they enable multiple EP's for triton inference server onnxruntime backend including OpenVINO EP.
https://github.com/triton-inference-server/onnxruntime_backend/blob/f1fec69b8668a59d923719a4da6560113a5f0584/CMakeLists.txt#L92

@mc-nv mc-nv marked this pull request as draft April 28, 2025 15:39
@mc-nv mc-nv closed this May 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants