Running inference with OpenVINOExecutionProvider fails if it was executed with CUDAExecutionProvider

[szobov]

Describe the issue

Hello, dear maintainers,
I faced the following:
I have a setup running onnx model with CUDAExecutor provider.
I changed the code and switched to OpenVINOExecutionProvider and got such an error after running:

ERROR - [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /home/onnxruntimedev/onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:53 onnxruntime::openvino_ep::OVExeNetwork onnxruntime::openvino_ep::OVCore::LoadNetwork(const string&, std::string&, ov::AnyMap&, std::string) [OpenVINO-EP]  Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_3_0Check 'false' failed at src/inference/src/core.cpp:149:
Check 'false' failed at src/frontends/onnx/frontend/src/core/graph_cache.cpp:25:
output/pos:0 node not found in graph cache

Traceback (most recent call last):
  File <stripped>/onnxwrapper.py", line 46, in __load_session
    session = onnxruntime.InferenceSession(
  File "/opt/BDK/python/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 388, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "<stripped>/python/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 440, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /home/onnxruntimedev/onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:53 onnxruntime::openvino_ep::OVExeNetwork onnxruntime::openvino_ep::OVCore::LoadNetwork(const string&, std::string&, ov::AnyMap&, std::string) [OpenVINO-EP]  Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_3_0Check 'false' failed at src/inference/src/core.cpp:149:
Check 'false' failed at src/frontends/onnx/frontend/src/core/graph_cache.cpp:25:
output/pos:0 node not found in graph cache

It works without issues if I copy and run the same model to a new location.

The only working around I found is passing provider_options options to the InferenceSession with the following data:
provider_options={"cache_dir": "/home/user/.cache/.onnx_cache/",} and creating this directory. Then, it works without any issues.

The versions I use are the following:

• Python 3.10.13 & onnxruntime-openvino 1.15.0
• Python 3.6 & onnxruntime-gpu 1.4.0
• NVIDIA-SMI 418.39
• Driver Version: 418.39
• CUDA Version: 10.1

Please ask me if you need any more information.
Thanks for your time and help.

To reproduce

Create an onnx-model.
Run the model with the InferenceSession with CUDAExecutionProvider.
Run the same model with the InferenceSession with OpenVINOExecutionProvider.

Urgency

I found a workaround, so I'm not blocked.

Platform

Linux

OS Version

Ubuntu 16.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.15.0

ONNX Runtime API

Python

Architecture

X86

Execution Provider

OpenVINO

Execution Provider Library Version

10.1

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[github-actions]

This issue has been automatically closed due to inactivity. Please reactivate if further support is needed.