[Performance] Openvino 2x slower than with OpenCV on an Intel HD Graphics 620 / 630

[JulienMaille]

Describe the issue

This is a crosspost from microsoft#25266
I hope it is no against the rules, but since this is an Intel-centric question (using OpenVino provider) I figured I could get more support here.

I'm in the middle of a migration from OpenCV inference engine to ONNXRuntime.
I have run a couple benchmarks and systematically get similar or better inference time with ORT. The single exception is on an old Surface laptop that is used for demos.
Openvino GPU backend when using OpenCV is twice faster than Openvino GPU provider using ORT (and also twice faster than DML provider using ORT).

		256²	512²	1024²
HD Graphics 620	ORT_Openvino	99ms	368ms	1510ms
	ORT_DML	99ms	353ms	1562ms
	OpenCV_OpenVino	42ms	165ms	814ms
HD Graphics 630	ORT_Openvino			1202ms
	ORT_DML			1434ms
	OpenCV_OpenVino			655ms

• the model is a CNN for computer vision (Unet-Resnet)
• both ORT (1.22) and OpenCV (4.12) were compiled in x64 with OpenVino 2025.2.0 (same results with 2025.0.0 and 2024.6.0)
• the model is trained on 256x256 images but benchmarks run on 1024x1024 images, hence I can't use the same onnx file for both benchmarks. OpenCV can work with a "normal" model with fixed 256² input but ORT requires the onnx to be exported with dynamic input/output layers
• I've played with almost all ORT/OpenVino provider options without success.
• this machine is running Windows 10
• Intel GPU drivers updated to the latest version: 31.0.101.2135

What could I try to bring ORT at the level of OpenCV inference in this specific context?

To reproduce

Windows version: 10.0 (build 19045)
CPU: Intel(R) Core(TM) i5-7300U CPU @ 2.60GHz
OpenCV version: 4.12.0-dev
ONNX Runtime version: 1.22.0
OpenVINO Intel GPU Adapter 0: Intel(R) HD Graphics 620
DML Adapter 0: Intel(R) HD Graphics 620
logical cores: 4

 ====== Generating image [1024,1024] ======
 == Running on ONNX_OpenVINOExecutionProviderAUTO:GPU,CPU ==
Loading model..done in 56ms
Warm-up inference done in 4817ms
        20 inferences..done in avg: 1802.3, min: 1281, median: 1536, std: 890.87
 == Running on ONNX_OpenVINOExecutionProviderHETERO:GPU,CPU ==
Loading model..done in 91ms
Warm-up inference done in 7730ms
        20 inferences..done in avg: 1546.3, min: 1502, median: 1533, std: 55.26
 == Running on ONNX_OpenVINOExecutionProviderGPU_0 ==
Loading model..done in 79ms
Warm-up inference done in 7883ms
        20 inferences..done in avg: 1521.3, min: 1504, median: 1510, std: 19.28
 == Running on ONNX_DmlExecutionProvider0 ==
Loading model..done in 411ms
Warm-up inference done in 2957ms
        20 inferences..done in avg: 1573.6, min: 1553, median: 1562, std: 35.98
 == Running on BACKEND_OPENVINO-TARGET_OPENCL ==
1 detected device(s)
name: Intel(R) HD Graphics 620
Loading model..done in 40ms
Warm-up inference done in 3369ms
        20 inferences..done in avg: 812.8, min: 740, median: 814, std: 18.25

Urgency

No response

Platform

Windows

OS Version

10.0 (build 19045)

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.22

ONNX Runtime API

C++

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

2025.2.0

Model File

Original model working with OpenCV dnn
resnet18.zip
Modified model with dynamic input for ONNXRuntime
resnet18_fixed_dyn.zip

Is this a quantized model?

No