[OVEP] OpenVINO EP 1.26.0 Development Release Updates

[ankitm3k]

Summary

Periodic upstream sync of Intel's OVEP branch (ovep_1_26_release) into ORT main. All changes are scoped to the OpenVINO EP and its tests.

OpenVINO 2026.0 / 2026.1 support

• Add V_2026_0 / V_2026_1 version enums; capability.cc default bumped to V_2026_1.
• Register FLOAT8E4M3FN / FLOAT8E5M2 initializer types on CPU / GPU / NPU.
• Disable OVEP-level QDQ-stripping on OV ≥ 2026.1 (OV handles it internally).
• Add ReduceSum to no-dimension-supported ops.

KV-cache / stateful CausalLM

• Rename ReorderKVCache → SetReorderKVCacheStatus across backend interfaces.
• Populate src_idx / dst_idx in PreProcessInferRequest with shape validation; clean state after inference and on RewindKVCache.
• FuseCacheReorder: beam_idx and src_idx/dst_idx paths are now mutually exclusive; reject models that already carry reorder inputs.
• Behavior change: RewindKVCache(index > 0) now throws when reorder is enabled (physical KV-cache eviction pass is a TODO).

NPU / provider options

• Force disable_dynamic_shapes=true on NPU unless enable_causallm is set.
• Preserve user-supplied NPU_COMPILATION_MODE_PARAMS; skip it when importing precompiled blobs.
• Preserve factory-level device_type when session options don't override it (fixes NPU mis-selection from Python).
• Behavior change: removed the ORT_OPENVINO_NPU_COMPILER_TYPE env override — OV's default NPU compiler is used now.

External initializers / weight sharing

• Drop the 32 MB embed threshold — always externalize when multiple external initializers are in memory.
• DumpOpenVINOEPModel rebuilds a self-contained proto when initializer data was stripped.
• AddExternalWeight validates re-adds against existing offset/size/location (parity with ABI EP); fix race in device-tensor mapping.
• ov_bin_manager: bounds-checked pointer view over mapped weights (fixes read-only blob import).
• qdq_stripping: use std::from_chars so offsets/lengths > 4 GB parse correctly.

Perf-count dump

• New ORT_OPENVINO_PERF_COUNT=<dir> env var writes per-subgraph CSV (Layer Name,Status,Layer Type,Real Time (us),Exec Type), replacing the old stdout-only debug dump. Requires ov::enable_profiling on the compiled model; logs a warning and no-ops otherwise.

Misc

• API: IBackend::Infer is no longer const (needed for perf-dump bookkeeping).
• Filter orphaned graph outputs from OVEP sub-graphs.
• Better error message for "cannot export dynamically compiled model" (points to reshape_input).
• Human-readable ovep_exception::type strings.
• ov::shutdown() on DLL unload.

Tests

• Add OVEP_ExtInit_DynamicEmbed_Tests and OVEP_ExtInit_EmptyRawData_Tests; refactor setup into SetUpTestSuite.
• Narrow OVEP exclusions in embed_layer_norm, fused_matmul, matmul_4bits, quantize_linear (skip only unsupported sub-cases).
• perftest: reset outputs per run to support data-dependent output shapes (e.g. NonZero).

Testing

Validated against the OpenVINO versions this release targets (2025.3 – 2026.1) on CPU / GPU / NPU:

• New OVEP tests pass: OVEP_ExtInit_Tests, OVEP_ExtInit_DynamicEmbed_Tests, OVEP_ExtInit_EmptyRawData_Tests
• Narrowed contrib-op exclusions verified against EmbedLayerNorm, FusedMatMul, MatMulNBits, QuantizeLinear
• Stateful CausalLM flow exercised for KV-cache reorder + rewind
• ORT_OPENVINO_PERF_COUNT=<dir> verified to produce per-subgraph CSVs
• 2+ GB external-initializers-in-memory model loads on CPU / GPU / NPU

[ankitm3k]

@adrianlizarraga Please review & merge this PR.

[Copilot]

Pull request overview

Upstream sync of Intel’s OpenVINO EP development branch into ONNX Runtime main, adding OpenVINO 2026.0/2026.1 support and updating OVEP behavior/tests around external initializers, stateful CausalLM KV-cache handling, and profiling/perf-count dumping.

Changes:

• Add OV 2026.0/2026.1 version handling and expand supported initializer/op capability metadata.
• Update KV-cache reorder/stateful infer-request interfaces and logic (incl. new failure behavior on RewindKVCache(index>0) when reorder is enabled).
• Improve external initializer handling/weight sharing + add/adjust OVEP tests; add env-driven perf-count CSV dumping; adjust perftest to reset outputs for data-dependent shapes.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc	Refactors/extents OVEP external-initializer tests; adds new edge-case coverage (dynamic embed + empty raw_data).
onnxruntime/test/providers/cpu/tensor/quantize_linear_test.cc	Adjusts skips/exclusions for QuantizeLinear coverage across EPs.
onnxruntime/test/perftest/ort_test_session.cc	Resets outputs between runs to handle data-dependent output shapes in perf tests.
onnxruntime/test/contrib_ops/matmul_4bits_test.cc	Updates tolerances and removes OpenVINO build guard to broaden test coverage.
onnxruntime/test/contrib_ops/fused_matmul_op_test.cc	Narrows OpenVINO exclusion to specific failing cases; keeps TensorRT excluded for unsupported dtype.
onnxruntime/test/contrib_ops/embed_layer_norm_op_test.cc	Only excludes OpenVINO when 3rd output is requested; otherwise runs normally.
onnxruntime/core/providers/openvino/qdq_transformations/qdq_stripping.cc	Switches offset/length parsing to from_chars for large (>4GB) external-data metadata.
onnxruntime/core/providers/openvino/ov_versions/data_ops.h	Adds V_2026_0 / V_2026_1 version enums.
onnxruntime/core/providers/openvino/ov_versions/data_ops.cc	Updates supported types/ops by OV version; adds FLOAT8 initializer types and ReduceSum no-dimension support.
onnxruntime/core/providers/openvino/ov_versions/capability.cc	Bumps default OV version mapping and filters orphaned graph outputs for subgraph outputs.
onnxruntime/core/providers/openvino/ov_stateful_patch_utils.cc	Updates cache reorder fusion to enforce mutual exclusivity of beam_idx vs src_idx/dst_idx paths.
onnxruntime/core/providers/openvino/ov_shared_context.h	Adds external-weight re-add validation and improves metadata insertion under lock.
onnxruntime/core/providers/openvino/ov_shared_context.cc	Adds locking around device-tensor mapping creation to fix races.
onnxruntime/core/providers/openvino/ov_interface.h	Renames KV-cache reorder API and adds cleanup hook for reorder status.
onnxruntime/core/providers/openvino/ov_interface.cc	Implements reorder tensor population/validation, cleanup after inference, and updated rewind behavior.
onnxruntime/core/providers/openvino/ov_bin_manager.cc	Adds bounds checks and changes mapped-blob tensor view construction for import.
onnxruntime/core/providers/openvino/openvino_provider_factory.cc	Updates provider option handling (NPU dynamic-shape defaulting; preserve user compilation params; preserve factory device_type).
onnxruntime/core/providers/openvino/openvino_provider_dllmain.cc	Calls ov::shutdown() on DLL unload.
onnxruntime/core/providers/openvino/openvino_execution_provider.cc	Renames dynamic option hook to call SetReorderKVCacheStatus.
onnxruntime/core/providers/openvino/ibackend.h	Makes Infer non-const and renames reorder interface.
onnxruntime/core/providers/openvino/exceptions.h	Adds human-readable exception type strings and adjusts error formatting.
onnxruntime/core/providers/openvino/backends/basic_backend.h	Adds perf-count dump helpers/state and updates interfaces for non-const infer + KV reorder rename.
onnxruntime/core/providers/openvino/backends/basic_backend.cc	Implements env-driven perf-count CSV dumping and skips compilation params on precompiled blob import.
onnxruntime/core/providers/openvino/backend_utils.h	Adds perf-count dump path accessor and changes perf-count print signatures.
onnxruntime/core/providers/openvino/backend_utils.cc	Implements ORT_OPENVINO_PERF_COUNT handling and CSV formatting for profiling output.
onnxruntime/core/providers/openvino/backend_manager.h	Renames backend-manager reorder interface.
onnxruntime/core/providers/openvino/backend_manager.cc	Updates export error messaging, gates OVEP QDQ stripping by OV version, adjusts external-initializer embedding heuristic, and updates debug model dumping.

Comments suppressed due to low confidence (1)

onnxruntime/core/providers/openvino/ov_interface.cc:534

• PreProcessInferRequest() sets src_idx/dst_idx twice: the block starting at if (is_kvcache_reorder_added) is duplicated later in the same function. This duplicates allocations/fills and makes the logic harder to maintain. Remove the second block (or factor into a helper) so the reorder tensors are prepared exactly once per inference.

  if (is_kvcache_reorder_added) {
    ov::Shape dst_idx_shape = ovInfReq.get_tensor("dst_idx").get_shape();
    const auto kv_num_heads = dst_idx_shape[1];
    const auto kv_head_size = dst_idx_shape[3];
    if (kv_src_indices.size() > 0) {
      ov::Tensor src_idx_tensor = ov::Tensor(ov::element::i32, {kv_src_indices.size()});
      const auto src_idx_ptr = src_idx_tensor.data<int32_t>();
      for (size_t i = 0; i < kv_src_indices.size(); ++i) {
        src_idx_ptr[i] = static_cast<int32_t>(kv_src_indices[i]);
      }
      ovInfReq.set_tensor("src_idx", src_idx_tensor);

      ov::Tensor dst_idx_tensor = ov::Tensor(ov::element::i32, {1, kv_num_heads, kv_dst_indices.size(), kv_head_size});
      const auto dst_idx_ptr = dst_idx_tensor.data<int32_t>();
      for (size_t i = 0; i < kv_num_heads; ++i) {
        for (size_t j = 0; j < kv_dst_indices.size(); ++j) {
          std::fill_n(dst_idx_ptr + (i * kv_dst_indices.size() + j) * kv_head_size, kv_head_size, kv_dst_indices[j]);
        }
      }
      ovInfReq.set_tensor("dst_idx", dst_idx_tensor);
    } else {
      FillTensor("src_idx", ov::element::i32, {0}, 0);
      FillTensor("dst_idx", ov::element::i32, {1, kv_num_heads, 0, kv_head_size}, 0);
    }
  }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MayureshV1]

@adrianlizarraga.. Addressed your review comments.