Skip to content

ONNX Runtime QNN Execution Provider v2.5.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Aug 01:26
· 94 commits to main since this release
v2.5.0
5324658
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ONNX Runtime Compatibility: >= 1.24.1 (compiled with v1.26.0)

QAIRT SDK Compatibility: 2.49.40

pip install onnxruntime==1.26.0
pip install onnxruntime-qnn==2.5.0

Packaging

Platform Support

Package Windows ARM64 Windows ARM64 (ARM64x) Windows x86_64 Linux ARM64 Linux x86_64 Android ARM64
Python Wheel Inference — AOT compilation Inference AOT compilation —
NuGet - Inference — — — —
ZIP Inference Inference AOT compilation — AOT compilation —
tgz — — — Inference — —
Maven — — — — — Inference

New Ops and Fusions

  • Shape (#510)
  • ReduceLogSumExp (#463)
  • QLinearMatMul (#467)
  • NonMaxSuppression (#559)
  • If (#409)
  • Reshape-Gemm-Reshape(-Reshape) fusion — Fuses Reshape(ND→2D) → Gemm → [Reshape(2D→MD) → [Reshape(MD→PD)]] into QNN_OP_FULLY_CONNECTED (+ QNN_OP_RESHAPE), matching the pattern ORT emits for ND matmul+add. (#232)
  • MatMul → Conv2D (1x1) for LPBQ — Lowers MatMul to QNN_OP_CONV_2D with 1x1 weights when LPBQ encodings are present. (#357)

For the full list of supported operators, see Supported ONNX Operators and for supported fusions, see Supported Operator Fusions.

Improvements

  • QNN GPU weight sharing — Extends weight sharing (previously HTP-only) to the QNN GPU backend, reducing per-session memory when multiple sessions share weights. (#67)
  • QNN context priority levels — Extended from 4 to 8 levels, adding NORMAL_LOW, HIGH_PLUS, CRITICAL, CRITICAL_PLUS. (#106, #568)
  • Framework op tracing — Added profiling↔trace auto-merge (onnx-op column in profiling output at detailed/optrace levels) and AOT Phase 2 sidecar discovery. (#488)
  • Framework op tracing — Added dump_qnn_ep_input_graph session config, a source-to-optimized ONNX structural matcher, and profiling-CSV enrichment with original ONNX source ops, closing the source→optimized→QNN-op provenance gap. (#491)
  • QNN API error diagnostics — graphFinalize, graphExecute, graphCreate, and contextApplyBinarySection failures now include the QNN symbolic error string, not just a bare integer code. (#532)
  • RMSNorm (HTP) — Removed the dummy all-zeros beta tensor on QAIRT SDK 2.49+, which now accepts beta as a truly optional input. (#581)
  • 4-bit block-quantized (BQ) weights on HTP — New enable_block_quant_weight_optimization session option converts int4 BQ weight encodings to LPBQ when supported, falling back to the standard BQ compatibility path otherwise. See Configuration Options for details and trade-offs. (#307)
  • HTP fp16 overflow clamping — New fp16_clamp_overflow HTP session option (default OFF, requires QNN API >= 2.38 / QAIRT 2.49) clamps fp16 Conv overflow instead of producing NaN/Inf. Intentionally undocumented as it changes HTP numerical behavior. (#686)

Op Translation Fixes

  • Pad — Added float16 support for the constant_value attribute on HTP, avoiding CPU fallback. (#546)
  • ReduceL2 (QDQ) — Registered in the QDQ node-unit selector so quantized ReduceL2 fuses correctly instead of silently decomposing into an unquantized float32 island. (#596)
  • Clip — Accepts min/max fed by a DequantizeLinear over a constant initializer (common AIMET-export pattern), by treating folded-constant DQ outputs the same as initializers. (#586)
  • Softmax — Fixed HTP misbehaving when AIMET populates a non-natural output encoding (outside [0, 1]) to satisfy older symmetric-input MatMul requirements; now emits the correct [0, 1] encoding plus a Convert op. Recovers major accuracy loss on YoloV10/V11 detection models. (#558)
  • SpaceToDepth CRD fusion — Fixed a regression where dynamic batch dimension (-1) in Reshape shape initializers was rejected, causing 6D-tensor CPU fallback on dynamic-batch models. (#567)
  • BatchNormalization — Fixed incorrect bias-tensor reuse when two BN nodes share the same ONNX bias initializer but have different mean/variance, silently producing wrong output. Recovers major accuracy loss on Sinet. (#582)
  • GroupQueryAttention — Fixed a shape mismatch where QNN's GQA op def requires total_sequence_length as a 0D scalar while ONNX's com.microsoft.GroupQueryAttention supplies it as a 1D [1] tensor, causing op validation failure. (#654)

Bug Fixes

  • Utils — Fixed undefined behavior from calling std::vector::assign with a reference into the same vector, which crashed MSVC debug builds. (#511)
  • Android NPU discovery — Standalone QNN EP now detects Qualcomm devices via the ro.soc.manufacturer system property instead of scanning /dev/fastrpc-cdsp*, which Android's SELinux policy blocks for untrusted apps. Fixes getEpDevices() returning no NPU device (and QNN EP being unusable) on Android even on supported devices. Linux ARM64 behavior is unchanged. (#683)

Full Changelog: rel-2.4.0...rel-2.5.0

Known Issues

  • LLM model load failure (QNN_MEMORY_ALLOCATION_ERROR, 1002) — Some LLM models fail to load with all EP context binaries. Workaround: enable the enable_vtcm_backup_buffer_sharing QNN EP option. This option is AOT-flow only and may not cover all use cases. Fix targeted for 2.6.0.

Contributors

This release includes contributions from:

Ashima Jain, Ashwath Shankarnarayan, Badri Narayanan, Calvin Nguyen, Chun-Chih Teng, Hua-Yu Chou, Kuan-Yu Lin, Kyle Romero, Matthew Sinclair, Mike Hsu, Min Fong Hong, Nageshwar Singh, Nischay Mamidi, Tirupathi Reddy T, Vineeth Jatoth, Yathindra Kota, Yu-Hung Chuang, Yuduo Wu

Assets 7
Loading