Skip to content

ONNX Runtime QNN Execution Provider v2.4.0

Latest
Latest

Choose a tag to compare

View all tags
@qti-mbadnara qti-mbadnara released this 14 Jul 18:52
· 143 commits to main since this release
v2.4.0
215ea95
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ONNX Runtime Compatibility: >= 1.24.1 (compiled with v1.26.0)

QAIRT SDK Compatibility: 2.48.40

pip install onnxruntime==1.26.0
pip install onnxruntime-qnn==2.4.0

Packaging

New in 2.4.0

  • ONNX Runtime — Compiled with v1.26.0. (#371)
  • ONNX opset library — Upleveled to v1.21.0, picking up opset 26. (#379)
  • Artifacts — Windows ARM64 (ARM64x), Windows x86_64, Windows ARM64EC Zip, Linux x86_64 tgz artifacts are now shipped as part of each Release.

Platform Support

Package Windows ARM64 Windows ARM64 (ARM64x) Windows x86_64 Linux ARM64 Linux x86_64 Android ARM64
Python Wheel Inference — AOT compilation Inference AOT compilation —
NuGet - Inference — — — —
ZIP Inference Inference AOT compilation — AOT compilation —
tgz — — — Inference — —
Maven — — — — — Inference

New Operators and Fusions

  • GRU — CPU (FP32) and HTP (QDQ/FP16). Time steps always unrolled. Default activations only; layout=1 and dynamic sequence lengths not supported. (#284)
  • IsInf — FP32/FP16 inputs; boolean output. (#403)
  • LpPool — p ∈ {1, 2}; ranks 3/4/5. CPU FP32; HTP FP32/FP16/BF16 (V81+); GPU FP32/FP16. QDQ not supported. (#437)
  • OneHot — depth and values must be constant initializers. HTP supports FP16/BF16 (V81+) and QDQ. Negative indices fall back to CPU EP. (#466)
  • Selu — CPU, HTP, and GPU. FP32/FP16. (#404)
  • Xor (#402)
  • Gather block quantize (GPU) — Added quantization support for gather blocks in LLMs, enabling execution on Qualcomm GPU through EpContextBinary. (#356)
  • GroupQueryAttention (GPU) — Support com.Microsoft.GroupQueryAttention by mapping onto QNN_OP_GROUP_QUERY_ATTENTION, enabling GQA nodes in LLMs to run on GPU for better performance. (#424)
  • DynamicQuantizeLinear → DequantizeLinear — Fuses DQL+DQ round-trips into an identity Transpose, eliminating CPU fallback in models with incomplete quantization conversion. (#490)

For the full list of supported operators, see Supported ONNX Operators and for supported fusions, see Supported Operator Fusions.

Improvements

  • DX12 shared memory (GPU) — Added DX12 shared memory allocator via enable_dx12_shared_memory_allocator provider option, leveraging QNN_MEM_TYPE_DX12 mem-handles. Eliminates CPU↔GPU copies of KV cache tensors for improved LLM inferencing speed. (#213)
  • MatMul / Gemm — Block-quantized (BW_FLOAT_BLOCK) weight support on HTP. INT4/UINT4/INT8/UINT8 weights. Requires QAIRT >= 2.47. (#476, #477)
  • LayerNormalization — Decomposes into LN → Mul + Add when scale/bias rank is not aligned with the normalization axes. (#417)
  • Resize — Routes rank-4 linear pytorch_half_pixel Resize to ResizeBilinear. (#393)
  • RoiAlign — Accepts coordinate_transformation_mode=half_pixel and sampling_ratio=0. (#389)
  • SimplifiedLayerNormalization — Added to the RMSNorm QDQ node-group selector, enabling QDQ execution on HTP. (#338)
  • DLC validation — backendValidateOpConfig now routes to the target backend (HTP/CPU/GPU) when dump_qnn_ir_dlc=1, instead of always using QnnIr. (#438)
  • Framework op tracing — New qnn.enable_framework_op_trace / qnn.framework_op_trace_dir session options for debugging graph partitioning. (#345)
  • QNN CPU backend — No longer advertised by default; set ORT_QNN_ENABLE_CPU_BACKEND to re-enable. (#500)
  • HNRD path warning — Logs a one-shot warning when the HNRD fallback path is active. (#453)
  • ETW profiling — No longer escalates via ETW when profiling_level is OFF. (#446)
  • prepare_only validation — Errors at session creation if prepare_only=1 is set without ep.context_enable=1. (#514)

Bug Fixes

  • BatchNormalization — Run in FP when input is u8/u16. (#434)
  • BatchNormalization — Allow quantized-input / no-output-Q BN as a float island on HTP; fixes graph capture failures in NASNet-style models. (#497)
  • Conv + Relu — Insert explicit Relu when fused output encoding has zp > 0. (#478)
  • GlobalAveragePool — Remove unconditional count_pad_for_edges = true; aligns DLC output with the native converter. (#460)
  • HardSigmoid — Fix output encodings to full [0, 1] range required by HTP. (#473)
  • RandomNormalLike / RandomUniformLike — Fix "Zero tensor size!" crash caused by missing QnnTensorWrapper for the shape-source x input. (#415)
  • GetGenieCapability — Fix crash from dangling session_options_ reference on Windows ARM64 LLM model load. (#513)
  • Linux ARM64 NPU detection — QNN EP now registers an NPU device on Linux ARM64 by probing /dev/fastrpc-cdsp*, fixing "No QNN EP devices found". (#557)
  • offload_graph_io_quantization — Fix crash on EPContext load by serializing the tensor-name rename map into the EPContext node at compile time. Backward compatible with previously generated context binaries. (#544)
  • Quantized Gelu ↔ Erf fusion — Support Div/Mul as the parent of Erf (optimizers can rewrite Div(sqrt(2)) as Mul(1/sqrt(2))), canonicalize duplicated intermediate root tensor names produced by ORT, validate against outer quantized I/O when wrapped by DQ/Q, and reject fusion when intermediate QDQGroups are present to avoid accuracy loss. (#309)

Full Changelog: rel-2.3.0...rel-2.4.0

Contributors

This release includes contributions from:

Ashwath Shankarnarayan, Badri Narayanan, Calvin Nguyen, Cheng-Hsin Weng, Chun-Chih Teng, Hua-Yu Chou, Hung-Jui Wang, Kuan-Yu Lin, Kyle Romero, Matthew Sinclair, Mike Hsu, Min Fong Hong, Mu-Chien Hsu, Nischay Mamidi, Sachin Jangid, Simon Janezic, Shubham Patel, Tirupathi Reddy T, Vineeth Jatoth, Yathindra Kota, Yuduo Wu, Yu-Hung Chuang

Assets 7
Loading