Skip to content

v0.5.3

Latest

Choose a tag to compare

@zz002 zz002 released this 09 Nov 10:17
· 36 commits to main since this release
cfca7f9

hip-ep v0.5.3, tagged at cfca7f99 on main.

Highlights since v0.5.2

New models and operators

  • Qwen3.8-Flash-Next: GatedDeltaNet, HyperConnection Pre/PostMix, sparse attention (MRotaryEmbedding, SparseAttentionIndexer, DynamicSparseAttention), Engram (NGramHashMapping, fp8 GatherBlockQuantized) and five norm/gate operators are supported. The 48.8 GB engram table stays on disk, and gathers read only the rows they need.
  • Laguna-S/XS-2.1: Attention with a key-padding mask stays on fused GQA, and the sigmoid-bias router is fused into QMoE. Attention prefill per 2k chunk drops from 222.3 ms to 35.4 ms.
  • Speculative decoding: on gfx1151 (Spec-Bench), Qwen3.8-27B EAGLE4 runs 2.31x, Qwen3.6-35B-A3B DFlash2 1.94x and Gemma-4-26B-A4B MTP 1.37x faster than plain decoding. A pipeline script for DFlash2/DSpark is included.
  • More coverage: the Whisper decoder compiles; QMoE supports 2-bit expert weights, relu2 and the router_weights input; ONNX Scan and If, quantized integer Conv/MatMul, EmbedLayerNormalization, NonMaxSuppression, ArgMax and int8 Resize are lowered.

Performance

  • DEFAULT memory is now backed by hipMalloc with a per-session transient arena. On gfx1151, Mistral-Small-4 image TTFT drops 13-15%; on gfx1201, Llama-3.1-8B at 2048 tokens has 27.8% lower TTFT and 89.8% higher decode throughput.
  • int4 WMMA prefill kernels on gfx1151: median TTFT across six models drops 9.8% on Windows and 11.1% on Linux, by up to 26.4%.
  • Gemma-4 E4B: decode TPS +12.1% (CI geomean); GPU peak memory at 16K context drops from 31.79 GiB to 13.48 GiB.
  • GQA prefill is 4-7% faster, and intermittent inf/NaN on gfx11 is fixed.
  • Targeted optimizations: gpt-oss decode; Qwen3.8-27B LinearAttention, SwiGLU and RMSNorm; GQA flash decode for 27B-class models; re-tuned dp4a GEMV; TopK with K > 256 now sorts on the GPU, about 19-68x faster.
  • Opt-in: FP16 Ring attention on gfx1151 (HIPDNN_EP_USE_RING_ATTENTION=1), +4.4% decode at 64K context.

Dependencies and platforms

  • ONNX Runtime 1.30.0 -> 1.31.0; ONNX Runtime GenAI 0.17.0 -> 0.19.0 with one upstream patch.
  • Linux TheRock 7.13.0 -> 10.1.0 (HIP 7.16, hipBLASLt 1.4). The Windows ROCm SDK now includes gfx117x, and gfx117x kernels are built once as gfx11-7-generic. Offline LUTs added for gfx1201 (RX 9070 XT).
  • Optional ROCTx ranges for profiling (HIPDNN_EP_ROCTX=1).

Assets

asset sha256
hipep_0.5.3_windows.zip ee1aadbf46142b1a4804fae531a6bc476966eca07e301f75310084f8e32f3c07
hipep_0.5.3_windows_python.zip ba06cbb818e8c51fa935bb9ed3d9f74aa674efa9013767ebefae187dc2dbf31c
hipep_0.5.3_linux.zip d0f72eb9b477c5fa990f4142f65a7c959d54a32f38bd936b9f8aa7581849ec76

Built from Windows run 42591814936 and Linux run 42591814969; the Windows binaries are code-signed by AMD. ORT 1.31.0 (no patches); OGA 0.19.0 with one upstream patch; umbrella EP f3dccce2 (no patches). TheRock Windows = rocm-dist-windows-gfx115x-gfx117x-7.16.0-hipblaslt-gfx117x-trimmed; MIGraphX Windows = migraphx-sdk-windows-7bb0654e; Linux = therock-dist-linux-gfx1151-10.1.0.

The Windows ROCm SDK tarball referenced by cmake/deps.txt is hosted on v0.5.2, and the MIGraphX SDK tarball on v0.5.1.