Repository navigation
hip-ep v0.5.3, tagged at cfca7f99 on main.
Highlights since v0.5.2
New models and operators
- Qwen3.8-Flash-Next: GatedDeltaNet, HyperConnection Pre/PostMix, sparse attention (MRotaryEmbedding, SparseAttentionIndexer, DynamicSparseAttention), Engram (NGramHashMapping, fp8 GatherBlockQuantized) and five norm/gate operators are supported. The 48.8 GB engram table stays on disk, and gathers read only the rows they need.
- Laguna-S/XS-2.1: Attention with a key-padding mask stays on fused GQA, and the sigmoid-bias router is fused into QMoE. Attention prefill per 2k chunk drops from 222.3 ms to 35.4 ms.
- Speculative decoding: on gfx1151 (Spec-Bench), Qwen3.8-27B EAGLE4 runs 2.31x, Qwen3.6-35B-A3B DFlash2 1.94x and Gemma-4-26B-A4B MTP 1.37x faster than plain decoding. A pipeline script for DFlash2/DSpark is included.
- More coverage: the Whisper decoder compiles; QMoE supports 2-bit expert weights, relu2 and the router_weights input; ONNX Scan and If, quantized integer Conv/MatMul, EmbedLayerNormalization, NonMaxSuppression, ArgMax and int8 Resize are lowered.
Performance
- DEFAULT memory is now backed by hipMalloc with a per-session transient arena. On gfx1151, Mistral-Small-4 image TTFT drops 13-15%; on gfx1201, Llama-3.1-8B at 2048 tokens has 27.8% lower TTFT and 89.8% higher decode throughput.
- int4 WMMA prefill kernels on gfx1151: median TTFT across six models drops 9.8% on Windows and 11.1% on Linux, by up to 26.4%.
- Gemma-4 E4B: decode TPS +12.1% (CI geomean); GPU peak memory at 16K context drops from 31.79 GiB to 13.48 GiB.
- GQA prefill is 4-7% faster, and intermittent inf/NaN on gfx11 is fixed.
- Targeted optimizations: gpt-oss decode; Qwen3.8-27B LinearAttention, SwiGLU and RMSNorm; GQA flash decode for 27B-class models; re-tuned dp4a GEMV; TopK with K > 256 now sorts on the GPU, about 19-68x faster.
- Opt-in: FP16 Ring attention on gfx1151 (
HIPDNN_EP_USE_RING_ATTENTION=1), +4.4% decode at 64K context.
Dependencies and platforms
- ONNX Runtime 1.30.0 -> 1.31.0; ONNX Runtime GenAI 0.17.0 -> 0.19.0 with one upstream patch.
- Linux TheRock 7.13.0 -> 10.1.0 (HIP 7.16, hipBLASLt 1.4). The Windows ROCm SDK now includes gfx117x, and gfx117x kernels are built once as
gfx11-7-generic. Offline LUTs added for gfx1201 (RX 9070 XT). - Optional ROCTx ranges for profiling (
HIPDNN_EP_ROCTX=1).
Assets
| asset | sha256 |
|---|---|
hipep_0.5.3_windows.zip |
ee1aadbf46142b1a4804fae531a6bc476966eca07e301f75310084f8e32f3c07 |
hipep_0.5.3_windows_python.zip |
ba06cbb818e8c51fa935bb9ed3d9f74aa674efa9013767ebefae187dc2dbf31c |
hipep_0.5.3_linux.zip |
d0f72eb9b477c5fa990f4142f65a7c959d54a32f38bd936b9f8aa7581849ec76 |
Built from Windows run 42591814936 and Linux run 42591814969; the Windows binaries are code-signed by AMD. ORT 1.31.0 (no patches); OGA 0.19.0 with one upstream patch; umbrella EP f3dccce2 (no patches). TheRock Windows = rocm-dist-windows-gfx115x-gfx117x-7.16.0-hipblaslt-gfx117x-trimmed; MIGraphX Windows = migraphx-sdk-windows-7bb0654e; Linux = therock-dist-linux-gfx1151-10.1.0.
The Windows ROCm SDK tarball referenced by cmake/deps.txt is hosted on v0.5.2, and the MIGraphX SDK tarball on v0.5.1.