Skip to content

v0.6.18+amd.2

Latest

Choose a tag to compare

@demandal25 demandal25 released this 22 Oct 18:21
· 8 commits to amd-integration since this release
5f2e2b2

Added

  • Speculative-decode verify (q_len_per_req > 1) on ROCm, so chain/MTP speculative decoding has a supported path end to end (#362) @demandal25
  • fp8 KV cache: paged prefill on the AITER route at 1.22–1.66× over bf16 on gfx942, and fp8 KV in the in-tree prefill kernel, including BatchDecodeWithPagedKVCacheWrapper(use_tensor_cores=True); every combination still unserved now fails with a sentence instead of a ninja log (#366, #368) @demandal25
  • AITER's asm prefill arm is reached for real and routed for qo_len >= 2048 on gfx950 (about 1.14×); gfx942 stays on CK Tile (#365, #370) @demandal25
  • torch.compile with FLASHINFER_USE_TORCH_CUSTOM_OPS=1 now traces AITER single/paged/ragged prefill, aiter_fused_moe and AITER batch decode with return_lse=True, the routes auto picks by default; PyTorch 2.12 can still miscompile an op that writes into a view of a larger tensor, so give the tensors an op writes their own storage, as docs/rocm/torch-compile.md explains (#387, #388, #397, #398) @demandal25
  • The benchmark runner's norm, rope, sampling, block-sparse and POD routines run on ROCm (#364, #369) @demandal25

Changed

  • Supported configuration is now ROCm 10.1.0 / PyTorch 2.12.0 / CPython 3.12 / Ubuntu 24.04 / amd-aiter 0.1.21.post2, built from source at a pinned tag in docker/Dockerfile.rocm, with every AITER prefill variant the shim loads prebuilt into a FlashInfer-owned store instead of compiled on first use; AITER decode still compiles at plan() (#363, #375, #378) @demandal25
  • In-tree prefill on CDNA4 replaces the per-MFMA V transpose with the transposing LDS read ds_read_b64_tr_b16: 1.71× on gfx950, gfx942 ISA byte-identical (#376) @demandal25
  • AITER flat-gather paged prefill builds its indices on device and gathers from the pool without copying it: plan() from 29–50 ms to under 1.5 ms, and run() reads only the batch's rows (#384) @demandal25
  • backend="auto" routing re-measured: short-query paged prefill no longer goes to AITER at 2.3–4.6× the in-tree cost, short-query ragged prefill is gated on gfx942, soft-capped causal prefill is ungated on gfx942 (1.99–5.36× it was paying) and on gfx950 under ROCm 10.1 (#361, #362, #372, #373, #378) @demandal25
  • Documentation brought up to the release: the README states the validated configuration and keeps to getting started, the support matrix and what upstream has that this does not, with torch.compile and environment-variable reference moved to docs/rocm/; comments and docstrings match the code; the ROCm example scripts and tutorial notebook are removed in favour of upstream's documentation (#385, #386, #398) @demandal25

Fixed

  • sinks was accepted and silently ignored, and fp8 descales were dropped on routes that never read them; both now raise. The PDL warning on every default run() is gone, and FLASHINFER_JIT_DEBUG works on ROCm (#382) @demandal25
  • AITER native paged prefill read past the end of the KV cache at head_dim 64 and 96 (fault, hang, or wrong reads); native paging is now routed only at head_dim 128/256 (#383) @demandal25
  • The paged prefill wrapper crashed on two cuDNN-only plan() hints, and the jit-cache version check accepted another release's wheel (#386) @demandal25
  • A customized AITER single-prefill module with its own parameter list registers again (#387) @demandal25
  • Under FLASHINFER_USE_TORCH_CUSTOM_OPS=1, compiled batch decode and fa2 paged prefill over a KV cache passed as views (a layer of a pool, K/V halves of one tensor) read the wrong K/V, overwrote V or faulted the GPU; AITER decode failed under mode="reduce-overhead" and silently misread K/V/q layouts PA v1 cannot address, which now raise or are packed (#397) @demandal25

Maintenance

  • Repository baseline adopted: governance files, Dependabot, and a security-scan workflow on every PR (bandit, CodeQL, gitleaks, trivy, zizmor) (#392, #393) @demandal25
  • gfx950 line coverage of the AMD-specific code from 90.66% to 95.04% (#377) @demandal25
  • The soft-cap gate test reaches the kernel it measures, the paged mask lifecycle is covered, and the fa2 decode head-count cap is profiled on both architectures with its negative results recorded (#371, #374) @demandal25

Contributors: @demandal25

Summary: A performance and correctness release on upstream v0.6.18, now on ROCm 10.1. On CDNA4 the in-tree prefill kernel is 1.71× faster through the transposing LDS read, and long prefills route to AITER's asm arm. fp8 KV caches work on both the AITER paged route and the in-tree kernel, speculative-decode verify has a supported path, and auto routing was re-measured shape by shape on both architectures. AITER is built from source at a pinned tag with every prefill variant prebuilt, so AITER prefill no longer compiles on the user's machine. Several silent defects are now errors or fixed: ignored attention sinks, dropped fp8 descales, an out-of-bounds KV read in AITER native paged prefill at head_dim 64/96, and compiled attention over KV-cache views reading the wrong memory. The full test suite passes at the release head on both gfx942 (MI300X) and gfx950 (MI355X) in the supported image.