Repository navigation
Added
- Speculative-decode verify (
q_len_per_req > 1) on ROCm, so chain/MTP speculative decoding has a supported path end to end (#362) @demandal25 - fp8 KV cache: paged prefill on the AITER route at 1.22–1.66× over bf16 on gfx942, and fp8 KV in the in-tree prefill kernel, including
BatchDecodeWithPagedKVCacheWrapper(use_tensor_cores=True); every combination still unserved now fails with a sentence instead of a ninja log (#366, #368) @demandal25 - AITER's asm prefill arm is reached for real and routed for
qo_len >= 2048on gfx950 (about 1.14×); gfx942 stays on CK Tile (#365, #370) @demandal25 torch.compilewithFLASHINFER_USE_TORCH_CUSTOM_OPS=1now traces AITER single/paged/ragged prefill,aiter_fused_moeand AITER batch decode withreturn_lse=True, the routesautopicks by default; PyTorch 2.12 can still miscompile an op that writes into a view of a larger tensor, so give the tensors an op writes their own storage, asdocs/rocm/torch-compile.mdexplains (#387, #388, #397, #398) @demandal25- The benchmark runner's norm, rope, sampling, block-sparse and POD routines run on ROCm (#364, #369) @demandal25
Changed
- Supported configuration is now ROCm 10.1.0 / PyTorch 2.12.0 / CPython 3.12 / Ubuntu 24.04 /
amd-aiter0.1.21.post2, built from source at a pinned tag indocker/Dockerfile.rocm, with every AITER prefill variant the shim loads prebuilt into a FlashInfer-owned store instead of compiled on first use; AITER decode still compiles atplan()(#363, #375, #378) @demandal25 - In-tree prefill on CDNA4 replaces the per-MFMA V transpose with the transposing LDS read
ds_read_b64_tr_b16: 1.71× on gfx950, gfx942 ISA byte-identical (#376) @demandal25 - AITER flat-gather paged prefill builds its indices on device and gathers from the pool without copying it:
plan()from 29–50 ms to under 1.5 ms, andrun()reads only the batch's rows (#384) @demandal25 backend="auto"routing re-measured: short-query paged prefill no longer goes to AITER at 2.3–4.6× the in-tree cost, short-query ragged prefill is gated on gfx942, soft-capped causal prefill is ungated on gfx942 (1.99–5.36× it was paying) and on gfx950 under ROCm 10.1 (#361, #362, #372, #373, #378) @demandal25- Documentation brought up to the release: the README states the validated configuration and keeps to getting started, the support matrix and what upstream has that this does not, with
torch.compileand environment-variable reference moved todocs/rocm/; comments and docstrings match the code; the ROCm example scripts and tutorial notebook are removed in favour of upstream's documentation (#385, #386, #398) @demandal25
Fixed
sinkswas accepted and silently ignored, and fp8 descales were dropped on routes that never read them; both now raise. The PDL warning on every defaultrun()is gone, andFLASHINFER_JIT_DEBUGworks on ROCm (#382) @demandal25- AITER native paged prefill read past the end of the KV cache at head_dim 64 and 96 (fault, hang, or wrong reads); native paging is now routed only at head_dim 128/256 (#383) @demandal25
- The paged prefill wrapper crashed on two cuDNN-only
plan()hints, and the jit-cache version check accepted another release's wheel (#386) @demandal25 - A customized AITER single-prefill module with its own parameter list registers again (#387) @demandal25
- Under
FLASHINFER_USE_TORCH_CUSTOM_OPS=1, compiled batch decode and fa2 paged prefill over a KV cache passed as views (a layer of a pool, K/V halves of one tensor) read the wrong K/V, overwrote V or faulted the GPU; AITER decode failed undermode="reduce-overhead"and silently misread K/V/q layouts PA v1 cannot address, which now raise or are packed (#397) @demandal25
Maintenance
- Repository baseline adopted: governance files, Dependabot, and a security-scan workflow on every PR (bandit, CodeQL, gitleaks, trivy, zizmor) (#392, #393) @demandal25
- gfx950 line coverage of the AMD-specific code from 90.66% to 95.04% (#377) @demandal25
- The soft-cap gate test reaches the kernel it measures, the paged mask lifecycle is covered, and the fa2 decode head-count cap is profiled on both architectures with its negative results recorded (#371, #374) @demandal25
Contributors: @demandal25
Summary: A performance and correctness release on upstream v0.6.18, now on ROCm 10.1. On CDNA4 the in-tree prefill kernel is 1.71× faster through the transposing LDS read, and long prefills route to AITER's asm arm. fp8 KV caches work on both the AITER paged route and the in-tree kernel, speculative-decode verify has a supported path, and auto routing was re-measured shape by shape on both architectures. AITER is built from source at a pinned tag with every prefill variant prebuilt, so AITER prefill no longer compiles on the user's machine. Several silent defects are now errors or fixed: ignored attention sinks, dropped fp8 descales, an out-of-bounds KV read in AITER native paged prefill at head_dim 64/96, and compiled attention over KV-cache views reading the wrong memory. The full test suite passes at the release head on both gfx942 (MI300X) and gfx950 (MI355X) in the supported image.