AITER v0.1.20

AITER v0.1.20 — bi-weekly release.

Cut from release/v0.1.20 @ fc2e5d57 and diffed against v0.1.19. This is the scheduled release line, not a post-release hotfix.

Wheels

Prebuilt manylinux_2_28 wheels, GPU_ARCHS=gfx942;gfx950:

ROCm	Python	Wheel
ROCm 7.0	cp310	amd_aiter-0.1.20+rocm7.0.manylinux.2.28-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.0	cp312	amd_aiter-0.1.20+rocm7.0.manylinux.2.28-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.1	cp310	amd_aiter-0.1.20+rocm7.1.manylinux.2.28-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.1	cp312	amd_aiter-0.1.20+rocm7.1.manylinux.2.28-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.2	cp310	amd_aiter-0.1.20+rocm7.2.manylinux.2.28-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.2	cp312	amd_aiter-0.1.20+rocm7.2.manylinux.2.28-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Build runs

• ROCm 7.0 cp310: https://github.com/ROCm/aiter/actions/runs/32124560585
• ROCm 7.0 cp312: https://github.com/ROCm/aiter/actions/runs/32215589704
• ROCm 7.1 cp310: https://github.com/ROCm/aiter/actions/runs/32211815213
• ROCm 7.1 cp312: https://github.com/ROCm/aiter/actions/runs/32124599999
• ROCm 7.2 cp310/cp312: https://github.com/ROCm/aiter/actions/runs/32124600442

What's Changed

Changes

• [Triton/Gluon] Streamline GEMM and MoE configs by @Boss2002n in #4379
• Fix: add missing end_sync barrier in fused allreduce+rmsnorm kernel by @yuzho-amd in #4346
• docs(readme): announce Kimi-K3 support in News by @carlushuang in #4404
• DeepSeek-V4 FP4: fused_compress FP4 scatter + rmsnorm_rope_rotate FP4 KV-cache kernel by @junhaha666 in #4029
• [module_pos_encoding] refactor and rm torch by @amd-ruitang3 in #4393
• Correcting errors and discrepancies in the MLA v4 mi355 kernel design documents by @ruanjm in #4392
• [FLYDSL] Support paged mqa logits fp4 varqlen kernel by @zhiding512 in #4230
• [CI] pin ruff and make its configuration explicit by @valarLip in #4403
• Fix Q UE8M0 quant and require fp32 LN params in fused DSv3.2 indexer kernel by @frida-andersson in #3451
• refactor(flydsl): vendor buffer_ops/vector into aiter by @Phil-amd in #4402
• [MLA][gluon] support nhead=96 for MLA by @yanxuer-999 in #4412
• fix(gemm_a8w8_blockscale): prevent scale OOB and support Triton 3.6 by @bingxche in #4406
• [tune] Kimi-K2.5/K2.6 fp4: port BM16 (flydsl_mxmoe_g*_a4w4) for decode gemm2 (+3~13% throughput) by @jiacao-amd in #4307
• [module_quick_all_reduce] refactor and rm torch by @amd-ruitang3 in #4421
• Enable stride aware indexing on top of strided blocks for block_size and heads (assumes contiguous head dim) of fused_qk_norm_rope_cache_pts_quant_shuffle() by @jhu960213 in #4312
• fix: synchronize custom collectives before return by @jpy794 in #4082
• Fix fused_qk_rope_concat_and_cache_mla for DCP by @yitingw1 in #4342
• [gfx1250][FlyDSL] Refactor GEMMs with layout-based API by @aoli26 in #4374
• fix(dsv4): use vec_size=32 for dim=1024 rotate-quant kernels by @XiaobingSuper in #4438
• fix(flydsl-aot): map Situv2 activation in MoE AOT parse_csv by @coderfeli in #4429
• [Triton] [Gluon] [GFX12] DSV4 Pro EP tunning (=TP1 shape) by @k50112113 in #4253
• fix(mla): refresh qh16 fp8 persistent decode HSACO for large page_id by @fangche123 in #4341
• Fix large-token FlyDSL MoE launch and output limits by @XiaobingSuper in #4417
• [module_rmsnorm_quant] refactor and rm torch by @amd-ruitang3 in #4434
• [MoE] Add swiglu_oai (OAI SwiGLU) for per-token fp8 CK XDL 2-stage MoE by @LJ-underdog in #3886
• [CK][VSA] Add thin sparse attention operator by @LiuYinfeng01 in #4373
• feat(topksoftmax): add 640-expert top-8 asm kernels for gfx942/gfx950 by @junhaha666 in #4454
• [triton] Optimized Unified Attention for Gemma-4-31b by @a-sidorova in #4044
• ci: extend multi-gpu test timeout by @gyohuangxin in #4468
• Fix for 32 bit GU offsets overflow by @JohnNikolay84 in #4449
• Add inverse_rope_group_quant op for DeepSeek-V4 wo_a input by @yzhou103 in #4428
• moe rebase and refactor by @yadaish in #4394
• [opus_moe] production A8W4 MoE stage1 kernels by @yifehuan in #4464
• [config] add k3 gemm&moe tuned configs by @gbyu-amd in #4435
• [Feature][FlyDSL] Add fused heterogeneous MXFP4/FP8 shared-expert MoE (FHMoE) and share the mixed-MoE kernel builders by @Fangzhou-Ai in #4269
• Flydsl kernel cleanup by @coderfeli in #4501
• Add an opt-in a4w4 SiTUv2 MoE path and fix three SiTUv2 tuner defects by @XiaobingSuper in #4463
• optimize ctypes marshalling by @amd-ruitang3 in #4465
• CI: auto-update split test FILE_TIMES by @aiter-gh-app[bot] in #4518
• fix(module_rmsnorm_quant): bound packed FP4 output stores by @gbyu-amd in #4467
• [Perf][FlyDSL] Tune DeepSeek V4 fused MoE for C1/C2/C32/C64 decode by @Fangzhou-Ai in #4314
• [gfx1250][FlyDSL] Unify&Rename GEMM kernels and refactor LDS load by @aoli26 in #4527
• chore(flydsl): bump flydsl dependency to 0.3.0 by @coderfeli in #4431
• Fix LDS allocation for B-to-LDS FlyDSL Split-K HGEMM by @xytpai in #4529
• Flash attention sliding window tests by @micmelesse in #4003
• gfx1250: fix the grouped-MoE expert scan above 512 experts, and compute SiTUv2 instead of SiLU by @XiaobingSuper in #4482
• ci: switch MI300X jobs to OCI runners by @gyohuangxin in #4520
• [fmoe][gfx950]Flydsl mxmoe v2 by @charlieguo1106 in #4179
• [CI] Fix SGLang downstream setup and enable DSV3.2 accuracy by @bingxche in #4516
• [Bugfix][Triton] Fix int32 KV-offset overflow in _mla_gluon >2GB path by @peizhang56 in #4474
• [gfx1250] update asm f4gemm, add fp8 out support, enhance ut by @dbyoung18 in #4335
• [PERF] Eliminate varlen prefill GDN D2H synchronization with reusable metadata by @yiijin in #4532
• fix mxmoe CI bug(flydsl0.2.4->0.3.0) by @charlieguo1106 in #4553
• [triton] fix transpose_scale in fused_rms_fp8_group_quant (was silently row-major) by @karverma-amd in #4506
• [TRITON][GLUON] Fixing Python 3.14 Compatibility Issue About @aggregate by @cagrikymk in #4508
• [fix][flydsl] fix oob scale descriptor in ptpc fp8 gemm by @gbyu-amd in #4546
• [gfx1250] Route the missing Kimi-K3 fused BF16 GEMM to Triton by @XiaobingSuper in #4552
• [dist] forward transpose_scale through fused AR+RMSNorm per-group quant by @yichiche in #4478
• fix(mla): refresh gfx950 MLA HSACO for large page_id KV addressing by @fangche123 in #4452
• configs: add DSv3-MXFP4 E=33/topk9 fused-MoE shape (shared-expert fus… by @rbrugaro-amd in #3739
• [Fix] Add assert for layernorm weight/input dtype mismatch by @yitingw1 in #4567
• [OpusMoe] optimize opus moe situv2 ability by @yifehuan in #4534
• MXMoE kernels for Qwen3.5-397B TP2 decode by @zijiecode in #4513
• MI350 MLA PS mode add ds32 opus kernel for nhead*qseqlen=128 case by @minmengdie in #4430
• feat(gemm): add tuned bf16 GEMM config for Qwen3-8B (gfx950) by @jiayyu in #4570
• [OPUS][ATOM] replace KV buffer loads with 64-bit global loads in DSv4 prefill by @kaiyang-1 in #4575
• Moe a8w4 async scales by @lburzawa in #4564
• Tune MHA config & small-head pipeline pathology by @nidal567 in #4414
• Tune & include fuse-aware gfx950 fused GEMM A8W8 blockscale mul_add by @nidal567 in #4223
• Spatial Attention: XCD-aware spatial workgroup mapping for MHA and GQA (SWIZZLE=1) by @mc186 in #3936
• [Triton] [Gluon] [GFX12] Re-enable FP4 KV cache for UA and MLA by @k50112113 in #4445
• [GFX950] Merge gluon dsv4 attn prefill api entry with pa_prefill_sparse by @leonling-ll in #4576
• [Triton] Enable conv2d Triton kernels for gfx1100 (RDNA3) and gfx1151 (RDNA3.5) by @saeid-rostami in #4265
• [TRITON][GLUON] Prefill MQA Logits kernel tuning for GLM 5.x by @cagrikymk in #4563
• MLA PS mode fp8 -n 16,3 16,4 and nhead=32,64,128 support cp round robin by @minmengdie in #4521
• [module_sample] refactor and detorch by @amd-ruitang3 in #4569
• tune: a8w8 gemm tuning for Qwen3.5 MXFP4-AttnFP8 model by @yuzho-amd in #4396
• fix for mxmoe gemm2 by @Bernard-Liu in #4543
• Temporarily pin rocm/pytorch:latest image digest by @gyohuangxin in #4589
• CI: move multi-GPU tests to DO MI350X runner by @gyohuangxin in #4600
• [OPUS][ATOM] Disable amdgpu-early-inline-all for DSv4 sparse prefill by @kaiyang-1 in #4591
• [fix][moe] Fix shuffle_scale_moe crashes with UnboundLocalError on gfx942 by @yitingw1 in #4588
• [FLYDSL] add MLA reduce decode kernel for gfx942 by @anhminhnguyenhoang in #3901
• Moe reduce extract by @coderfeli in #4596
• Removed block pointers in lean_atten_paged.py and fix large-KV indexing by @nidal567 in #4579
• fix(torch_guard): drop redundant aiter:: prefix from define schema for torch 2.13 by @ZhangLirong-amd in #4593
• Carhuang/fix the rock compile by @valarLip in #4608
• Add unit-scale FP8 KV cache support to fused_qknorm_idxrqknorm by @zijiecode in #4362
• [FlyDSL] Tiled-copy refactor + cleanup: swiglu/silu_fq/qk_norm_rope/fp8_mqa by @coderfeli in #4606
• Fix reduce_scatter pynccl path by @yitingw1 in #4605
• Fix profiler UnicodeDecodeError in aiter add/sigmoid op tests by @fangche123 in #4623
• [TRITON][GLUON][GFX950][DSV4] Paged Sparse Attention Gluon Kernel by @cagrikymk in #4382
• [Bugfix] mp_tuner: default --timeout so a dead worker is reaped instead of hanging by @vanshbhatia-amd in #4413
• [PERF] gfx942 FMoE: a16 kernel dispatch for GLM-5 decode (T=16-256) by @alexioslyrakis-amd in #4558
• [Bugfix] Fix the vector import for FlyDsl by @JadenMathias in #4635
• feat(flydsl): support strided GDN decode state routing by @junna2016 in #4573
• [FlyDSL] Raw-dialect cleanup (others): gdr_decode / chunk_gated_delta_h / pa_mqa_logits_fp4 by @coderfeli in #4609
• MoE: correct + faster a16w4 (bf16 A x MXFP4 W) SiTUv2 kernel (replaces old a16w4) by @coderfeli in #4502
• [module_aiter_unary] refactor and detorch by @amd-ruitang3 in #4590
• [opus_moe] support sorted intermediate layout by @yifehuan in #4586
• MI350 MLA ps mode BF16 case support mask0 by @minmengdie in #4565
• [MLA] Support 96-head 128-dim reduction by @LiuYinfeng01 in #4625
• chore(aot/flydsl): drop per-kernel "[OK] compile" log; silence false-positive mxmoe warning by @zhiding512 in #4611
• [Trition] Add chunk_delta_attn triton kernels by @Liang-jianhao97 in #4568
• tune Kimi-K3 prefill GEMMs for gfx950 by @LiuYinfeng01 in #4479
• [fmoe] Add runtime-keyed Kimi-K3 A8W4 config by @LiuYinfeng01 in #4652
• megamoe by @GwilliamHu in #4439
• fix: fix gpu arch detection by @PerryZhang01 in #4660
• Add lse to fmha asm kernels by @JohnNikolay84 in #4657
• [module_ropes*] refactor and rm-torch by @amd-ruitang3 in #4654
• [Triton/Gluon] Add dytpe dirs to the configs by @Boss2002n in #4613
• [Gluon] fused_mxfp4_quant for gfx1250 by @amd-jrosas in #3093
• [DSV4][Triton] Support block sizes > 1 in paged MQA logits by @skysnow2001 in #4440
• [Triton] unify the loading and add a UT for configs by @Boss2002n in #4630
• remove kpack from 950 configs by @Boss2002n in #4662
• [config] add gfx1250 96-head GEMM shapes to gptoss BF16 tuned config by @akao-amd in #4650
• [feat][HIP]: enable chunk-gated-delta-rule-fwd-h on gfx1201. by @stevenshenyj in #4523
• configs: add DSv4 FP8/FP4 E=385/topk7 inter_dim=384 fused-MoE shape (shared-expert fusion) by @karverma-amd in #4666
• [FLYDSL] mfma16_hip GDR K5 prefill chunk_gdn_fwd_h for MI308 by @huizzhan in #4353
• [HIP] Add gfx950 packed BF16 GDR decode kernel by @zijiecode in #4491
• feat(topk): length-adaptive deterministic top-k for sparse-MLA indexer by @chuanbowang2026 in #4415
• fix(v4): stop turning row and block numbers into 32-bit addresses; add a direct-row SWA write by @valarLip in #4680
• [Bugfix] Fix two build failures: hipify qualifier loss and std::min by @echen4096 in #4639
• [fix][fmoe] propagate flat_mode through fmoe_g1u1 dispatch by @alexioslyrakis-amd in #4675
• moe a8w4: GUGU act+quant fusion by @Boss2002n in #4170
• swap gfx950 kernel to hold 64bit memory addr by @liyjiang in #4578
• [CI] bump dawidd6/action-download-artifact to v21 by @micmelesse in #4684
• [Triton] Fix LDS OOM issue on MI300 by @Liang-jianhao97 in #4671
• Add CODEOWNERS for aiter/ops/triton by @Dewei-Wang-sh in #4678
• gemm_a8w8_blockscale_bpreshuffle: add optional inplace out= buffer by @ZhangLirong-amd in #4674
• Fix causal fmha f8 hd256 kernel, return missing diagonal pairing by @JohnNikolay84 in #4682
• Revert "Add CODEOWNERS for aiter/ops/triton" by @zufayu in #4700
• [FlyDSL]Optimize MoE mxfp4 stage2 for gfx950 by @binding7012 in #4642
• Fix Triton cache flooding in _fwd_kernel_stage2_asm (batch size as constexpr) by @amd-mvarjoka in #4545
• CI: run vLLM disagg from upstream main by @JiaoliangYu in #4619
• [triton] add attn_res kernel for K3 by @yanxuer-999 in #4572
• stage2 logits block load by @yanxuer-999 in #4555
• [FlyDSL] Use native FP4 conversion in MoE stage1 by @XiaobingSuper in #4670
• Fix ASM split-K semaphore deadlock under CUDA graph capture by @JohnQinAMD in #4494
• CI: add FFM Triton test workflow by @gyohuangxin in #3596
• Add opus fp8 mxscale BMM kernels for gfx950 by @yzhou103 in #4320
• Revert "Fix ASM split-K semaphore deadlock under CUDA graph capture" by @amd-ruitang3 in #4709
• Restore RNG seed in test_fused_rms_quant (fix flaky mxfp4 quant CI) by @johannes-graner in #4705
• [TRITON] [GLUON] GFX1250 Gluon MoE A4W4 Kernel by @farlukas in #2513
• [GLUON] Add more fine grained tuning based on M by @lburzawa in #4470
• [TRITON] add m<32 support for gfx1250 mxfp4 by @Boss2002n in #4693
• [MLA] Support 24-head 512-dim reduction by @LiuYinfeng01 in #4710
• Fix NaN in MLA Gluon MTP decode by zeroing fully causal-masked KV spl… by @yanxuer-999 in #4711
• fix: car graph mode err when pytorch set expandable_segments:True by @TennyWang1223 in #4621
• Clean rocprim/hipcub in hip kernels by @junhaha666 in #4594
• FlyDSL: port a16wi4 to new pipeline, clean old moe_gemm_2stage by @coderfeli in #4646
• [module_topk_*] de-torch topk_per_row / topk_plain + externalize workspaces by @amd-ruitang3 in #4702
• [fix] add missing header in topk_plain_kernels.cu by @amd-ruitang3 in #4725
• [MI355] add 8wave pipeline to a8w8 bpreshuffle gemm by @solinzby1 in #4714
• feat: support fp32 chunk states in GDN prefill by @junna2016 in #4366
• Restore default GitHub Pages docs deployment by @gyohuangxin in #4728
• CI: publish per-case vLLM DI accuracy and enable Kimi by @JiaoliangYu in #4722
• Avoid duplicate fp32 output alloc in torch_moe_stage2 reference by @johannes-graner in #4717
• [Triton] Optimize chunk_delta_attn performance. by @Liang-jianhao97 in #4683
• [CI] Skip Triton test suites on docs-only changes by @Boss2002n in #4734
• [Triton] add a copilot review instructions and modify readme file by @Boss2002n in #4720
• [triton] fix sparse decode gathering zeros from a strided cache by @jiacao-amd in #4673
• [opus_moe] Unify A8W4 Stage2 with a runtime-K decode pipeline by @yifehuan in #4723
• Add Opus hd192 hybrid buffer path for large KV (>4GiB). by @fangche123 in #4473
• CI: add always-run Aiter test gate by @gyohuangxin in #4418
• feat: add param for combine quant by @JiaoliangYu in #4746
• [MLA] Support 48-head 128-dim reduction by @LiuYinfeng01 in #4727
• [Perf][FlyDSL] Add gdn_prepare: a fused intra-chunk GDN prefill prepare kernel by @yiijin in #4598
• ci: pin PyTorch test image to rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0 by @gyohuangxin in #4744
• docs: fix ISA kernel optimization guide and example scripts by @mario-ant in #4561
• [fused_qk_norm_rope_cache_quant] remove in-C++ scratch allocation by @amd-ruitang3 in #4724
• [module_mla_metadata*] refactor and remove torch by @amd-ruitang3 in #4729
• CI: auto-update split test FILE_TIMES by @aiter-gh-app[bot] in #4649
• gfx1250 opus gemm splitk fuse by @demonsan in #4246
• [Triton MOE routing] Fix memory access fault surfacing in 0.1.16.post5 -> 0.1.19 update by @fxmarty-amd in #4530
• fused_qk_rope_reshape_and_cache kernel optimizations (gfx950) by @amirumoAMD in #4719
• [triton][moe] Add sigmoid score_mode to the routing top-k by @lijinpei-amd in #4688
• [GFX1250] [Gluon] Add multicast support for MoE a8w4 by @nsusanto in #4562
• [dtype] Map FP8 to torch.float8_e4m3fn on RDNA3 by @skysnow2001 in #4764
• MLA PS mode add 16mx8_32nx1_fp8fp8 opus kernel, performance increases by 3-8% compared with asm kernel by @minmengdie in #4706
• [GFX950] Relocate MLA Gluon kernel and unify decode dispatch by @LiuYinfeng01 in #4450
• fix(opus_gemm): use align instead of alignas after shared on … by @demonsan in #4777
• Fix 32-bit KV block offsets in gluon paged-MQA logits kernel (silent zero logits past 2 GiB) by @zzw09773 in #4774
• [gfx1250][FlyDSL] Update a8w8 gemm tuned config by @aoli26 in #4760
• fix: LL proto ar dispatch by @TennyWang1223 in #4753
• [detorch] remove vestigial py_itfs_common.h includes (non-CK); top_k_per_row now torch-free by @amd-ruitang3 in #4756
• ci: centralize PyTorch test image config by @gyohuangxin in #4750
• [FlyDSL][LLVM] update flydsl version and Adapt flydsl kernels to internal LLVM ROCDL API changes by @jli-melchior in #4436
• CI: auto-update split test FILE_TIMES by @aiter-gh-app[bot] in #4790
• Revert "[GFX950] Relocate MLA Gluon kernel and unify decode dispatch … by @Dewei-Wang-sh in #4792
• [Qwen3.8 MXFP4][Tuning] Add MXFP4 fused-MoE configs for gfx950 (TP8) by @nholmber in #4767
• Add full 16-tier Qwen3-VL FP4 MoE tuned config for gfx950 by @johannes-graner in #4759
• [Gluon][MLA] Drop the batch_size == 1 constraint from the fp8 KV regime by @ZhengGong-amd in #4480
• new MHA v4 entrypoint ; a spectrum of quant & sparse attention kernels by @jcaraban in #4627
• Replace block_ptr in HSTU attn kernel by @scxiao in #4736
• [TRITON] Make RNG deterministic in KV cache unit test by @brunomazzottiamd in #4738
• hd256 fp8 attention perf improvement by @JohnNikolay84 in #4733
• fix(fmoe): retune GLM-5 FP8 decode kernels by @akii96 in #4811
• [module_mla_reduce] detorch reduce.cu + delete dead no_redundant Python port by @amd-ruitang3 in #4796
• Resolve the pybind develop-path lookups once instead of per call by @fangche123 in #4793
• ci: add network diagnostics for test artifact downloads by @gyohuangxin in #4795
• [CI] PR auto tag by @Boss2002n in #4737
• add support for v2 gemm2 a8w8 by @Bernard-Liu in #4655
• [JIT] Use shared runtime GPU architecture detection by @fsx950223 in #4651
• [FlyDSL] fix(flydsl): stabilize SiTUv2 AOT cache keys by @charlieguo1106 in #4707
• topk_gating: support softmax + need_renorm and refactor test by @yzhou103 in #4460
• config(glm5): retune MXFP4 MoE token=8 to a4w4 by @jiaryang in #4798
• [Bugfix] Disable cpp_itfs sampling on ROCm 10 by @fsx950223 in #4754

New Contributors

• @yuzho-amd made their first contribution in #4346
• @Phil-amd made their first contribution in #4402
• @jiacao-amd made their first contribution in #4307
• @jpy794 made their first contribution in #4082
• @a-sidorova made their first contribution in #4044
• @Fangzhou-Ai made their first contribution in #4269
• @peizhang56 made their first contribution in #4474
• @karverma-amd made their first contribution in #4506
• @zijiecode made their first contribution in #4513
• @mc186 made their first contribution in #3936
• @vanshbhatia-amd made their first contribution in #4413
• @JadenMathias made their first contribution in #4635
• @junna2016 made their first contribution in #4573
• @GwilliamHu made their first contribution in #4439
• @amd-jrosas made their first contribution in #3093
• @echen4096 made their first contribution in #4639
• @binding7012 made their first contribution in #4642
• @amd-mvarjoka made their first contribution in #4545
• @johannes-graner made their first contribution in #4705
• @mario-ant made their first contribution in #4561
• @zzw09773 made their first contribution in #4774
• @ZhengGong-amd made their first contribution in #4480
• @jiaryang made their first contribution in #4798

Full Changelog: v0.1.19...v0.1.20