Skip to content

AITER v0.1.23

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Sep 04:01
v0.1.23
50da036
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

AITER v0.1.23 - bi-weekly release

Scheduled release from release/v0.1.23.

Diff base: v0.1.22

Wheels

Prebuilt manylinux_2_28 wheels, GPU_ARCHS=gfx942;gfx950:

ROCm Python Wheel
ROCm 7.0 cp310 amd_aiter-0.1.23+rocm7.0.manylinux.2.28-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.0 cp312 amd_aiter-0.1.23+rocm7.0.manylinux.2.28-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.1 cp310 amd_aiter-0.1.23+rocm7.1.manylinux.2.28-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.1 cp312 amd_aiter-0.1.23+rocm7.1.manylinux.2.28-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.2 cp310 amd_aiter-0.1.23+rocm7.2.manylinux.2.28-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
ROCm 7.2 cp312 amd_aiter-0.1.23+rocm7.2.manylinux.2.28-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
What's Changed

Changes

  • [FlyDSL] [CI] add gfx942 FMHA varlen backward kernel for d_qk=192/d_v=128 by @amd-wangfan in #4875
  • [Triton/Gluon] Add per-token-scaled grouped MoE GEMM kernel by @WuLei-AMD in #5338
  • [HIP] Add FP4 output mode to indexer_qk_rope_quant_and_cache by @XiaobingSuper in #5484
  • test(activation): skip fp4 silu_and_mul_quant on non-gfx950 archs by @zjin-lcf in #4925
  • [Config] update gemm stage2 v2 kernel by @charlieguo1106 in #4789
  • [Config] configs: GLM-5.3 PTPC qkv a8w8-bpreshuffle rows for gfx950 by @Raiden-Makoto in #5421
  • [Triton/Gluon] Add MoE weight-gradient kernel by @WuLei-AMD in #5339
  • [FlyDSL] Support runtime softmax scale in gfx950 FP8 FMHA by @gbyu-amd in #5405
  • [FlyDSL] feat(topk): per-row top-k across five selectors behind one entry by @valarLip in #5499
  • [Triton/Gluon] Fix gfx1250 Gluon MLA cache launch args by @amd-ruitang3 in #5501
  • [FlyDSL] flydsl implementation of a16w16 gemm by @omuhamma in #2725
  • [Triton/Gluon] Fix Triton backward autotune dimension keys by @brunomazzottiamd in #5372
  • [Triton/Gluon] [GFX1250] fused_add_rmsnorm_pad() gluon equivalent function by @amd-jrosas in #4146
  • [Triton/Gluon] bf16 persistent gemm by @amirumoAMD in #4860
  • [CK] [FlyDSL] Clean up kernel IR and consolidate shared helpers by @coderfeli in #5476
  • [Triton/Gluon] KDA K1 gluon opt and k2 fused affine by @Liang-jianhao97 in #5436
  • [Triton/Gluon] Fix test_mha_v3 AssertionError on Mismatched elements by @leonling-ll in #5463
  • [FlyDSL] Widen the FlyDSL gather + kv_b_proj backend past a 128+128 head by @XiaobingSuper in #5494
  • [ASM] Tune 16x128, 16x256 mxfp4 kernels by @JohnNikolay84 in #5388
  • [HIP] [FlyDSL] add mla for gfx1250 by @xiangM99 in #4862
  • [CI] Enable ci:extended-test label to dispatch internal workflow by @gyohuangxin in #5522
  • [CI] Reuse existing extended-test dispatch path by @gyohuangxin in #5527
  • [Triton/Gluon] EP Routing fix by @k50112113 in #5511
  • [Config] Extend the DeepSeek-V4 a8w8 blockscale GEMM tunings for gfx950 by @LiuYinfeng01 in #5485
  • [Config] Retune Qwen3-VL MXFP4 MoE without stage-2 reduce kernels by @vorapolsiloai in #5155
  • Capture FCLK in SMI data for micros by @JArnoldAMD in #5509
  • [FlyDSL] fix bw formula by @yadaish in #5528
  • [FlyDSL] topk_select: half formats for k=1, routing re-fit, and a barrier the streaming selector was missing by @valarLip in #5526
  • [Triton/Gluon] Clean up MoE elementwise kernels by @vgokhale in #5259
  • [Triton/Gluon] [Config] gfx942 config by @amirumoAMD in #5516
  • [Triton/Gluon] [Bugfix] Guard RDNA unified attention against LDS overflow by @amd-xavierwang in #4868
  • [Triton/Gluon] [GFX950] Add Return LSE/Softmax to the MHA Gluon Kernel by @lucas-santos-amd in #5458
  • [Triton/Gluon] Fail on checkAllclose mismatches instead of only logging them by @Boss2002n in #5422
  • [Config] Retune Kimi-K3 a8w4 fp8 route-out stage2 by @amd-wsung102 in #5468
  • [OPUS] opus gemm 256tile ring by @demonsan in #5397
  • [HIP] Perf/gfx1250 per group quant by @yzhou103 in #5273
  • [FlyDSL] [gfx1250] Quantize each MoE source token once instead of once per route by @XingerZhu in #5274
  • [HIP] [FlyDSL] feat: add packed BF16 mHC computation and gfx1250 tuning by @junhaha666 in #5412
  • [FlyDSL] [JIT] [gfx1250] Add a8w8 mxfp8_128 GEMM A-preshuffle and fused split-k by @aoli26 in #5406
  • [Triton/Gluon] Add DSV4 sparse-MLA training and indexer ops by @WuLei-AMD in #5491
  • [FlyDSL] Revert " [gfx1250] Quantize each MoE source token once instead of once per route" by @junhaha666 in #5581
  • Expose the fused_moe activation dtype resolution for dispatch dtype selection by @Duyi-Wang in #5221
  • [Triton/Gluon] Add MHC forward and backward support for DSV4 by @WuLei-AMD in #5492
  • fix(fused_moe): pad MXFP4 A4W4 MoE sort extent to a block_size multiple by @zejunchen-zejun in #5573
  • [HIP] [FlyDSL] [Bugfix] Skip invalid expert IDs in MoE sorting by @tuukkjs in #5295
  • [FlyDSL] fix: select FMoE GEMM2 A addressing by stored buffer size by @Bernard-Liu in #5519
  • [Triton/Gluon] fix triton 3.8 regression for rmsnorm, cap blocked BLOCK_SIZE, raise forward occupancy by @nidal567 in #5175
  • [Triton/Gluon] Fix MoE routing kernel compile failure by @micah-wil in #5558
  • [Triton/Gluon] Route test print() output through the aiter logger by @Boss2002n in #5423
  • Add FP8 MoE intermediate option for Kimi-K3 a4w4 by @amd-wsung102 in #5439
  • [Triton/Gluon] no CSV under pytest, no leaked default device by @Boss2002n in #5424
  • [Triton/Gluon] [GFX950] Add split-k support for fp8 mqa logits by @cagrikymk in #5603
  • [Config] [gfx950][DSV4] Tune a8w8 blockscale bpreshuffle shared-expert down/gate_up large-M by @lixiufei-leo in #5440
  • [Config] configs: add tuned configs for Qwen3.8-27B MXFP4 (gfx950) by @vorapolsiloai in #5350
  • [FlyDSL] [Gfx1250][MoE] opt for dsv4 conc2048 decoding mega MoE by @Zzz9990 in #5489
  • [HIP] [OPUS] Unify OPUS GEMM/BMM interfaces and use Torch workspaces by @Fyzyukk in #4961
  • [ASM] [HIP] [gfx1250]asm mha bf16 hd192x128 by @shay-li77 in #5043
  • [Bugfix] Fix ASM split-K semaphore deadlock under CUDA graph capture by @JohnQinAMD in #4916
  • [ASM] [gfx1250] mla v4 prefill: rebuild sparse_pfl with the tail store rework by @junxiaguo in #5580
  • [Triton/Gluon] Take the autotuning search out of the unit tests by @Boss2002n in #5590
  • [Triton/Gluon] [JIT] # RDNA3 fp8 enablement + tuned DSV4 A8W8 block-scale GEMM configs by @amd-xavierwang in #5217
  • [Triton/Gluon] Triton 3.6 FP8 MQA Logits gluon kernel fix by @cagrikymk in #5627
  • [CI] Enable 2P1D cases for vllm DI CI tests and update vllm container to use latest v0.29.0 release version. by @lcskrishna in #5596
  • [Perf] Route untuned a8w8 blockscale GEMMs to triton above a per-arch M by @ZhengGong-amd in #5586
  • [Config] [GEMM] Tune GLM-5.2 native-MTP M=4 projections for gfx950 by @chrisaberger in #5378
  • [CK] Add GLM-5.2 (TP4) a8w8 blockscale GEMM tunings for gfx942 (MI300X/MI325) by @jin-amd in #5072
  • [Bugfix] Prevent resource leaks on repeated hipBLASLt extension initialization by @Mazukiri in #5325
  • [HIP] Fix a missing block_reduce barrier and rewrite indexer_qk_rope_quant_and_cache around the wave by @XiaobingSuper in #5576
  • [CK] [FlyDSL] Add FlyDSL FP8 MoE kernels (decode weight-decompression + prefill) for MI308 — Qwen3.5-35B/397B, Hunyuan3 by @luocheng25 in #3987
  • [FlyDSL] Add flydsl radix topk one block kernel by @lirui927 in #5363
  • [ASM] [HIP] Increase the MLA PS mode BF16 nhead96 perf by @minmengdie in #5582
  • [ASM] [HIP] [OPUS] Add a gfx950 bf16 opus MLA decode path and consolidate the opus decode modules by @kaiyang-1 in #5651
  • [FlyDSL] Bind comm-fused GEMM2 to the sorted-inter layout used by stage2 by @charlieguo1106 in #5622
  • [FlyDSL] fix moe aux kernel ut by @Zzz9990 in #5655
  • [HIP] [OPUS] [JIT] [gfx1250] Add auto-detect stepping and B0-only guard by @dbyoung18 in #5107
  • [CK] [Bugfix] ck_batched_gemm_bf16: derive operand strides from the tensors by @heslami in #5385
  • [CI] Bump triton from 111ff227 to 7cb7b059 by @yuyzhang512 in #5612
  • [Triton/Gluon] Tune PA prefill/decode configs for triton 3.8 backend regres… by @mengfei-jiang in #5613
  • [FlyDSL] [gfx1250] Quantize the MoE a1 scale and shuffle. by @lalala-sh in #5507
  • [HIP] opt dynmic_per_group_scaled_quant by @yzhou103 in #5631
  • [ASM] [gfx1250] avoid padded CSR tail replay in QH128 prefill by @junxiaguo in #5635
  • [HIP] [Bugfix] Give the 4096<n<=6144 RMSNorm bucket a grouped-quant shape by @valarLip in #5653
  • [FlyDSL] perf(hca): the gfx1250 hca_compress optimization by @XingerZhu in #5373
  • [Triton/Gluon] Avoid vectorizing A16W4 upper clamp by @lijinpei-amd in #5636
  • [FlyDSL] [MoE] Raise the stage-2 invalid-row sentinel from 2 GiB to 4 GiB by @Bernard-Liu in #5660
  • [Triton/Gluon] [GFX950] Sparse paged attention and Sparse MLA by @cagrikymk in #4919
  • [Triton/Gluon] [GFX12] Gluon BF16 GEMM config by @k50112113 in #5639
  • [Triton/Gluon] Create benchmark for fused_add_rmsnorm_pad.py by @nidal567 in #5574
  • [Config] Modify kimik3_bf16_tuned_gemm.csv by @hmahmad26 in #5353
  • [CI] Use PR aiter when installing Flash Attention Triton by @gyohuangxin in #5620
  • [Triton/Gluon] [CI] [GFX950] Fix mha_varlen_with_pe miscompile and optimize perf by @leonling-ll in #5629
  • fix(tuned_gemm): honour otype instead of casting a BF16 result to it by @valarLip in #5681
  • [CI] [Build] chore(flydsl): bump flydsl dependency to 0.3.4.1 by @coderfeli in #5457
  • [HIP] [Perf] topk_gating: add E=512 prefill_n dispatch for softmax by @mjkvaak-amd in #5334
  • [CI] [ut] support gfx950 wideEp moe test for a8w4 dsv4,a4w4 kimi k3 by @james-huang09 in #5579
  • [FlyDSL] Enable FP8 FMHA split-K for short cached-prefix queries by @gbyu-amd in #5508
  • [FlyDSL] [Feature] Support FP8 output in gather_kv_b_proj by @gbyu-amd in #5400
  • [HIP] Add register-resident grouped topk for single-group MoE. by @lirui927 in #5632
  • [Config] [Tune] Add GLM-5.3 Flash GEMM configs for gfx950 by @jamesETsmith in #5602
  • [FlyDSL] [gfx1250] TDM dispatch for mega-moe EP by @XingerZhu in #5447
  • [CI] CI: lower Kimi vLLM GPU memory utilization by @gyohuangxin in #5697
  • [Triton/Gluon] [FlyDSL] [Perf] argmax: take the split count as an argument, not a build parameter by @valarLip in #5703
  • [Config] configs: GLM-5.3 routed MoE rows for gfx950 by @Raiden-Makoto in #5599
  • feat(mori-ep): add the low-latency kernels and a cached handle set by @JiaoliangYu in #5493
  • [CI] Re enable contexted kv attention tests by @mengfei-jiang in #5659
  • [HIP] Lift the 2 GiB limit on mla_reduce_v1 final output and LSE by @kaiyang-1 in #5698
  • [FlyDSL] [fused_moe] Do not route MXFP4 MoE to CK-Tile when inter_dim % 256 != 0 by @xiaofei-zheng in #5232
  • CI: run Kimi-K2.5 perf nightly only by @gyohuangxin in #5716
  • [Config] Revert " Modify kimik3_bf16_tuned_gemm.csv" by @XiaobingSuper in #5714
  • [Triton/Gluon] [Config] [Tune][gfx942] Add per-shape GEMM configs for MI308X (gfx942) by @WuLei-AMD in #5691
  • [CI] CI: auto-update split test FILE_TIMES by @aiter-gh-app[bot] in #5713
  • [CK] fix(moe): handle 64-aligned A16W16 shards in CK dispatch by @junna2016 in #5630
  • [FlyDSL] Allocate split-K scratch buffers from a private MemPool by @XiaobingSuper in #5430
  • [OPUS] fix(opus): prevent DSV4 DPA CI OOM with bounded route workspace buckets by @yifehuan in #5710
  • [Triton/Gluon] Log with % placeholders instead of f-strings by @Boss2002n in #5545
  • [Triton/Gluon] Create benchmark for Norm/Layernorm by @nidal567 in #5608
  • [Triton/Gluon] Create benchmark for fused_rmsnorm_add_bench by @nidal567 in #5676
  • [Config] Tune Kimi-K3 BF16 GEMMs for gfx950 by @XiaobingSuper in #5702
  • [FlyDSL] Expose FP8 attention and gather capability checks by @gbyu-amd in #5718
  • [FlyDSL] gfx950 FP8 MQA logits indexer kernel by @vpietila-amd in #4538
  • [HIP] [JIT] Add a DSv4 paged K gather+dequant kernel by @akii96 in #5680
  • Opt qk_norm_rope_group_quant in gfx1250 by @yzhou103 in #5712
  • [Triton/Gluon] [FlyDSL] feat(flydsl): Add HSTU Backward kernel by @SamiAario-AMD in #4885
  • [HIP] [OPUS] [JIT] [GFX950]OPUS PA MQA Logits MXFP4 by @shay-li77 in #5332
  • [FlyDSL] [CI] [a16w4] Support MXFP4 weights on gfx942 (CDNA3) by @MHYangAMD in #5583
  • [Config] [AMD][DSV4] [Tune][gfx950] a8w8 blockscale bpreshuffle configs for DSPARK by @karverma-amd in #5575
  • [HIP] Add DSV4.1 A4W4 fused MoE configs for gfx950 by @yifehuan in #5722
  • [HIP] [OPUS] [JIT] [GFX1250]OPUS PA MQA Logits MXFP4 by @shay-li77 in #5656
  • [FlyDSL] Optimize gfx950 A8W8 preshuffle GEMM shape coverage by @XiaobingSuper in #5690
  • [Triton/Gluon] [FlyDSL] [CI] feat(flydsl): Add paged-attention Tile kernel by @fsx950223 in #4332
  • [CI] Cache Triton wheelhouse across test jobs by @gyohuangxin in #5717
  • [Triton/Gluon] fix: avoid Python 3.10 source inspection failures in MXFP4 JIT imports by @junna2016 in #5572
  • [CI] Fail the Triton test jobs if any kernel autotunes at runtime by @Boss2002n in #5425
  • [Config] Add Qwen3.6 35B-A3B FMoE configs for gfx1201 by @keneoneth in #4815
  • [FlyDSL] QRInt4: INT4 two-shot all-reduce for gfx942/gfx950 by @samremes in #4970
  • [Triton/Gluon] [Kernel] Add gfx950 stochastic MXFP4 quantization by @DaiXindi-AMD in #5531
  • [FlyDSL] gfx942 a16wi4 gemm2 atomic: use fx.atomic_add by @msaffari-amd in #5755
  • [Triton/Gluon] fused KDA decode: order conv_state shifts after the tap loads by @Boss2002n in #5619
  • [ASM] [HIP] [CI] Mixed MXFP6/MXFP4 GEMMs by @ksikiric in #5587
  • [HIP] fix(custom_all_reduce): uniform trip count for the block barriers by @Arist12 in #5694
  • [ASM] [HIP] MLA PS mode asm and opus kernel support nhead12,8 by @minmengdie in #5695
  • [HIP] [Activation] Add standalone relu2 (Squared ReLU) elementwise kernel for non-gated MoE MLPs by @shantipriya-amd in #5689
  • [HIP] Fix grouped topk tie handling for GLM-5.2 accuracy by @lirui927 in #5726
  • [Config] configs: GLM-5.3 fused shared MoE rows for gfx950 by @Raiden-Makoto in #5735
  • fused_moe: remove incorrect topk adjustment in get_2stage_cfgs for EP by @psakhamo in #5734
  • [CI] ci: add @aiter-bot review (self-hosted GLM PR reviewer) by @zufayu in #5666
  • [Triton/Gluon] [Kernel] Add 32x32 block-scaled MXFP4 quantization by @DaiXindi-AMD in #5548
  • [FlyDSL] feat: add gfx1250 natural FP4 compressor scatter by @junhaha666 in #5731
  • [Config] Revert 13 regressed Kimi-K3 A8W8 rows by @XiaobingSuper in #5780
  • [ASM] [HIP] feat(fmha_fwd): add bf16 hd=256 forward ASM kernel for gfx950 by @alexioslyrakis-amd in #5403
  • [ASM] [HIP] feat(fmha_bwd): add bf16 hd=256 fused dKdV+dQ ASM kernels for gfx950 by @alexioslyrakis-amd in #5376
  • [Triton/Gluon] Merge MoE MXFP8 GEMM into existing _moe_gemm_a8w8 kernel by @WuLei-AMD in #5658
  • [ASM][gfx1250] refresh QH128 sparse-prefill and fix softmax mlog order by @junxiaguo in #5746
  • [CK] [FlyDSL] [FEAT] support fp8/fp4 quant in mega stage2 combine by @yanboshao in #5176
  • [Triton/Gluon] [Bugfix] [Perf] Support page size 8 in paged MQA logits on both layouts, and fix large KV offsets by @zhiding512 in #5749
  • [CI] review-pr: fix agent-timeout envelope so large-diff reviews do not spuriously time out by @zufayu in #5785
  • [Triton/Gluon] Drop GRID_MN from the batched MXFP4 GEMM specialisation keys by @eppaneamd in #5672
  • [OPUS] Optimize Opus integration flow by @yifehuan in #5748
  • [ASM] [HIP] [Kernel] [gfx1250] Add F8GEMM 256x256 split-K and 128x128 AP0/AP1 kernels by @WangJessie926 in #5699
  • [ASM] [MLA v4 nm] Swap gfx950 asm decode kernel by @liyjiang in #5195
  • [FlyDSL] [JIT] AOT-compile the gfx950 FP8 flash-attention forward by @gbyu-amd in #5796
  • [HIP] [Perf] Size the MLA metadata LDS scratch to the chunk, not the batch count by @ZhiweiYan-96 in #5724
  • [HIP] [FlyDSL] [JIT] Support Conv3d for Qwen-Image / Wan2.1 VAE on MI35X by @huizzhan in #5370
  • [CK] [Bugfix][Perf] fused_moe: tune the non-temporal load hint per tuned row by @ZhengGong-amd in #5443
  • [HIP] opt inverse rope group quant gfx1250 by @yzhou103 in #4991
  • [Config] Retune DSR1/V3 FP8 fmoe rows on gfx950 by @eky-amd in #5641
  • [Triton/Gluon] Gluon MOE compatibility with upstream triton compiler by @k50112113 in #5740
  • [FlyDSL] [CI] optimization perf of mega stage1 by @yanboshao in #5814
  • [Triton/Gluon] [GFX950] Use a prefill launch config for sparse MLA launches by @cagrikymk in #5797
  • [HIP] [Kernel] Fix group quant dispatch for hidden sizes in (4096, 6144] by @Fangzhou-Ai in #5480
  • fix: custom_all_reduce ci hang by @TennyWang1223 in #5813
  • fix mega ubench by @yanboshao in #5838
  • [Triton/Gluon] [JIT] [Feature] Add tuned native group32 A8W8 Triton GEMM for gfx950 by @valarLip in #5750
  • [Config] [Perf] [gfx950] Add MiniMax-M3-MXFP4 BF16 tuned GEMM configs by @amd-bishwoadhikari in #5758
  • [FlyDSL] optimize mxfp4 moe kernels perf by @binding7012 in #5482
  • [Triton/Gluon] [Config] Add gfx1250 gluon a16w16 config for MLA q_b_proj (N=24576, K=1536) by @karverma-amd in #5828
  • [Triton/Gluon] [gfx1151] Enable unshuffled MXFP4 GEMM by @JeremiahM37 in #5470
  • [Triton/Gluon] [Kernel] Add packed FP4 logical transpose by @DaiXindi-AMD in #5538
  • [Triton/Gluon] drop stale use_aot flag from bench_gemm_afp4wfp4 --shuffle by @matthiasdiener in #5647
  • [Triton/Gluon] [gfx950] [dsv4.1-flash] mHC fused kernel by @ahmed-bsod in #5824
  • [CI] One backend per PR docs update by @Boss2002n in #5853
  • [Triton/Gluon] FlashKDA in-kernel paged state_cache I/O by @Liang-jianhao97 in #5754
  • [Triton/Gluon] Sparse MLA Improvements and Hardened Invalid index Handling by @cagrikymk in #5833
  • [Triton/Gluon] [GFX950] Add Unified Attention Gluon Kernel by @cagrikymk in #4614
  • [CI] copilot instructions update by @Boss2002n in #5856
  • [Triton/Gluon] [Bugfix][MLA] Mask the >2GB global_load path in mla_gluon decode (fixes gfx950 IMA) by @Rohan138 in #5648
  • [HIP] [Perf] Specialize partial tiles in two-stage custom all-reduce by @AndreasKaratzas in #5807
  • [Triton/Gluon] [Config] Gfx1151 blockscale tuned jsons by @keneoneth in #5557
  • [Triton/Gluon] [Config] gfx1151 gemma4 attn prefill tuning by @amd-xavierwang in #5601
  • [CI] Add PRs to the AITER-triton Kernels board without relying on labels by @Boss2002n in #5685
  • [Triton/Gluon] Add paged mxfp4 Gluon kernel with sparsity support and related kernels for vllm integration by @cagrikymk in #5834
  • [Triton/Gluon] Gluon MoE A16W4 gfx950 kernels by @rahulbatra85 in #5355
  • [Bugfix][MLA] Size PS prefill partial buffers from the real token budget by @simondanielsson in #5759

New Contributors

Full Changelog: v0.1.22...v0.1.23

Contributors

vgokhale, matthiasdiener, and 104 other contributors
Assets 8
Loading