Skip to content

Release v0.6.18

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 29 Aug 02:01
· 74 commits to main since this release
v0.6.18
69ff11f

These highlights are also published at flashinfer.ai/releases.

v0.6.18 Highlights

This release completes NVIDIA Rubin (SM107) support, brings whole-layer expert-parallel MoE to Hopper, adds decode paths for DeepSeek-V4 sparse attention and Kimi K3 linear attention, and broadens low-precision MoE coverage with MXFP4 on Blackwell RTX PRO and DGX Spark and weight-only W4A16 on B200 and B300.

SM107 (Rubin) support

FlashInfer 0.6.18 completes support for NVIDIA Rubin (SM107), begun in 0.6.16. Rubin devices now dispatch through the same unified APIs as Blackwell — attention, GEMM, MoE, and quantization.

Attention gains trtllm-gen FMHA for SM107, including sparse compression and FP16 softmax. The PrimTS attention path accepts Rubin as well. On the GEMM side, batched and low-latency GEMM both run on Rubin, and CUTLASS NVFP4 SVDQuant is enabled. A CuTe-DSL kernel family specialized for Rubin ships alongside them. trtllm-gen MoE now selects valid tactics on Rubin, and router GEMM and topk_varlen accept SM107.

Expert-parallel MoE runs whole-layer on Hopper

The moe_ep mega-kernel stack was Blackwell-only. Two Hopper FP8 backends, Sm90PullFp8MegaMoeConfig and Sm90PushFp8MegaMoeConfig, now let Hopper deployments run dispatch, FC1, SwiGLU, FC2, and combine as one fused layer behind the existing MoEEpLayer, instead of composing an NCCL all-to-all with a local fused-MoE operator. The pull backend reaches 562 TFLOPS/rank at a 384-expert DeepSeek-class geometry. The push backend supports CUDA Graph capture, and its opt-in fused FC1 epilogue drops an approximately 1 GiB per-rank activation buffer at the DeepSeek-V3 EP8 shape. Its grouped GEMM requires CUDA Toolkit 12.8 or newer.

HCA decode backend and top-K selection for DeepSeek-V4

Sparse attention picks the top-K KV positions per request on every decode step, then attends over the compressed cache; both halves now have dedicated paths. FP8 Heavily Compressed Attention (HCA) arrives for SM100/SM103 via trtllm_batch_decode_sparse_mla_dsv4(..., backend="cute-dsl"), taking arbitrary sliding-window row order including ring rotation and wraparound while keeping the compressed cache paged. The new flashinfer.top_k_varlen handles ragged batches through a Blackwell radix kernel, a guess-verify-refine kernel that warm-starts from the previous step's indices, and a CUTLASS fallback for any GPU. SM120/121 also picks up top-k 192 and 256.

Kimi K3 decode fuses into one Blackwell kernel

flashinfer.fused_kda_decode folds Kimi K3's width-four depthwise causal convolution, SiLU, recurrent Kimi Delta Attention update, and gated RMSNorm into a single SM100 launch, covering the production head_dim 128 shapes at 12, 24, 48, and 96 heads and updating the convolution cache and FP32 state in place. On B200 under CUDA Graphs it is 1.33x the vLLM fused kernel at one row (table geomean 1.13x). A T=1 fast path inside recurrent_kda and shared SM100-family recurrent kernels round out the KDA stack.

MXFP4 MoE and video sparse attention on Blackwell RTX PRO

MXFP4 checkpoints run natively on SM120/121: b12x_fused_moe and B12xMoEWrapper accept quant_mode="mxfp4" across the existing fused schedules including CUDA Graph reuse, the b12x dense GEMM gains the matching path, and on an RTX PRO 6000 Blackwell Server Edition MXFP4 tracks NVFP4 against a strict quantized reference. Video Sparse Attention, until now datacenter-Blackwell only, reaches these parts through a vsa_sm120_blk64 backend on BlockSparseAttentionWrapper. Gemma 4 gains asymmetric VO-split NVFP4 paged prefill on SM120/121.

W4A16 MoE and dense GEMM extend to B200 and B300

Weight-only NVFP4 against BF16 activations, which 0.6.14 shipped for SM12x, now runs on the SM100 family: CuteDslMoEWrapper and cute_dsl_fused_moe_nvfp4 accept quant_mode="w4a16", decoding weights to BF16 inside the kernel so no separate activation-quantization or repack launch is needed, and mm_bf16_fp4 gains a dedicated SM100/SM103 kernel. Consuming BF16 directly pays off where MoE decode is memory-bound: at a DeepSeek EP8 shape on B200, W4A16 is 1.50x the W4A4 baseline at one token; W4A4 still wins large-batch prefill. GeGLU-tanh and SiTU are supported.

Unified MoE API adds shared experts, MXINT4, and CUTLASS runners

Shared experts now work through the unified API rather than low-level kernel entry points, via ExpertConfig.num_fused_shared_experts on the block-FP8 and FP4 runners, completing in the unified API what 0.6.15 and 0.6.17 added for FP8 and FP4. The API also gains MXINT4, CUTLASS BF16 and W4A16 runners on SM90, packed per-tensor FP8 routing, BF16 FromLogits routing, and TopKSigmoid.

Fused MNNVL all-reduce tail for tensor-parallel MoE

allreduce_fusion gains a BF16 Blackwell CuTe-DSL backend for the latency-critical tail of tensor-parallel MoE layers, fusing all-reduce, residual add, and RMSNorm, optionally preceded by MoE finalize and shared-expert add, over MNNVL/NVLink multicast. One backend spans decode to prefill by switching protocol with token count, with initial profiles targeting GB300 on TP8 and TP16.

Smaller JIT-cache wheels; SM75 and single-request FA2 are JIT-only

The flashinfer-jit-cache wheels no longer ship precompiled kernels for SM75 (Turing). Those GPUs still run; the kernels compile on first use. CUDA 13 AArch64 wheels also drop native SM121a cubins (DGX Spark keeps running via SM120 family cubins), and the single-request single_decode_with_kv_cache / single_prefill_with_kv_cache FA2 modules are no longer AOT-prebuilt — those APIs still JIT. Fatbins use size-oriented compression.

0.6.17 0.6.18
cu129 x86_64 1.94 GB 1.02 GB
cu130 x86_64 1.51 GB 1.02 GB
cu130 aarch64 1.69 GB 1.13 GB

What's Changed

  • test(moe): add tests for trtllm-gen fused MoE with GeGLU activation by @Aneureka in #4265
  • perf: optimize trtllm_fmha_v2 fp8 causal attention q-tile scheduling & decoding for uniform seqlen by @akhilg-nv in #3575
  • docs: improve Ulysses communicator and MoE EP docs by @kangbintNV in #4240
  • fix(docker): stop pip from swapping the +cuXXX torch in CI image by @bkryu in #4284
  • fix: skip LogitsTransform on lanes beyond the split-KV chunk boundary in FA2 kernels by @yichengj0 in #3890
  • fix(test): repair main CI regressions from #4280 (artifacts Rubin pins + CuTe-DSL MoE device guard) by @bkryu in #4301
  • fix: preserve DeepSeek no-group sigmoid routing weights by @alexeldeib in #3875
  • fix(gdn): use block-end decay for SM100 state updates by @guangyunh-nv in #4311
  • feat(cake_kda): add optimized B200 recurrent prefill backend by @yyihuang in #4262
  • feat(cake_kda): add optimized B200 recurrent decode backend by @yyihuang in #4279
  • test: Add sharding support to scripts/task_run_unit_tests.sh by @righthandabacus in #4141
  • test(msa_ops): fix stale split-K heuristic expectation on high-SM GPUs by @jimmyzho in #4303
  • perf(moe): sync SM12x NVFP4 fused-MoE kernels to b12x HEAD by @yichengj0 in #4285
  • feat(moe): sync SM12x W4A16 fused MoE family to b12x HEAD by @yichengj0 in #4255
  • fix: support fp8 e5m2 output in rmsnorm_quant and fused_add_rmsnorm_quant by @elwhyjay in #4202
  • feat(moe): add packed per-tensor FP8 and BF16 FromLogits routing to unified MoE API by @feih-nv in #4227
  • perf: remove dead cudaGetDeviceProperties in sm120 groupwise GEMM by @aws-jiadingg in #3523
  • fix(quantization): use 64-bit row addressing in per-token NVFP4 quantizer by @S1ro1 in #4263
  • Fix duplicated words in API documentation by @cupkk in #4194
  • feat(cake_kda): share recurrent prefill kernels across SM100 family by @yyihuang in #4313
  • feat(moe): enable BiasType::Mn (LoRA delta) for nvfp4/mxfp4 MoE by @zetacat in #3987
  • chore: drop unused <nvrtc.h> includes from three launchers by @aleozlx in #4293
  • perf: remove NVFP4 TMA input padding copy by @Barry-Delaney in #4210
  • Fix the expert correction bias checking inconsistency in trtllm_mxint4_block_scale_moe by @b8zhong in #3898
  • Update CODEOWNERS for moe_ep and autotuner sections by @aleozlx in #4332
  • feat(moe): enable MxInt4 in unified API by @feih-nv in #4320
  • feat(cake_kda): share recurrent decode kernels across SM100 family by @yyihuang in #4314
  • [chore] Add jiahanc to gemm code owner by @jiahanc in #4327
  • feat(kda): add fused Kimi K3 decode kernel by @djmmoss in #4243
  • feat(cake_tinygemm2): add CAKE-generated SM100/SM103 tinygemm2 variants with bit-identical outputs by @xslingcn in #4274
  • feat: add sm100 cp delta rule prefill by @guangyunh-nv in #4078
  • fix(cake_kda): support non-aligned recurrent prefill head counts by @yyihuang in #4351
  • Revert "test: Add sharding support to scripts/task_run_unit_tests.sh" by @bkryu in #4344
  • perf(topk): skip output index sort for tie-break selection by @zianglih in #4295
  • Upgrade CuTe DSL FMHA cubins by @xrq-phys in #4291
  • fix(moe): handle CuTe DSL finalize output tails by @S1ro1 in #4186
  • MoE monokernel Bug fix, barrrier remove and kernel rewrite. by @yugong333 in #4027
  • MSA decode path improvements by @rmhaskarnvidia in #4324
  • Add CuTe DSL HCA backend for DeepSeek V4 sparse MLA by @myu-guo in #3943
  • Add @StudyingShao to CODEOWNERS for multiple sections by @aleozlx in #4364
  • fix(moe_ep): let the host framework pin the CUDA device via Bootstrap by @mhoqueanik in #4348
  • [feat] Add Blackwell cutedsl BF16 splitk dense gemm by @jiahanc in #4266
  • Accept unpacked pre-routed topk in fp8 block-scale and bf16 MoE by @b8zhong in #4352
  • fix(comm): keep fd_exchange importable on Python 3.10/3.11 by @nv-guomingz in #4354
  • feat: log git commit hash on import (FLASHINFER_LOGLEVEL >= 1) by @aleozlx in #4286
  • fix(gemm): gate cuDNN out of SM12x bmm_fp8 auto when override_shape unavailable by @Saddss in #4165
  • perf: normalize autotuner nearest-profile cache keys by @wongsingfo in #3984
  • feat(moe): add CuTe-DSL GeGLU-tanh and SiTU activations by @murphymatt in #4009
  • bump version to 0.6.18 by @aleozlx in #4384
  • feat(moe): support BF16 activations in CuTe DSL NVFP4 MoE by @zianglih in #4048
  • docs: document tinygemm2 escape hatch and MonoMoE scratchpad by @kangbintNV in #4381
  • feat: optimize gated SM12x dynamic NVFP4 MoE by @EricChen02 in #4329
  • fix(monomoe): restore CUDA 12.0+ compatibility in tma_load_2d by @aleozlx in #4406
  • feat(moe_ep): SM90 (Hopper) pull-style FP8 mega-MoE backend by @mhoqueanik in #4113
  • feat(topk): Add top_k_varlen with GVR and radix backends for sparse-attention KV selection by @dhiraj113 in #3901
  • feat(sm120): consolidate DSV4 sparse MLA top-k 192/256 support by @lucifer1004 in #4380
  • Fix/quantized allreduce ci harness by @cindyzxq in #4330
  • perf(moe): persist b12x MoE CuTe-DSL kernels to the disk cache by @Smallfu666 in #4331
  • feat(topk): support compact page table transforms by @zianglih in #4315
  • fix(moe_ep): preserve singleton expert TMA modes by @cherichy in #4296
  • feat(moe): add unified CUTLASS BF16 and W4A16 runners by @feih-nv in #4328
  • feat(sm120): add native MXFP4 W4A4 fused MoE by @Yuening-wa in #4290
  • fix(moe): pad BF16 TRTLLM-Gen intermediates to 128 KiB by @nvpohanh in #4319
  • require explicit ci triggering for all pull requests by @yongwww in #4383
  • test incidental support for fp32 unpacked expert weights from #3763 by @jdebache in #4448
  • fix(moe): prepare MXFP8 MXFP4 profiler inputs by @ormandj in #4308
  • Feat/deepseek fused moe fp8 blockwise swigluoai by @vitamin-chaos in #4405
  • feat(comm): add Blackwell MNNVL CuTe DSL all-reduce fusion backend by @qiangyicheng in #4358
  • perf(gdn): optimize cp host launch overhead for sm90 and sm120 by @guangyunh-nv in #4374
  • fix(moe): use per-expert Humming residual scales in SM90 CUTLASS MXFP4xFP8 MoE backend by @StudyingShao in #4431
  • perf(norm): speed up fused add RMSNorm FP4 quantization by @soodoshll in #4416
  • Add task-scheduled Blackwell attention kernels by @PerkzZheng in #4357
  • fix(gemm): validate cached CuTeDSL BF16 tactics by @jiahanc in #4455
  • perf(cake_tinygemm2): add a STAGES=16 kernel tier for single-wave large-K shapes by @xslingcn in #4423
  • perf(moe): optimize CUTLASS fused MoE mem-bound kernels by @Aneureka in #3761
  • remove spot instances from pr-test, use on-demand only by @yongwww in #4454
  • test: Add sharding support to scripts/task_run_unit_tests.sh by @righthandabacus in #4359
  • feat(gdn): u/d cache spec-decode kernels for replayssm by @ameynaik-hub in #4081
  • Mirror nightly release to flashinfer-ai/whl repo by @dierksen in #4353
  • perf(sm120): wave+residue tile-selection for plain groupwise MoE GEMM by @CarstyYou in #4318
  • refactor(moe): enforce build() for all runners and refine CUTLASS autotuning in unified MoE API by @feih-nv in #4376
  • Update Docker CI tags to 20260811-e673f7f by @flashinfer-bot in #4457
  • fix(gdn): convert fp16 decode operands and fix cache identity by @kahyunnam in #4219
  • fix(fmha): propagate LSE reduction launch errors by @leejnau in #4389
  • feat: paged KV cache support for the modular CuTe-DSL Blackwell prefill by @pgera in #4212
  • chore: add KDA CODEOWNERS by @kahyunnam in #4482
  • Fix HCA Gather4 support for public CuTe DSL by @myu-guo in #4368
  • feat(moe_ep): add SM90 push FP8 mega-MoE backend for Hopper by @leonardHONG in #4069
  • fix(comm): align MNNVL two-shot workspace stages by @aoshen02 in #4473
  • Fused shared experts support via the unified API by @feih-nv in #4326
  • feat(attention): asymmetric VO-split NVFP4 paged prefill (qk=512/vo=256) for Gemma-4 on SM120/121 by @jethac in #3684
  • ci: centralize minimum Python version at 3.10 by @dierksen in #4464
  • perf(gemm): split-K, occupancy, and m=1 stream-GEMV decode tactics for mm_bf16_fp4 by @yichengj0 in #4038
  • feat(moe): MXFP8 x MXFP4 CuTe-DSL fused MoE for SM100, plus large-batch routing locality by @vitamin-chaos in #4440
  • Add distribution-aware autotuning for TRT-LLM MoE by @samuel21119 in #4106
  • feat(quantization): cuTile per-token-group 8bit quant + fused RoPE-FP8 by @yifeis-nv in #4019
  • feat(gemm): port SM12x MXFP8 dense GEMM from b12x by @yichengj0 in #4305
  • perf(gdn): reuse pretranspose kernels across pool capacity and stride by @leonardHONG in #4444
  • chore(codeowners): add @jimmyzho to tests/attention/ by @jimmyzho in #4508
  • skip the deliberate-trap test under compute-sanitizer by @qsang-nv in #4493
  • feat(msa): fp8 index-k and index-q support in the SM12x proxy-score kernels by @yichengj0 in #4345
  • Fix TRTLLM ragged prefill edge cases by @alexeldeib in #3779
  • Support non-interleaved KV input to fmha_v2 by @akhilg-nv in #4124
  • feat(attention): cuDNN paged prefill via direct mixed-form cu_seq_lens by @egilliam-nv in #4222
  • perf(gdn): reuse MTP decode kernels across cache modes by @hebo1221 in #4128
  • ci: add PR API and documentation checks by @kangbintNV in #3917
  • Add SM120 FP8 FMHAv2 self-attention by @rosenrodt in #4272
  • feat(kda): add packed-input CuTe decode kernel by @ameynaik-hub in #4417
  • chore: explicitly set mMultiCtasKvMode in ragged attention launcher by @namgyu-youn in #3469
  • integrate SM120 VSA (Video Sparse Attention) block-sparse backend by @hsr1234563 in #4259
  • ci: upgrade sccache and retain stats by @dierksen in #4505
  • fix: correct trtllm-gen reduction indexing for FMHA decode at long q_len by @jhaotingc in #4382
  • ci: upload GitHub unit-test JUnit reports by @dierksen in #4488
  • perf(moe): fold the SiTU beta reciprocal in the CuTe DSL MoE epilogue by @sychen52 in #4506
  • feat: unpacked FP8 per-tensor scaling support for TRTLLM fused MoE by @jdebache in #4478
  • refactor(moe_ep): taxonomy/provenance restructure; incorporate SM90 push-style FP8 backend; sync CuTe-DSL 4.7 quant-staging fix by @mhoqueanik in #4449
  • gemm: enable CUTLASS NVFP4 SVDQuant on SM107 by @Vinnie6167 in #4509
  • ci: grant @flashinfer-bot access to collaborators who can apply labels by @yongwww in #4510
  • feat: support FP8 KV NoPE MLA on SM90 by @JustinTong0323 in #4373
  • fix(ci): correct documentation finding locations and skipped runs by @kangbintNV in #4520
  • test(gdn): trim GDN decode test matrix to distinct kernel specializations by @kahyunnam in #4513
  • feat: add BF16Q FP8KV transform-mode selection by @djmmoss in #3544
  • [prims-ts] Refactor FMHA decode padding-task scheduling by @IwakuraRein in #4413
  • feat(moe): support GeGLU and SiTU in W4A16 by @zianglih in #4394
  • feat(cake_kda): add strided prefill state checkpoints and packed decode by @yyihuang in #4445
  • WIP jit-cache wheel size fix: drop 12.1a from the cu129 aarch64 jit-cache wheel by @kahyunnam in #4527
  • docs: include top_k in topk API docs by @kangbintNV in #4501
  • Fix/missing source tree files by @cindyzxq in #4517
  • ci: install quack-kernels with --no-deps so it cannot downgrade CuTe DSL by @kahyunnam in #4556
  • test: make heavy attention test files more efficient (bulk JIT precompile + faster references) by @bkryu in #4511
  • fix(moe): release W4A16 CuTe DSL scratch workspaces by @zianglih in #4558
  • [fix] disable tileN=192 for trtllmgen per-token NVFP4 MoE by @jiahanc in #4563
  • feat: sm100 cute_dsl w4a16 gemm by @IwakuraRein in #4466
  • feat(b12x): support MXFP4 dense GEMM on SM120 by @ayrnb in #4479
  • feat: Grouped-token MLA support for the TRTLLM-Gen FMHA backend. by @farazkh80 in #3849
  • fix(build): restore the nixl v1.3.1 submodule pin accidentally rolled back in #3759 by @mhoqueanik in #4530
  • Fix/missing source tree files for two more moe_ep test files by @cindyzxq in #4580
  • fix(xqa): out-of-bounds read of attention sinks in SM90 kernel epilogue by @bkryu in #4525
  • feat(moe): add TopKSigmoid routing method (TopK -> Sigmoid) by @EdalatiAli in #4404
  • feat(xqa): support non-MLA head_dim 512 (Gemma-style GQA decode) on SM12x by @bkryu in #4570
  • perf(moe): fold W4A16 SiTU reciprocals by @zianglih in #4540
  • fix: guard oversized shared memory driver enums with CUDA >= 13.4 by @JiaxuanBai in #4377
  • [MoE] Expose zero-copy MegaMoE workspace output view by @foraxe in #4341
  • refactor(moe): split finalize knobs out of ExecutionConfig into MoEFinalizeConfig by @aleozlx in #4385
  • feat(cake_kda): add optimized H12 packed decode across SM100 family by @yyihuang in #4562
  • ci: use the CI image's packages and share a pip cache between jobs by @yongwww in #4592
  • stabilize pytest node IDs for external scopes to fix issues 4499 and 4500 by @righthandabacus in #4538
  • fix(gemm,moe): restore bmm_fp8 auto fallback and drop an over-strict SM107 tactic guard by @Vinnie6167 in #4645
  • fix: align the nvidia-cutlass-dsl floor on release-v0.6.18 with main (>=4.6.2a0) by @kahyunnam in #4715
  • ci: pin nvidia-cutlass-dsl 4.6.2 on release-v0.6.18 by @kahyunnam in #4758
  • ci: keep Rubin cute-dsl pins after the 4.6.2 CI pin by @kahyunnam in #4762
  • Cherry-pick unit, wheel, infra, and Rubin follow-ups onto release-v0.6.18 by @kahyunnam in #4764
  • fix: skip SM107 low-latency GEMM cubins on Blackwell (#4773) by @kahyunnam in #4786
  • fix(moe): filter the trtllm-gen kernel manifest per arch to restore MoE JIT build time by @jimmyzho in #4789
  • [release-v0.6.18] Rubin (SM107) open-issue fixes by @Vinnie6167 in #4790
  • [release-v0.6.18] Five SM107 (Rubin) fixes: 133 CI failures by @Vinnie6167 in #4792

New Contributors

Full Changelog: v0.6.17rc5...v0.6.18

Contributors

ormandj, jethac, and 74 other contributors
Assets 11
Loading