Nightly Release v0.6.18-20260819

Automated nightly build for version 0.6.18 (dev20260819)

Nightly Release v0.6.18-20260818

Automated nightly build for version 0.6.18 (dev20260818)

Nightly Release v0.6.18-20260817

Automated nightly build for version 0.6.18 (dev20260817)

Nightly Release v0.6.18-20260816

Automated nightly build for version 0.6.18 (dev20260816)

Nightly Release v0.6.18-20260814

Automated nightly build for version 0.6.18 (dev20260814)

Nightly Release v0.6.18-20260813

Automated nightly build for version 0.6.18 (dev20260813)

Release v0.6.17

These highlights are also published at flashinfer.ai/releases.

v0.6.17 Highlights

This release brings MoE expert parallelism into serving engines, refreshes the Blackwell SM12x fused-MoE kernels with an FP4 accuracy fix, extends the unified MoE API to MXFP4 W4A8/W4A16 and shared experts, and adds decode coverage for Kimi K3 MLA and MiniMax-M3 sparse attention on vLLM.

MoE expert parallelism production-ready in vLLM

The MegaMoE path in flashinfer.moe_ep is ready for serving engines: full CUDA-graph capture and replay, a fused single-launch quantize-and-stage hot path, prequantized weight packs, symmetric-buffer workspaces pooled across layers, and a persistent knob cache that resolves tuned knobs by lookup, so production sessions run with no in-engine autotuning. An opt-in fault-tolerance rank mask, available over both NCCL-EP and NIXL-EP, masks and skips a peer that times out during dispatch or combine, keeping the job alive when a rank dies. Deployment gets simpler too: the CuTe-DSL runtime floor returns to 4.5.2, the version vLLM 0.25.1 pins, and a new BootstrapConfig.device lets the host framework pin each worker's CUDA device.

• #4079
• #4183
• #4101
• #4348

Blackwell SM12x fused MoE refreshed, with an FP4 accuracy fix

W4A4 serving on DGX Spark and RTX PRO parts (GB10, SM120/SM121) now delivers the output quality its benchmark scores imply, with two NVFP4 quantization bugs fixed and a new input_global_scale that lets integrators pass a checkpoint's weight scale directly. The SM12x fused-MoE families are synced to current b12x: the NVFP4 W4A4 backend reaches kernel parity across decode and prefill, and the W4A16 family adds cooperative persistent launches, a tensor-core decode path for small batches, and shape-stable route packing that keeps decode batch-size changes recompile-free.

• #3932
• #4285
• #4255
• #4253
• #4130

Unified MoE API adds MXFP4 W4A8 and W4A16, shared experts, and SiTU

TRTLLM-gen MXFP4 weights now run through the unified MoELayer API against both MXFP8 activations (W4A8) and BF16 activations (W4A16), completing the FP8 series begun in 0.6.16, alongside per-tensor routed FP8. Routing adds an unpacked pre-routed FP4 mode that accepts topk_ids and topk_weights as separate contiguous tensors. TRTLLM-gen FP4 MoE also gains shared-expert fusion, extending to FP4 what 0.6.15 added for FP8, plus SiTU activation for MXFP4 x MXFP8 and NVFP4 x NVFP4.

• #4159
• #4088
• #4104
• #4239
• #4180

Kimi K3 MLA decode, and MiniMax-M3 sparse attention under vLLM

Blackwell decode now covers Kimi K3's MLA geometry — 96 global query heads against one KV head, TP-local head counts down to 6, speculative query lengths up to 8, and context parallelism for long contexts — by packing query-token and query-head rows into shared CuTe-DSL tiles, adding compact variable-length Q, and extending TRTLLM-gen dense and sparse MLA to non-power-of-two head counts. Sparse MLA decode also serves the no-rotary-tail shape (kv_lora_rank=512, qk_rope_head_dim=0) natively. MiniMax Sparse Attention accepts vLLM's packed paged KV layout for MiniMax-M3, unblocking the vLLM integration on SM120/SM121, with lower per-call overhead on paged decode.

• #4178
• #4108
• #4039
• #4324

Ulysses sequence parallelism for long-context and video diffusion

Ulysses sequence parallelism gets its head-scatter / sequence-gather all-to-all as a public API, for video diffusion transformers and other long-sequence attention workloads. A fused NVLink-P2P kernel folds the layout permutation directly into the cross-GPU writes over CUDA IPC for a single coalesced push, with automatic NCCL fallback when P2P is unavailable.

• #3820
• #4240

What's Changed

• feat: close feature gap by wiring up per-tensor routed FP8 fused-moe by @jdebache in #4088
• Revert PR 4122 by @jimmyzho in #4171
• [GDN] improve sm100 GDN performance by @Observer007 in #4133
• fix(gdn): support WY decode on SM121 by @kahyunnam in #4117
• fix(norm): convert float2 to e4m3 directly in packed cast by @elwhyjay in #4167
• perf(test): bulk-precompile XQA decode kernels to cut test wall time ~4x by @bkryu in #4119
• [perf] Optimize TRT-LLM routing for high-expert, high-top-k workloads by @jiahanc in #4152
• feat(xqa): ragged Q and per-row sliding-window masking for speculative decode by @yichengj0 in #4137
• test(jit): assert BMM export symlink under GEN_SRC_DIR by @kahyunnam in #4187
• Feat/ulysses p2p a2a by @forrestl111 in #3820
• feat(moe_ep): MegaMoE framework integration ready: CUDA graph support, fused quant+stage launch, persistent knob cache, and prequantized weight packs by @mhoqueanik in #4079
• docs: document CuTe prefill scheduling override by @kangbintNV in #4162
• docs: add missing trtllm_fp8_per_tensor_scale_routed_moe API entry by @kangbintNV in #4175
• docs(mamba): document checkpointing varlen arguments by @hebo1221 in #4129
• Yanqinz/fix-gemm-and-grouped-mm-test-issue by @yanqinz2 in #4185
• feat(comm): extend trtllm_allreduce to SM12x and fix lamport buffer pointer packing by @yichengj0 in #3903
• feat(moe): add unified unpacked pre-routed FP4 mode by @feih-nv in #4104
• feat(mla): support packed low-head and variable-Q decode by @PerkzZheng in #4178
• bump version to 0.6.16 by @jimmyzho in #4142
• [fix]fix xqa flaky test on spark by @qsang-nv in #4161
• fix: make mxfp8 gemm test pass by having it quantize along the correct dimension by @jdebache in #3882
• fix(moe): serialize CuTe DSL autotune replay by @zianglih in #4192
• feat(moe_ep): fault-tolerance rank mask (NCCL-EP + NIXL-EP) by @Anerudhan in #4183
• fix(xqa): fix PDL load ordering and SM90 fp8 draft-mask dispatch by @yichengj0 in #4199
• feat(msa): accept K/V views split from a packed paged KV cache by @yichengj0 in #4039
• feat(moe): add TRTLLM MXFP4 W4A8 and W4A16 unified API support by @feih-nv in #4159
• [cli] add CLI helper for flashinfer-jit-cache and flashinfer-cubin wheel installs by @dierksen in #3142
• [feat] Add SITU trtllmgen MOE by @jiahanc in #4180
• feat(sm120): fused MoE (SwiGLU) via moe_gemm is_gated for cute SM120 groupwise GEMM by @CarstyYou in #4130
• fix(moe): pad trtllm-gen route map by one element to avoid OOB read by @syuoni in #4237
• perf(moe_ep): CuTe-DSL 4.5.2 mainloop WAR — drop the 4.6.1 runtime floor by @mhoqueanik in #4101
• Fix the routing inconsistency for num_groups > 1 by @b8zhong in #3946
• fix: support host global scale in CuTe-DSL NVFP4 quantization by @akurathiswaraj in #4138
• fix/test(moe_ep): self-bootstrap 1-rank process group in dg mega oracle test by @mhoqueanik in #4221
• feat: support native qk_rope_head_dim=0 sparse MLA decode in trtllm-gen by @JustinTong0323 in #4108
• feat(topk): support separate page table row starts by @zianglih in #4169
• feat(comm): make mixed-comm VMM workspaces checkpointable by @galletas1712 in #3910
• SM 107 Reland + Merge Back from v0.6.16 Release Branch by @Vinnie6167 in #4280
• test(sm103): fix FP4 autotuner cache inspection by @tiffany940107 in #4145
• bump version to 0.6.17 by @aleozlx in #4283
• feat(gemm): ...

Nightly Release v0.6.18-20260811

Automated nightly build for version 0.6.18 (dev20260811)

Release v0.6.16.post4

v0.6.16.post4

Restores import flashinfer.comm on Python 3.10 and 3.11. A type annotation in
flashinfer/comm/fd_exchange.py evaluated only on Python 3.12+, and flashinfer.comm
imports that module at import time, so the package failed to import on interpreters
inside the supported range. Downstream packages that touch flashinfer.comm during
their own initialization were affected as well.

Upgrade to this release if you run FlashInfer on Python 3.10 or 3.11.

• #4354

Full Changelog: v0.6.16.post3...v0.6.16.post4

Nightly Release v0.6.18-20260810

Automated nightly build for version 0.6.18 (dev20260810)