[Tracking] Qwen3.5/Qwen3-Next Optimizations

[hlu1]

Hybrid Liear Attention

• hybrid_linear_attn attention refactor Remove hybrid_linear_attn attention backend and refactor attention registry #10816

GDN Kernels

Not arch specific

• Optimize Triton GDN decode (triton) 1 Optimize GDN decode for Qwen3 Next #17094
• Optimize Triton GDN decode (triton) 2 [Qwen3Next] Optimize fused_sigmoid_gating_delta_rule_update_kernel #18271
• Prefill (cutlass, SM90) and decode (cutedsl, SM90/100) kernels from FlashInfer, no radix cache support by the prefill kernel: feat(gdn): add FlashInfer K-last SSM layout support for GDN prefill and decode for Hopper #18361 Feat/gdn decode pooled flashinfer-ai/flashinfer#2521
• [GDN][Qwen3-Next][Qwen3.5] Fuse fused_gdn_gating and fused_recurrent_gated_delta_rule_update in verify_target #19775
• [GDN] Change Attention State Layout from [N, HV, K, V] to [N, HV, V, K] #20283

SM100

• Prefill in Gluon [Qwen3-Next] Optimize Prefill Kernel, add GDN Gluon kernel and optimize cumsum kernel #17983
• FlashInfer CuteDSL decode kernel [jit-kernel] Add CuTe DSL GDN Decode Kernel #15631
• Another CuteDSL decode kernel [Qwen3-Next] Add cutedsl decode/mtp kernel with transposed ssm_state and prefill gluon kernel for blackwell. #17981
• New SM100 FlashInfer CuteDSL kernel Ameyn/gdn decode cutedsl kernel flashinfer-ai/flashinfer#2498 [NVIDIA] Integrate FlashInfer decode kernel (Blackwell) for Qwen3.5 #19150 [Draft][NVIDIA] Triton GDN to support pretranspose state #19528

MoE

• bf16 cutlass fused MoE for Hopper Add support for bf16 x bf16 cutlass fused MoE #10275
• BF16 TRTLLM MoE for Blackwell [NVIDIA] Enable TRTLLM BF16 MoE on Blackwell GPUs #13798
• NVFP4 TRTLLM GEN MoE for Blackwell [Feat][NVFP4] Enable NVFP4 MoE for Qwen series models (eg. Qwen3-Next) #13761 #13761

Gemm

• Support CuteDSL mm_fp4 backend #18801

Full Attention

• Enable XQA for SM90 and SM120 Enable XQA for SM90 and SM120 #17115
• TRTLLM_MHA backend for Blackwell Allow use of TRTLLM_MHA backend for hybrid attention on Blackwell #11138
• Enable accidentally disabled Flashinfer backend Use Flashinfer for target_verify in GDN model for SM120 #20604
• [Qwen3.5] Set full attn_backend to trtllm_mha on SM100 by default when possible #19030

Disagg

• [Bug][Qwen3.5] Tracking: Qwen3.5 model issues with PD disaggregation #19045
• [Disagg] GPU staging buffer with dynamic ring allocator for heterogeneous TP KV transfer #19890
• feat: WIP NIXL transfer through CPU buffer for better performance with mixed TP sizes #18968

Communications

• Fix regression caused by enabling symmetric memory: Register tensors with symmetric memory for qwen #18643
• Use TRTLLM allreduce fusion for Qwen 3.5 #19889
• [Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api #12787

NVFP4 kv cache

• FP4 KV Cache for SM120 NVFP4 KV Cache for SM120 #18314
• FP4 KV Cache for SM100 [WIP] FP4 KV Cache on B200 #17733

Runtime

• piecewise cuda graph for TRTLLM-GEN backends for Blackwell Add piecewise cuda graph for Qwen3-Next FP8 flashinfer_trtllm moe backend #18184
• [Spec V2] Support specV2 for qwen3next #18808
• On hold due to conflict with radix cache: [GDN/Qwen3-Next] Avoid SSM and conv state copy for speculative decoding - up to 9.47% e2e speedup #12892

Qwen3-Next

• [Qwen3-Next] Enable fused_qkvzba_split_reshape_cat also for prefill #18917
• Huggingface NVFP4 model support [feat] Support nvfp4 quantized model of Qwen3-Next #17627

Qwen3.5 text model

• [Qwen3.5] Enable nvfp4 checkpoint #18937
• [Qwen3.5] Enable MTP spec_v2 and add test for nvidia/Qwen3.5-397B-A17B-NVFP4 #19391
• perf(qwen3_5): replace einops rearrange with torch.flatten in GatedDe… #20386
• [Qwen3.5] Fuse DeltaNet input projections #20074

Qwen3.5 Vit

• ViT: [Feature] Optimizations for Qwen3VL models #18784 Optimizations for Qwen3VL models #18559
• [Feature] Optimizations for JPEG input on NVIDIA GPU #19749
• [Feature] Optimizations for class Qwen3VLMoeVisionModel (Conv3d to Linear) in Qwen3VL #19788

[hlu1]

cc @BBuf @yizhang2077 @hebiao064 @Fridge003

[hebiao064]

Many thanks for organizing the PRs — really appreciate the effort!

[yuan-luo]

#15631

[Swipe4057]

The VRAM leak on Nvidia H100 with the Qwen Next model still persists (specifically during mixed inference requests combined with structured output).

[YAMY1234]

#18917

[vincentzed]

To add:

• GDN Decode & State Copies: FlashInfer decode via #19150 (gated_delta_rule_decode_pretranspose — prefill remains untouched). This unblocks #19528 to completely remove the expensive synchronous prefill-to-decode state copies.
• SpecV2 Lazy Fetch: #12892 removes synchronous CPU-blocking state copies in SpecV2 by fetching SSM/Conv states lazily. Blocker: This currently conflicts with Radix caching mechanics and requires alignment.
• GDN Kernel Fusion (Planned): We plan to execute GDN Elementwise Kernel Fusion, folding the fragmented elementwise ops directly into fused_gdn_gating_kernel. In Fused Sigmoid-Mul-Add Kernel (sigmoid, not silu). #19631
• CuteDSL FP4 GEMMs: #18801 implements CuteDSL for mm_fp4, which should improve over cutlass. [High priority]