[Bug] CUTLASS MoE backend unavailable on SM_120/SM_121 (consumer Blackwell / DGX Spark) for tensor/token-scaled FP8 models

[tgmerritt]

[Bug] CUTLASS MoE backend unavailable on SM_120/SM_121 (consumer Blackwell / DGX Spark) for tensor/token-scaled FP8 models

Summary

On consumer Blackwell hardware (NVIDIA RTX 50 series = SM_120, NVIDIA GB10 / DGX Spark = SM_121), vLLM's FP8 MoE backend selector picks TRITON instead of VLLM_CUTLASS for tensor/token-scaled FP8 models (e.g. RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic). This is correct fallback behavior — but the underlying reason is a real upstream gap, not just a configuration mistake. CUTLASS 4.5 does not ship a CollectiveBuilder specialization for tensor/token-scaled FP8 grouped GEMM on SM_120/SM_121. Investigation on real DGX Spark hardware (2026-05-23) confirms this.

Filing this so other GB10/Spark owners stop wasting cycles rebuilding vLLM trying to enable CUTLASS MoE — the path doesn't exist yet at the CUTLASS layer.

Affected hardware

• NVIDIA GB10 (DGX Spark) → SM_121
• NVIDIA RTX 50 series consumer Blackwell → SM_120
• Any future SM_12x consumer Blackwell variant

Symptom

In server logs at startup:

INFO [fp8.py:405] Using TRITON Fp8 MoE backend out of potential backends: 
[AITER, FLASHINFER_TRTLLM, FLASHINFER_CUTLASS, DEEPGEMM, VLLM_CUTLASS, TRITON, MARLIN, ...]

Despite VLLM_CUTLASS appearing earlier in the priority list, it is never selected on SM_120/SM_121.

Why VLLM_CUTLASS doesn't get picked — precise diagnosis

The selection in vllm/model_executor/layers/fused_moe/oracle/fp8.py walks the priority list and asks each backend's is_supported_config(...). For VLLM_CUTLASS → TritonOrCutlassExperts → CutlassExpertsFp8 → _supports_current_device() returns the result of cutlass_group_gemm_supported().

That function lives in vllm/_custom_ops.py:844:

def cutlass_group_gemm_supported(cuda_device_capability: int) -> bool:
    if cuda_device_capability < 90 or cuda_device_capability >= 110:
        return False
    try:
        return torch.ops._C.cutlass_group_gemm_supported(cuda_device_capability)
    except AttributeError:
        return False

For SM_121 the Python gate currently returns False because 121 >= 110.

Patching the Python gate to allow SM_12x doesn't help — the underlying C++ entry point torch.ops._C.cutlass_group_gemm_supported is not registered in the compiled _C_stable_libtorch extension on SM_12x builds either, because the source csrc/libtorch_stable/quantization/w8a8/cutlass/moe/grouped_mm_c3x_*.cu only has variants for SM_90 (Hopper) and SM_100 (datacenter Blackwell):

$ ls csrc/libtorch_stable/quantization/w8a8/cutlass/moe/
grouped_mm_c3x.cuh
grouped_mm_c3x_sm100.cu
grouped_mm_c3x_sm90.cu

Writing grouped_mm_c3x_sm120.cu is where you hit the actual upstream wall. See "What we tried" below.

What we tried (so others don't have to)

Revived #32237 (closed, needs-rebase). Adapted to the post-libtorch_stable refactor. Build infrastructure (CMake, scaled_mm_entry.cu dispatch, file path) all wired up correctly. Build chain: TORCH_CUDA_ARCH_LIST="12.0a;12.1a;12.1+PTX", ENABLE_CUTLASS_MOE_SM120=1, gencode compute_120a,code=sm_120a, CUDA 13.0.2, CUTLASS 4.5.x (latest at time of testing). Compile attempts:

Kernel schedule tried	Result
KernelPtrArrayTmaWarpSpecialized1SmSm120 (PR #32237's choice)	❌ Symbol does not exist in CUTLASS 4.5 (was a pre-release name)
KernelScheduleSm120Blockwise (from CUTLASS example 87c)	❌ Expects per-block scale factor layouts tuple<LayoutA*, LayoutSFA*> in mainloop. Our model uses tensor/token-scaled FP8 (scaling in epilogue, plain LayoutA*). Could not build a collective for given parameters.
KernelPtrArrayTmaWarpSpecializedCooperativeSm120<N>	❌ Template exists in dispatch_policy.hpp but has zero usages anywhere in CUTLASS (no examples, no tests, no CollectiveBuilder specialization). Compile fails with argument list for class template is missing; supplying a SchedulerPipelineStageCount then hits "Could not build a collective."

The actual upstream gap

CUTLASS 4.5 ships SM_120 grouped GEMM for:

• ✅ NVFP4 (4-bit float) — example 79d_blackwell_geforce_nvfp4_grouped_gemm.cu
• ✅ Blockwise FP8 (per-block scale factors in mainloop) — example 87c_blackwell_geforce_fp8_bf16_grouped_gemm_groupwise.cu
• ✅ MXFP8 blockscaled — test/unit/gemm/device/sm120_blockscaled_tensorop_gemm/

Missing: a CollectiveBuilder specialization for tensor or token-scaled FP8 grouped GEMM on SM_120. The dispatch policies (KernelPtrArrayTmaWarpSpecializedCooperativeSm120, KernelPtrArrayTmaWarpSpecializedPingpongSm120) are defined in include/cutlass/gemm/dispatch_policy.hpp but nothing in CUTLASS itself ever instantiates them — they appear to be placeholders for future work.

This is the same code path that grouped_mm_c3x_sm100.cu (datacenter Blackwell) uses successfully via KernelPtrArrayTmaWarpSpecialized1SmSm100 + PtrArrayTmaWarpSpecialized1Sm epilogue. There is no SM_120 equivalent yet.

Reproduction (validates the gap on real GB10/Spark hardware)

# Inside a vLLM container built for SM_121 with full ARM64 + CUDA 13.0 + CUTLASS 4.5:
python3 -c "
import torch
cap = torch.cuda.get_device_capability(0)
cap_int = cap[0]*10 + cap[1]
print(f'Compute capability: {cap_int}')
try:
    print(torch.ops._C.cutlass_group_gemm_supported(cap_int))
except AttributeError as e:
    print(f'NOT REGISTERED: {e}')
"
# On GB10: prints '121' then 'NOT REGISTERED' — confirms the kernel was never compiled.

What would actually fix this

In order of likelihood / scope:

1. CUTLASS adds a CollectiveBuilder specialization for SM_120 ptr-array FP8 grouped GEMM with tensor/token scaling (i.e. the SM_100 KernelPtrArrayTmaWarpSpecialized*SmSm100 pattern, but for arch::Sm120 and without the 2-CTA cluster variants since consumer Blackwell doesn't have those). This is the right home for the fix.
2. Once (1) lands, vLLM adds grouped_mm_c3x_sm120.cu mirroring grouped_mm_c3x_sm100.cu with arch::Sm120 and the new schedule type, plus the dispatch branch in scaled_mm_entry.cu (we have this work ready locally, can submit a PR the moment CUTLASS ships the specialization).
3. Workaround for users today: stay on the TRITON MoE backend (current behavior; correct).

Workaround / next steps for users on GB10 today

You are already on the right backend (Triton MoE). The "performance might be sub-optimal" warning about device-specific config is real but the backend choice itself is correct. Generating a tuned Triton MoE config for NVIDIA_GB10 via benchmarks/kernels/benchmark_moe.py is a separate, lower-leverage optimization that is possible today (one caveat: the benchmark script as of this writing crashes on Gemma 4 with AttributeError: 'Gemma4TextConfig' object has no attribute 'num_local_experts' — needs config-schema adaptation; separate issue).

Local work product (ready to upstream when CUTLASS ships the missing builder)

Build infrastructure changes that would be reused once the CUTLASS gap closes:

• CMakeLists.txt: third branch after grouped_mm_c3x_sm100 for 12.0a;12.1a arches, defining ENABLE_CUTLASS_MOE_SM120=1 and appending grouped_mm_c3x_sm120.cu to VLLM_STABLE_EXT_SRC
• csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu: extern decl + version_num >= 120 && version_num < 130 dispatch branch (mirrors the existing SM_100 / SM_90 branches)
• csrc/libtorch_stable/quantization/w8a8/cutlass/moe/grouped_mm_c3x_sm120.cu: single-config template skeleton, the only blocker for compilation is the missing CollectiveBuilder specialization upstream

Happy to send the rebased-but-blocked-on-CUTLASS PR if a maintainer would like it cached for whenever CUTLASS catches up.

---

AI assistance disclosure: This investigation was performed with AI (Claude) assistance over a multi-hour session that included reading the failed PR, attempting four rebuild iterations on a physical NVIDIA DGX Spark, and reading CUTLASS upstream source. All compile errors and conclusions are from actual builds on SM_121 hardware. The human submitter (Tyler Merritt) reviewed and validated the diagnosis end-to-end before filing.

[pasta-paul]

Cross-linking from a parallel investigation — canada-quant has been chasing a related SM_120 / Marlin path issue on a different stack (DeepSeek-V4-Flash W4A16-FP8-MTP, also on RTX PRO 6000 Server Edition).

We've isolated a two-stage cascade on the W4A16 Marlin MoE decode path under concurrent thinking-mode generation that you may find useful as third-party corroboration of the SM_120 backend gap:

1. Stage 1: Pre-PR #40923 — JIT-PTX fallback corrupts tokens (14/30 wrong outputs on AIME at c=4 thinking-mode).
2. Stage 2: Post-#40923 (native sm_120a cubins) — 14/30 corruption → 0/30 corruption, but 29/30 requests now crash with CUDA error: an illegal memory access was encountered. Sequential (c=1) and chat-mode are clean.

Our hypothesis (from source review of csrc/moe/marlin_moe_wna16/): the c_tmp FP32 reduce buffer is undersized for moe_block_size==8 decode with small batches (the closed PR #36889 attempted to fix this). The PTX-JIT path silently read from caching-allocator slack; the native sm_120a cubin lays out memory differently and exposes the OOB as a hard fault.

Cross-card control on identical hardware + same vLLM build + same workload — the NVFP4 path is essentially clean (1/30 corrupted at c=4 thinking on the NVFP4-MTP sibling artifact), reinforcing your point that "the NVFP4 path is the only stable consumer-Blackwell MoE backend today."

Full debug log: canada-quant/dsv4-flash-w4a16-fp8-mtp/docs/findings/sm12x_token_corruption_2026_05_24.md. Bench JSONs: benchmarks/rtxpro6000/.

Independent hardware reproductions of the SM_12x MoE backend story:

• This issue (DGX Spark / SM_121, CUTLASS MoE).
• canada-quant (RTX PRO 6000 Server Edition / SM_120, Marlin MoE wna16).
• #26558 (RTX 6000 Pro + GLM-4.6-GPTQ, same crash site).
• #36811 (RTX A6000 + Qwen3.5-A3B, sustained load).

The pattern is that consumer/workstation Blackwell SKUs need both (a) native arch-list cubins (per #40923) and (b) a workspace/c_tmp audit on the wna16 path. Happy to test any candidate patches against canada-quant's artifacts — full repro is publicly accessible.