[Usage]: Help Running NVFP4 model on 2x DGX Spark with vLLM + Ray (multi-node)

[letsrock85]

Your current environment

Help: Running NVFP4 model on 2x DGX Spark with vLLM + Ray (multi-node)

Hardware

• 2x DGX Spark (GB10 GPU each, sm_121a / compute capability 12.1)
• Connected via 200GbE ConnectX-7/Ethernet
• Driver: 580.95.05, Host CUDA: 13.0

Goal

Run lukealonso/GLM-4.6-NVFP4 (357B MoE model, NVFP4 quantization) across both nodes using vLLM with Ray distributed backend.

What I've Tried

1. nvcr.io/nvidia/vllm:25.11-py3 (NGC)

• vLLM 0.11.0
• Error: FlashInfer kernels unavailable for ModelOptNvFp4FusedMoE on current platform
• NVFP4 requires vLLM 0.12.0+

2. vllm/vllm-openai:nightly-aarch64 (vLLM 0.11.2.dev575)

• With VLLM_USE_FLASHINFER_MOE_FP4=1
• Error: ptxas fatal: Value 'sm_121a' is not defined for option 'gpu-name'
• Triton's bundled ptxas 12.8 doesn't support GB10

3. vllm/vllm-openai:v0.12.0-aarch64 (vLLM 0.12.0)

• Fixed ptxas with symlink: ln -sf /usr/local/cuda/bin/ptxas /usr/local/lib/python3.12/dist-packages/triton/backends/nvidia/bin/ptxas
• Triton compilation passes ✅
• Error: RuntimeError: [FP4 gemm Runner] Failed to run cutlass FP4 gemm on sm120. Error: Error Internal

4. Tried both parallelism modes:

• --tensor-parallel-size 2 → same CUTLASS error
• --pipeline-parallel-size 2 → same CUTLASS error

5. --enforce-eager flag

• Not fully tested yet

Environment Details

Component	Version
Host Driver	580.95.05
Host CUDA	13.0
Container CUDA	12.9
Container ptxas	12.9.86 (supports sm_121a ✅)
Triton bundled ptxas	12.8 (NO sm_121a ❌)
PyTorch	2.9.0+cu129

The Blocking Error

vLLM correctly loads weights (41/41 shards), then during profile_run:

INFO [flashinfer_utils.py:289] Flashinfer TRTLLM MOE backend is only supported on SM100 and later, using CUTLASS backend instead
INFO [modelopt.py:1142] Using FlashInfer CUTLASS kernels for ModelOptNvFp4FusedMoE.
...
RuntimeError: [FP4 gemm Runner] Failed to run cutlass FP4 gemm on sm120. Error: Error Internal

FlashInfer detects GB10 is not SM100 (B200), falls back to CUTLASS - but CUTLASS FP4 also fails.

Key Question

Are CUTLASS FP4 GEMM kernels compiled for GB10 (sm_121a)?

Is there:

1. A vLLM build with CUTLASS kernels for sm_121?
2. A way to force Marlin FP4 fallback on GB10?
3. Recommended Docker image for DGX Spark + NVFP4?

I see NVFP4 models tested on:

• B200 (sm_100) ✅
• H100/A100 with Marlin FP4 fallback ✅

But GB10 is sm_121 (Blackwell desktop/workstation variant). The error says sm120 which seems wrong - GB10 should be sm_121a.

References

• GLM-4.6-NVFP4(https://huggingface.co/lukealonso/GLM-4.6-NVFP4)

• Firworks/GLM-4.5-Air-nvfp4

Thanks!

[AdamLouly]

Try using nvidia's docker image, https://build.nvidia.com/spark/vllm/instructions

[jasonacox]

@AdamLouly currently the Nvidia builds only supports up to vLLM 0.11.0 as @letsrock85 noted above (also see build notes). I would love to get NVFP4 working on the GB10.

[CuriousDolphin]

@letsrock85 try this https://github.com/eugr/spark-vllm-docker

it's a very great work of eugr:

https://forums.developer.nvidia.com/t/run-vllm-in-spark/348862/115

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[eelbaz]

I've been debugging the exact same stack (DGX Spark, GB10, NVFP4 MoE, vLLM 0.12.x+) for the better part of a year. A few things that might help:

1. The sm120 vs sm_121 carveout mismatch is the root cause — GB10 has 99 KiB smem per block (SM121) but CUTLASS kernels compiled for compute_120f assume 228 KiB (SM120 family max). Build CUTLASS from main which has the StageCountAutoCarveout fix, or use a recent CUTLASS 4.4.2+ that addresses it.

2. Workaround that works today: we got stable 8× DGX Spark TP=8 inference running with --enforce-eager + VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM=1. Slower but production-viable for interactive use.

3. Triton ptxas symlink (you probably already found this):

   ln -sf /usr/local/cuda/bin/ptxas /usr/local/lib/python3.12/dist-packages/triton/backends/nvidia/bin/ptxas

We run a 33-model PRISM NVFP4 family (including GLM-4.6-NVFP4) on GB10 clusters — happy to jump on a screen-share if you want to get this fully sorted. Our Docker setup and config are in our dgx-spark-vllm-setup repo.