[Feature]: Add support of Blackwell SM121(DGX Spark)

[yanyunl1991]

🚀 Feature Request

Summary

Request native support for NVIDIA DGX Spark platform running Blackwell (GB10) GPU with CUDA 13.0 and ARM64 architecture.

Environment

• Hardware: NVIDIA DGX Spark with GB10 GPU (Blackwell architecture)
• Architecture: ARM64 (aarch64)
• CUDA Version: 13.0
• PyTorch Version: 2.10.0a0 (from nvcr.io/nvidia/pytorch:25.11-py3)
• vLLM Version: 0.13.0

Current Issue

vLLM 0.13.0 requires torch==2.9.0, but:

1. PyTorch 2.9.0 CUDA wheels are not available for ARM64 - The official PyTorch cu124 index only has x86_64 wheels
2. NVIDIA NGC PyTorch 25.11 image ships with PyTorch 2.10 - This is the only source of ARM64 + CUDA-enabled PyTorch for Blackwell GPUs
3. vLLM's pre-compiled CUDA kernels expect CUDA 12 - Causes libcudart.so.12 errors on CUDA 13 systems

Current Workaround

Using --enforce-eager flag with --no-deps installation:

pip install vllm==0.13.0 --no-deps
pip install [vllm dependencies except torch]
python -m vllm.entrypoints.openai.api_server --model ... --enforce-eagerThis works but disables CUDA graph optimizations, resulting in ~20-30% slower inference.

Requested Features

1. Support PyTorch 2.10.x - Relax the strict torch==2.9.0 requirement
2. Provide CUDA 13 compatible wheels - Pre-compiled vllm-x.x.x+cu130-...-aarch64.whl
3. Native Blackwell (sm_120) support - Without requiring --enforce-eager

Additional Context

DGX Spark is NVIDIA's latest edge AI platform targeting enterprise deployment. Native vLLM support would greatly benefit users deploying LLM inference on this hardware.

Related

• NVIDIA NGC PyTorch container: nvcr.io/nvidia/pytorch:25.11-py3
• Blackwell architecture: Compute Capability 12.0 (sm_120)

Thank you for your consideration!

[yanyunl1991]

@khluu Hi Kevin, could you help support DGX Spark?

[DarkLight1337]

It should be usable if you install from source. At least I got it to work with NVIDIA Thor on CUDA 13.

uv pip install --force-reinstall torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130

python3 use_existing_torch.py
uv pip install -r requirements/build.txt
uv pip install --no-build-isolation -e .

TRITON_PTXAS_PATH="/usr/local/cuda/bin/ptxas" vllm serve <args>

Based on #26791

[DarkLight1337]

Also see #30435 (comment)

[eugr]

vLLM supports CUDA 13 wheels (including aarch64) starting from 0.13.0 and for nightly builds. It's recommended to use uv and create a virtual environment if doing it on the host.

To install release version:

export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//') 
uv pip install -U https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu130-cp38-abi3-manylinux_2_35_aarch64.whl --torch-backend=auto

To install nightly wheels:

uv pip install -U vllm \
        --torch-backend=auto \
        --extra-index-url https://wheels.vllm.ai/nightly/cu130

You can also use my docker build optimized for Spark (especially if you are using it on the cluster), or just use it for reference purposes: https://github.com/eugr/spark-vllm-docker

You may also find NVIDIA community forums useful: https://forums.developer.nvidia.com/c/accelerated-computing/dgx-spark-gb10/dgx-spark-gb10/721

[yanyunl1991]

The key issue it that when I use nvcr.io/nvidia/pytorch:25.11-py3, the latest vllm is not able to use, since it needs pytorch 2.9

[yanyunl1991]

It should be usable if you install from source. At least I got it to work with NVIDIA Thor on CUDA 13.

uv pip install --force-reinstall torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130

python3 use_existing_torch.py
uv pip install -r requirements/build.txt
uv pip install --no-build-isolation -e .

TRITON_PTXAS_PATH="/usr/local/cuda/bin/ptxas" vllm serve <args>

Based on #26791

Do you run into such isuue: FlashAttention requires building with sm version sm80-sm90, but was built for < 8.0! ?

and the LLM's output is wrong.

Summary: vLLM on NVIDIA Blackwell (GB10) with CUDA 13
Environment
GPU: NVIDIA GB10 (Blackwell architecture, sm121 / cuda capability 12.1)
CUDA: 13.0
PyTorch: 2.9.1 (CUDA 13.0)
vLLM: Built from source (main branch)
Model: Qwen/Qwen2.5-7B-Instruct
Issue
vLLM starts successfully but produces completely garbled output.
Observations
Startup Warnings:
Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)
Found GPU0 NVIDIA GB10 which is of cuda capability 12.1. Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)
Attention Backend Fallback:
FlashAttention doesn't support Blackwell (sm120+)
vLLM fell back to TRITON_ATTN (or similar backend)
Example Output (for prompt "Hello"):
搦魔法师和tá."&kontakte萧📐',...
."&rende موقف📐\grid =>$otine
⚗ayette.GetAxis =>$kontakte precio⚗...
搦魔法师和tá."&kontakte萧📐',... ."&rende موقف📐\grid =>$otine ⚗ayette.GetAxis =>$kontakte precio⚗...
Speed: Fast (~4-5 seconds per request), but results are unusable.
Root Cause
The combination of:
PyTorch 2.9.1 not fully supporting cuda capability 12.1
No FlashAttention support for sm120+
Fallback attention backends (TRITON_ATTN) producing incorrect computations on Blackwell
Current Workaround
Using Hugging Face Transformers + NGC PyTorch container instead of vLLM. Slower (~47s/request) but produces correct outputs.
Recommendation
Wait for official FlashAttention/vLLM support for Blackwell architecture before deploying vLLM on GB10.

[DarkLight1337]

No, did you install flash attention from a source outside of vLLM?

[yanyunl1991]

No, vLLM install flash attention failed, then it seems that a different one was used for rollback.
In the end, the service started successfully, but when I use, the returned items were in a mess

[yanyunl1991]

https://forums.developer.nvidia.com/t/run-vllm-in-spark/348862/9. This worked well for me.
@eugr Thank you very much