DeepSeek-V4-Flash on 2× DGX Spark (GB10, SM121), tensor-parallel

Running DeepSeek-V4-Flash-0731 across two DGX Sparks with --tensor-parallel-size 2, speculative decoding active, on a current vLLM tree.

This is a reproduction recipe and a measurement record, not a vLLM fork. The distinction is deliberate: nearly everything needed already exists upstream or in an open pull request, and our own delta is one commit touching two files. Calling that a "patch series" would oversell it.

What is actually ours

patches/0001-*.patch	DeepGEMM pin. 4 lines added, 2 removed, 2 files. Filed upstream.
scripts/	build, two-node Ray bring-up, serve, hardware telemetry
bench/	benchmark harness, including per-series speculation acceptance
results/	raw JSON draws, not just summary numbers

The rest comes from vLLM main and from PR #51538 (lucifer1004), open at the time of writing. Without that PR, none of this runs.

The stack that works

vLLM            main + PR #51538 (db39e675)   ← not merged upstream
DeepGEMM        deepseek-ai/DeepGEMM a6b593d  ← our patch; upstream pin lacks SM120
flashinfer      0.6.18.dev20260811
torch           2.13.0+cu132
CUDA            13.2 (driver 595.84)
hardware        2× NVIDIA GB10, SM121, aarch64, 128 GB unified each
interconnect    direct QSFP56, RoCEv2, dual HCA, NCCL_IB_MERGE_NICS=1

The one real trap: DeepGEMM

DeepSeek-V4-Flash-0731 is configured with hc_mult=4, so every forward pass goes through DeepGEMM's hyperconnection kernel. The revision vLLM pins (vllm-project/DeepGEMM @ e21c821) ships no sm120_* kernel at all, and its dispatch in csrc/apis/hyperconnection.hpp handles arch_major 9 and 10 only. Family-12 Blackwell falls straight through to:

DG_HOST_UNREACHABLE("Unsupported architecture");

Every SM120/SM121 part is affected, RTX 5090 and RTX PRO 6000 included, not just the Spark. deepseek-ai/DeepGEMM @ a6b593d carries the four sm120_* kernels and the arch_major == 12 branch.

Two repositories are named DeepGEMM. That is the whole bug, and it cost us an evening.

The second trap: FlashInfer must postdate PR #4380

The top_k the SM120 sparse-MLA decode dispatch requires is not the model's index_topk. It comes from get_dspark_swa_index_width (vllm/v1/attention/backends/mla/compressor_utils.py):

top_k = ceil((sliding_window + num_speculative_tokens) / 64) * 64

This checkpoint has sliding_window=128, so every k from 1 to 64 requires the 192 specialization and only k=0 stays at 128. Stable flashinfer 0.6.16 and 0.6.17 ship 128, 512 and 1024 only; 192 and 256 arrived with flashinfer PR #4380, merged 2026-08-08.

Any DSpark speculation at all therefore needs a flashinfer build containing #4380. vLLM itself still pins flashinfer-python==0.6.15.post1, so a clean install will not have it. We checked by downgrading to 0.6.17 and reading the failure:

RuntimeError: FLASHINFER_MLA_SPARSE_DSV4 on SM120 requires a FlashInfer DSV4 sparse
MLA decode specialization for (num_q_heads=32, top_k=192).
Install a FlashInfer build containing flashinfer-ai/flashinfer#4380.

Caveat we keep attached to this patch: the two hyperconnection kernels shared by both trees are byte-identical, but other kernels have diverged (sm90_bf16_gemm.hpp differs), so the fork carries vLLM-specific work that a blanket repoint would drop. The cleaner upstream fix is probably to port the sm120_* kernels into the fork. We have no SM90/SM100 hardware and have not tested those paths.

Results

Single stream, k=5, 5 draws per row, acceptance read from /metrics deltas:

series	tok/s	acceptance	mean accepted length
code, reasoning off	72.77	90.1 %	5.51
code, reasoning on	49.45	54.1 %	3.71
prose, reasoning off	43.31	44.6 %	3.23
prose, reasoning on	37.27	34.8 %	2.74

Long context, isolated request, 320 output tokens: 73–78 tok/s flat from 4 k to 131 k input tokens, acceptance 95.6–100 %.

All figures above were taken at --max-num-seqs 2. See the warning below before raising it.

A silent correctness bug: CUDA graphs above 2 sequences

Raising --max-num-seqs with CUDA graphs enabled produces degenerate output on this stack. The server starts, serves fast, reports healthy speculation — and the model answers nonsense (1. 2. 3. 4. …) to any prompt.

Bisected, one variable at a time, asking 17 x 23 and checking for 391:

max_num_seqs	cudagraph_mode	output
2	FULL_AND_PIECEWISE	correct
4, 8, 10, 12, 16	FULL_AND_PIECEWISE	degenerate
16	PIECEWISE	degenerate
16	NONE	correct
2	any	correct

So it is the graph capture, not the concurrency: sixteen sequences are fine as soon as the graphs are off. max_num_batched_tokens is not involved — 8192 is correct at max_num_seqs=2.

With cudagraph_mode: NONE and 16 sequences: 269 tok/s aggregate across 16 streams, 55 tok/s single-stream, output verified coherent.

We initially published 300 tok/s at 16 streams here. That measurement was taken with graphs on and was counting degenerate generation — throughput of nonsense is still nonsense. The lesson we took from it: read an output after every configuration change, never trust the token rate alone.

The result we did not expect

Divide throughput by mean accepted length:

72.77 / 5.51 = 13.22    49.45 / 3.71 = 13.34
43.31 / 3.23 = 13.42    37.27 / 2.74 = 13.60      (+ 13.62 and 13.58 on two more runs)

(ratios computed at full precision from the stored draws, not from the rounded values shown here)

The engine performs about 13.5 verify steps per second no matter what it is writing. Six independent measurements span 13.22 to 13.62, a spread of 3 %. Throughput varies by a factor of 1.95 across those series, mean accepted length by 2.01. It is the same quantity.

The decode step costs roughly 74 ms, flat, and everything else follows from how many tokens each step yields:

• the structural ceiling here is steps/s × (k+1) ≈ 80 tok/s at k=5, and code already reaches 72.77;
• single-stream figures above that need a larger k, not faster silicon;
• reasoning is expensive because it lowers acceptance, 90.1 % → 54.1 % on code, not because any step gets slower.

The hardware telemetry agrees. 95 % reported SM utilisation at about 39 W, no throttling of any kind (the SW Power Capping counter advanced 0.0 s over 45.8 s of generation, thermals sat at 56–60 °C), and RoCE traffic at 1.1 Gb/s on a 200 Gb/s link. A step whose cost does not depend on content, drawing that little power, is memory-bandwidth bound. We did not measure DRAM counters, so treat that as the most probable reading rather than an established fact.

Reproducing

git clone https://github.com/vllm-project/vllm && cd vllm
gh pr checkout 51538
git am ../patches/0001-*.patch
TORCH_CUDA_ARCH_LIST="12.1a" MAX_JOBS=18 CUTE_DSL_ARCH=sm_121a  bash scripts/build.sh
bash scripts/cluster.sh    # both nodes, Ray
bash scripts/serve.sh

The scripts take the second node from $PEER (default sparky) and use $HOME for paths. The two RoCE addresses 192.168.100.10/11 in scripts/cluster.sh and scripts/serve.sh are ours, on a direct cable between the two machines. Change them to match your link, along with NCCL_IB_HCA and NCCL_SOCKET_IFNAME.

Notes that cost us time:

• --gpu-memory-utilization 0.80. Memory is unified: page cache and GPU allocations share the same 128 GB. At 0.85 a worker dies with NV_ERR_NO_MEMORY. Drop caches before launching (sync; echo 3 | sudo tee /proc/sys/vm/drop_caches).
• Export the CUDA environment in the Ray bring-up script, not just in your shell. Ray workers inherit it, and without nvcc on PATH the remote worker quietly decides has_flashinfer() is false and takes a different code path than the head.
• --chat-template-kwargs is not a vllm serve flag in this version. Reasoning is controlled per request: {"chat_template_kwargs": {"thinking": true, "reasoning_effort": "low"}}.
• In streaming, reasoning arrives on delta.reasoning, not delta.reasoning_content. Measuring TTFT on content alone produces nonsense: we generated a 1620 tok/s artefact that way before catching it.

Things we could not establish

Stated plainly so nobody builds on them:

• Accuracy. We measured throughput and acceptance only. No evaluation was run.
• Whether decode is memory-bound. Strongly suggested by the constant step cost and low power draw, not measured.
• SM120 versus SM121. We only have GB10. The DeepGEMM finding applies to family 12 by inspection of the dispatch, but we tested one part.

Licence

Recipe, scripts and measurements: MIT. The patch under patches/ is against vLLM and carries vLLM's Apache 2.0 licence.