[Benchmark] Nemotron 3 Super 120B-FP8 on 8x RTX PRO 6000 (SM120): 3,215 tok/s burst, 150ms TTFT, 17.9M token KV capacity

[stewtong]

Benchmarked Nemotron 3 Super 120B-A12B-FP8 on 8x NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120) via AWS g7e.48xlarge. This is the fifth model in my SM120 characterization series (M2.5 in #18870, Qwen3.5-122B in #19603) and the first hybrid Mamba-2 + MoE model I've tested on this hardware.

The short version: Nemotron 3 Super is the most SM120-friendly model I've tested on g7e. GQA attention means FlashInfer + CUDA graphs just work (no --attention-backend triton forced). The Mamba-2 layers use O(1) recurrent state instead of per-token KV, giving 17.9M token capacity out of the box. Online serving TTFT is 150ms at 4 rps — the fastest I've measured on this hardware.

The blocked paths: MTP/EAGLE OOM kills (data on #20470), and trtllm_mha attention needs both SM120 FMHA cubins and NemotronH backend porting.

Environment

Component	Value
Instance	AWS g7e.48xlarge
GPUs	8x NVIDIA RTX PRO 6000 Blackwell Server Edition (SM120, 96GB GDDR7)
VRAM	~88 GB / 96 GB per GPU (model + KV cache + CUDA graphs)
System RAM	2.0 TiB
SGLang	0.5.6.post2 (built from main, post PR #20407 merge)
Triton	3.5.1
PyTorch	2.9.1+cu128
CUDA	13.0 (nvcc 12.8)
Driver	580.126.09
OS	Ubuntu 22.04 (kernel 6.8.0-1047-aws)

SM120 Backend Compatibility Matrix

Nemotron 3 Super has the simplest SM120 bring-up of any model I've tested — 2 required flags (same as M2.5), everything else works at defaults.

Component	Backend	SM120 Status	Notes
FP8 GEMM	DeepGemm	FAIL	kernel_runtime.hpp:45 assert (same as all models)
FP8 GEMM	CUTLASS	FAIL	SM100/SM90-only gate
FP8 GEMM	Triton	WORKS
Attention	FlashInfer	WORKS	GQA (32 Q / 2 KV heads) — no workarounds needed
Attention	Triton	WORKS	Tested, minor regression vs FlashInfer
Attention	trtllm_mha	FAIL	Prefill: SM100-only gate. Decode-only: AssertionError in nemotron_h.py:408 — HybridLinearAttnBackend expected, not yet ported for trtllm
MoE runner	Triton	WORKS
CUDA graphs	—	WORKS	52 batch sizes, no SMEM overflow (GQA headroom)
KV cache	fp8_e4m3	WORKS
HiCache	—	PARTIAL	Bug 1 fixed (#20504). Bug 2: _insert_helper() arg mismatch (reported on #20495)
MTP/EAGLE	—	WORKS	Requires --speculative-moe-runner-backend triton (tip from @nbyers-altira on #20470). Accept rate 0.81, accept length 3.2. See MTP section below
EP=8	—	WORKS	512 experts / 8 GPUs = 64 per GPU
Mamba-2 layers	Triton	WORKS	mamba_backend='triton' auto-selected
DeepEP	—	FAIL	PCIe P2P topology (universal g7e blocker)

Required flags: --fp8-gemm-backend triton --moe-runner-backend triton

For context — SM120 bring-up complexity across models:

• M2.5 ([Benchmark] MiniMax-M2.5 FP8 on 8× RTX PRO 6000 (SM120): bench_serving results + tuned fused_moe_triton config #18870): 2 required flags. FlashInfer, CUDA graphs, FP8 KV all work at defaults.
• Qwen3.5-122B ([Benchmark] Qwen3.5-122B-A10B FP8 weights / bf16 KV on 8x RTX PRO 6000 (SM120): 1,985 tok/s burst, MTP 2.75x, fp8 KV silent corruption finding #19603): 6 required flags. DeltaNet forces triton attention, bf16 KV, disabled CUDA graphs.
• Nemotron 3 Super: 2 required flags. Same as M2.5 — GQA is the path of least resistance on SM120.

Repro

SGLANG_DISABLE_DEEP_GEMM=1 python -m sglang.launch_server \
    --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \
    --tp-size 8 --ep-size 8 \
    --context-length 262144 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.85 \
    --fp8-gemm-backend triton \
    --moe-runner-backend triton \
    --schedule-policy lpm \
    --mamba-scheduler-strategy no_buffer \
    --reasoning-parser nemotron_3 \
    --tool-call-parser qwen3_coder \
    --enable-metrics --enable-cache-report \
    --trust-remote-code \
    --host 0.0.0.0 --port 8000

Model-specific notes:

• --mamba-scheduler-strategy no_buffer — extra_buffer asserts "not supported for NemotronHForCausalLM"
• --reasoning-parser nemotron_3 — not nano_v3 (which doesn't exist) or super_v3 (not in SGLang yet)
• No --disable-cuda-graph needed (GQA doesn't trigger the SMEM overflow that DeltaNet models hit)
• No kernel configs needed — 512-expert Triton MoE works without pre-tuned configs

Correctness Gate

5/5 deterministic checks pass (temp=0, "2+2" = 4 on every run). KV capacity: 17,892,656 tokens. Model loads in ~14s from cached weights (first deploy takes ~45 min for FP8 ModelOpt processing — see Gotchas below).

Results — Throughput (3-run validated, seed=1)

Burst (500 prompts, 200in/200out, inf rate)

Run	Output tok/s	Peak tok/s	Req/s	Mean TTFT (ms)	Median TTFT (ms)	Median ITL (ms)	P99 ITL (ms)
R1 (cold)	3,267	6,198	32.2	2,755	2,646	61	144
R2 (warm)	3,232	6,178	31.9	2,876	2,791	62	146
R3 (warm)	3,198	6,219	31.5	2,935	2,658	61	284
R2-R3 mean	3,215 ± 24	6,199 ± 29		2,906	2,725	61	215

Near-zero cold-start penalty (R1 within 2% of R2-R3). FP8 ModelOpt checkpoint skips runtime quantization — no Triton JIT warmup needed.

Note on output length: seed=1 produced ~101 tok/req average (50,722 total output tokens). This is model-dependent — different models generate different lengths from the same seed. I documented this effect in the M2.5 post. Burst throughput is sensitive to output length, so exercise caution when comparing burst tok/s across models.

Online Serving (300 prompts, 200in/200out, Poisson arrival)

Rate	Output tok/s	Mean TTFT (ms)	Median TTFT (ms)	Median ITL (ms)	P99 ITL (ms)
4 rps	376 ± 0.04	150 ± 2	134	20	265
8 rps	692 ± 1	171 ± 7	157	36	757

Rock-solid at 4 rps — throughput identical to 3 decimal places across 3 runs. 20ms median ITL. Still not saturated at 8 rps.

For reference, M2.5 on the same hardware at 4 rps produced 404 tok/s with 274ms TTFT (#18870). Online serving metrics are rate-limited and less sensitive to the output-length issue that affects burst comparisons.

Long Context (300 prompts, 16384 input / 200 output, 4 rps)

Run	Output tok/s	Mean TTFT (ms)	Notes
R1 (cold)	192	43,391	No HiCache — full 16K prefill, no caching
R2 (warm)	344	3,782	Partial radix cache hits
R3 (warm)	395	871	Radix cache warmed

No HiCache available (bugs above), so long-context performance relies on radix cache warming across runs.

MTP / Speculative Decoding (EAGLE)

MTP works on SM120 with one additional flag that isn't obvious: --speculative-moe-runner-backend triton. Without it, the draft worker's MoE layers default to a backend that OOM-kills on SM120. Credit to @nbyers-altira for the tip on #20470.

Config (added to baseline):

--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--speculative-moe-runner-backend triton \
--disable-radix-cache \
--mem-fraction-static 0.80

Note: mem-fraction-static dropped from 0.85 to 0.80 (MTP at 0.85 OOMs under 500-prompt burst) and --disable-radix-cache is required by MTP. Both reduce effective serving capacity.

What I can confirm:

• Server starts, CUDA graphs captured (23 batch sizes vs 52 without MTP)
• Correctness: 5/5 pass at temp=0
• Accept rate: 0.81, accept length: 3.2 tokens/step (from server decode logs)

What I haven't tested yet:

• Online serving (4/8 rps) with MTP — this is where speculative decoding's per-token speedup matters most (low concurrency, GPU not saturated)
• Single-request latency comparison (MTP's primary use case)
• Whether the throughput tradeoff changes at different concurrency levels

Initial burst observation (not a conclusion): 500-prompt burst with MTP produced 977 tok/s vs 3,215 tok/s without. But this comparison has three confounding variables (lower mem-fraction, disabled radix cache, and MTP overhead), so I'm not attributing the difference to MTP alone. More controlled testing needed.

The takeaway: MTP is functional on SM120 for Nemotron 3 Super — the accept rate is healthy and correctness checks out. Whether it helps throughput vs latency under realistic serving conditions is an open question I didn't have time to answer on this instance.

nvidia-smi (Post-Benchmark)

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    On  |   00000000:92:00.0 Off |                    0 |
| N/A   33C    P0             83W /  600W |   89510MiB /  97887MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    On  |   00000000:93:00.0 Off |                    0 |
| N/A   33C    P0             83W /  600W |   89510MiB /  97887MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+
|       ... (8 GPUs, all ~89.5 GiB / 97.9 GiB, 83-154W idle)                              |
+-----------------------------------------------------------------------------------------+

Gotchas

Reasoning parser name

Use --reasoning-parser nemotron_3. The HF model card mentions super_v3 but that parser doesn't exist in SGLang. nano_v3 (from older docs) doesn't exist either.

--max-running-requests kills burst benchmarks

Adding --max-running-requests 48 dropped burst throughput from 3,215 to 478 tok/s (-86%). With 500 concurrent prompts, a 48-request cap creates ~10 batches of queued work. Tail requests wait 50+ seconds before prefill. Don't set this for burst benchmarks.

extra_buffer mamba scheduler not supported

--mamba-scheduler-strategy extra_buffer asserts at startup: "mamba extra_buffer is not supported for NemotronHForCausalLM model". Use no_buffer.

Init time varies dramatically

First deploy: ~45 min FP8 ModelOpt weight processing in CPU RAM (128GB checkpoint → system memory processing → GPU transfer). Subsequent restarts from cached weights: ~14s weight load + ~20s CUDA graph capture (52 batch sizes). If you're iterating on config, restart the systemd service — don't reinstall.

HiCache on Mamba-2 — two bugs, first fixed fast

Bug 1: HiMambaRadixCache missing token_to_kv_pool_host → fixed in #20504 (12 hours after I filed #20495).
Bug 2: _insert_helper() arg count mismatch → still open (reported as comment on #20495). Tracking #20415 (radix cache refactor).

Key Takeaways

1. Most SM120-friendly model tested. 2 required flags, FlashInfer + CUDA graphs + FP8 KV all at defaults. GQA attention is the path of least resistance on SM120 — novel attention mechanisms (MLA, DSA, DeltaNet) consistently hit kernel gaps.

2. 17.9M token KV pool without HiCache. Mamba-2 layers use O(1) recurrent state — only ~8 GQA attention layers contribute to per-token KV growth. At 256K context per request, that's ~68 concurrent full-context requests before the pool fills. M2.5 with ~500K pool needed HiCache just to serve a handful of concurrent long-context requests. The Mamba-2 advantage isn't longer context — it's massive serving headroom at the model's native context length.

3. Near-zero cold-start penalty. FP8 ModelOpt checkpoint skips runtime quantization. R1 burst throughput is within 2% of R2-R3, unlike models that need Triton JIT warmup.

4. MTP works — with a non-obvious flag. --speculative-moe-runner-backend triton is required for the draft worker's MoE layers on SM120. Without it, OOM. With it, 81% accept rate and 3.2 tokens/step. Single-request and low-concurrency serving impact is untested but promising. Credit: @nbyers-altira ([Feature] NVIDIA-Nemotron-3-Super-120B-A12B Speculative Decoding Support #20470).

5. trtllm attention are still blocked. trtllm_mha needs SM120 FMHA cubins + NemotronH backend porting. When those land, the ceiling is higher.

6. Five models, same hardware. The SM120 compatibility matrix is getting thick enough to be useful. If there's interest, I can consolidate the cross-model matrix into a standalone reference.

Reproduction Checklist

• g7e.48xlarge (or any 8x RTX PRO 6000 SM120 system)
• SGLang from main (post PR [Model] Support Nemotron 3 Super NVFP4 #20407 merge)
• SGLANG_DISABLE_DEEP_GEMM=1
• --fp8-gemm-backend triton --moe-runner-backend triton
• --mamba-scheduler-strategy no_buffer
• --seed 1 for all bench_serving runs
• 3 runs minimum, report R2-R3 mean ± std (R1 is cold-start)

[hzh0425]

#20495 (comment)
We can continue the discussion in this issue~

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.