🧪 NVFP4 lands — 4 slugs for Hopper/Blackwell. GB10 / 5090 owners: you ARE the validation

[noonghunna]

NVFP4 support just landed (#603, #607): four new slugs serving NVIDIA's official modelopt NVFP4 quants of Qwen3.6-27B and Qwen3.6-35B-A3B — and they're the first catalog tier we cannot run ourselves. The maintainer rig is 2× RTX 3090 (Ampere, sm_86); NVFP4 kernels need Hopper or Blackwell. So these ship 🧪 authored blind, and if you have a GB10 / DGX Spark, RTX 5090, RTX 6000 Pro, or H100 — you are the validation.

The slugs

slug	model	topology	ctx default	kv-calc prediction (not measured!)
vllm/qwen-27b-single-nvfp4	27B dense	1× ≥32 GB	131K	5090: 28.5/29.4 GB (97%, tight) · GB10: full 262K @ 29% · H100: 262K @ 46%
vllm/qwen-27b-dual-nvfp4	27B dense	TP=2	262K	2× 5090: 22.1 GB/card (75%)
vllm/qwen-35b-a3b-single-nvfp4	35B MoE (3B active)	1× ≥32 GB	131K	5090: 27.4/29.4 GB (93%) · GB10: full 262K @ 25%
vllm/qwen-35b-a3b-dual-nvfp4	35B MoE	TP=2	262K	2× 5090: 15.7 GB/card (53%)
vllm/qwen-35b-a3b-dual-nvfp4-fast	35B MoE (unsloth export)	TP=2	262K	MEASURED on 2× 3090: 22.46 GB/card @ 262K — the one slug in this table validated first-party (added #675; promoted ⚠️ Production w/ caveats same day, #676 — full gate incl. stress 8/8 + soak PASS; no --force needed)

If you have a GB10 / Spark, start with the MoE single. A 3B-active MoE is the right shape for big-capacity, lower-bandwidth unified memory — its KV is tiny (~1.3 GB @ 131K), so the full 262K context costs only ~25% of your budget. Raise MAX_MODEL_LEN=262144 and go.

What's actually in these checkpoints

We did the forensics even though we can't boot them:

• 27B: NVFP4 (gs16) FFN + FP8-static attention + an unquantized MTP head → we ship it with MTP n=3 on.
• 35B-A3B: NVFP4 expert FFNs + FP8 attention — and on this MoE our own A/Bs found the built-in MTP head net-negative (it shares the MoE forward), so this one ships MTP off, deliberately. Please don't re-add it without an A/B.
• KV, both models (corrected after a deeper look at the safetensors indexes): hf_quant_config declares FP8 KV, but neither checkpoint ships k_scale/v_scale tensors — so fp8 (e4m3) KV runs at scale=1.0, which is exactly the regime our production 27B FP8 tier quality-tied at (109/150 on 2× 3090, and the identical 109/150 on a 2× 5090 this week). vLLM's modelopt integration supports this pairing natively (KV_CACHE_QUANT_ALGOS = ["FP8", "NVFP4"]).
• Both run --quantization modelopt on stock vllm/vllm-openai:v0.24.0 with fp8 (e4m3) KV. Not nvfp4-KV — consumer Blackwell has no FP4 FMHA (vLLM #43562).

The launchers refuse these slugs on Ampere with the real reason (requires sm >= 9), and the c3 cockpit hides them by default on rigs that can't run them — so nobody downloads 20+ GB expecting a boot that can't happen.

How to test (15 min for a boot verdict, ~1 h for the full chain)

git pull
bash scripts/switch.sh vllm/qwen-35b-a3b-single-nvfp4   # or any of the four
bash scripts/verify-full.sh                              # MODEL=qwen3.6-35b-a3b for the MoE
bash scripts/rebench-full.sh                             # full eval chain, resumable

# then capture EVERYTHING in one attachable file — hardware, stack, boot log,
# quality tooling state + the full test battery (~43 min):
bash scripts/report.sh --full > nvfp4-report.md

Attach nvfp4-report.md to your issue and we have it all in one shot (it's path/host-redacted by default). If you only have 15 minutes, the plain bash scripts/report.sh > nvfp4-report.md (~2 s) + your boot log is already plenty.

Then open a numbers-from-your-rig issue (or use the c3 cockpit's funnel) with whatever you got. A boot failure is a fully useful result — paste the crash and your nvidia-smi header and we'll fix or demote the slug. Things we specifically can't know from here:

• Does MTP load on the modelopt path? (27B — if --speculative-config fails, drop it and tell us)
• fp8 KV at scale=1.0 quality on the MoE (quality-tied on our 27B FP8 tier; unmeasured on this checkpoint)
• Long-prompt behavior at >50K on Blackwell (our "Cliff 2" is measured on Ampere only)
• MoE routing throughput on unified memory — the number everyone with a Spark wants to see

First validated numbers become the slug's shipped baseline (rig-labelled), move it out of 🧪, and get credited in BENCHMARKS. Related: the arch-aware KV A/B in #571 is still collecting 4090/5090 datapoints — same rigs, adjacent experiment, happy to take both in one session.

[paulp83]

(Single 5090)

Early note, had to run with --force, will report back once results post

~/club-3090$ bash scripts/switch.sh vllm/qwen-35b-a3b-single-nvfp4
[switch] MODEL_DIR=/home/paul/models
[switch] bringing down: vllm-qwen36-27b-minimal
[+] Running 2/2
 ✔ Container vllm-qwen36-27b-minimal  Removed                                                                                                                                                                                                      1.2s 
 ✔ Network autoround-int4_default     Removed                                                                                                                                                                                                      0.1s 
[switch] ERROR: 'vllm/qwen-35b-a3b-single-nvfp4' is (NA: experimental) — not a reliable config.  Qwen3.6-35B-A3B NVFP4 (nvidia modelopt MIXED_PRECISION MoE: NVFP4 expert FFNs + FP8 attention; ~23.4 GB), single Hopper/Blackwell card (required_sm=9.0; NOT Ampere). 🧪 AUTHORED BLIND — sm_86 dev rig cannot boot NVFP4; first community boot + rebench-full validates (funnel). THE GB10/DGX-Spark single-card ask: 3B-active MoE suits unified-memory parts — GB10 128 GB runs the full 262K via MAX_MODEL_LEN env (131K default sized for 5090 32 GB). NO MTP by design: the built-in head is net-negative on this MoE (-51% measured on the AutoRound tier — drafter type matters, learnings). fp8/e4m3 KV @ scale=1.0 (no baked KV scales in this checkpoint). No DEFAULTS row (opt-in only).
[switch]        It is surfaced for visibility, but won't launch without an explicit override.
[switch]        Re-run with --force if you know what you're doing:
[switch]          bash scripts/switch.sh --force vllm/qwen-35b-a3b-single-nvfp4

[noonghunna]

Update 2026-07-11: NVFP4 runs on Ampere (3090s) — and the catalog now shows it honestly.

Credit where due: @superalesha (on X) spotted that vLLM v0.24 quietly runs NVFP4 checkpoints on Ampere via --quantization modelopt — a Marlin W4A16 weight-only fallback (the code gates on compute capability ≥ 7.5, no Hopper check). We verified it end-to-end on the reference 2× RTX 3090:

• The shipped vllm/qwen-27b-dual-nvfp4 compose boots the full production shape on sm_86: 262K ctx + fp8/e4m3 KV + MTP n=3 (draft acceptance 97–99%). Decode 69.7 narr / 85.5 code tok/s, 23.77 GB/card.
• Quality: 8-pack think-off 110/150 — a statistical tie with our fp8 production tier's 109. The fallback path costs zero quality (the weights are identical to what native-FP4 rigs run; only the activation path differs).

What changed in the repo (#670): the NVFP4 slugs are no longer hidden as "incompatible-hw" on sub-sm_90 cards. They now carry a fallback_sm: 7.5 floor, kv-calc prices the fit normally and annotates it, and the cockpit shows them with a ⚑ fallback badge: "runs via weight-only fallback (Marlin); works, no speed edge vs native 4-bit quants on this card."

Honest guidance for 3090 owners: it works, but for the base Qwen models it's ~20% slower than the AutoRound tier on the same cards — so AutoRound/fp8 remain the recommended Ampere configs. Where the fallback genuinely matters is models whose only quant is NVFP4 (a growing category — e.g. migtissera's new Tess-4-27B-NVFP4), which are now vLLM-servable on 24 GB Ampere cards for the first time.

Still wanted from the community: native-FP4 numbers (5090/GB10/H100) — the fallback tells us nothing about native FP4-activation throughput or quality, and the slugs' native validation asks in this thread all still stand. 🙏

[noonghunna]

Heads-up for first boots on SM120 (RTX 5090 / RTX PRO 6000 / B200-class): a known FlashInfer startup abort + workaround.

@huginnfork reports (on the Tess-4-27B-NVFP4A16 card, RTX PRO 6000, vLLM v0.24-era) that FlashInfer's JIT arch probe currently mis-detects SM120 and aborts engine startup. If your first boot of vllm/qwen-27b-{single,dual}-nvfp4 or the MoE pair dies during engine init with FlashInfer/JIT errors rather than an OOM or quant error, try:

TORCH_CUDA_ARCH_LIST=12.0+PTX VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_USE_FLASHINFER=0

Two asks if you hit it:

1. Report the original failure signature (the JIT error lines) even if the workaround fixes it — we'll fold it into the compose Caveats so the next person doesn't burn an evening.
2. Disabling FlashInfer changes which attention backend vLLM selects, which can move perf — so alongside any TPS numbers, paste the boot line Using <X> attention backend out of potential backends: [...] so we know which backend actually served your run.

This doesn't change the slugs' status — they remain 🧪 awaiting a first boot report either way; this just removes one known wall in front of it.

[noonghunna]

Ampere update #2: the MoE NVFP4 story upgraded from "compatible" to "competitive" — and it now has a slug.

Validated on the reference 2×3090 (vLLM v0.24.0, TP=2) and merged today (#675): vllm/qwen-35b-a3b-dual-nvfp4-fast — unsloth/Qwen3.6-35B-A3B-NVFP4-Fast, the first NVFP4 slug that's Ampere-validated first (inverse of the authored-blind pair above).

• Boots via the MARLIN NvFp4 MoE backend — first confirmed boot of that path on Ampere anywhere.
• Double statistical tie with the shipped AutoRound tier: decode 179.5 / 179.4 tok/s vs 182.3 / 182.3 (n=5, CV ≤0.8%), 8-pack think-off 103/150 vs the tier's 104-equivalent — with cli-40 at 20/40, the best we've measured on this MoE. Full 262K at 22.46 GB/card. Rows in BENCHMARKS.md.
• Why the dense 27B's ~20% fallback penalty vanishes here: the A3B MoE's decode is barely weight-bandwidth-bound.
• It ships 🧪, so launch with --force: bash scripts/switch.sh --force vllm/qwen-35b-a3b-dual-nvfp4-fast. verify-stress + soak are the remaining promotion gate; one verify-full check fails with the known family signature (streaming tool-calls + thinking-on → finish=length — same class Tess documents), everything else passes.

Dense 27B provider A/B also completed (unsloth vs nvidia export, both dual + MTP n=3): tie on every axis — decode 66.9/90.4 vs 69.7/85.5, 8-pack 109 vs 110, NIAH clean to 240K. So no second 27B slug (the nvidia-pinned one stays); unsloth's runs fine BYO via pull.sh. One notable mechanical finding: unsloth's unquantized MTP head loads through the compressed-tensors path — first time we've seen a non-modelopt NVFP4 export spec-decode in vLLM.

Standing bounds, unchanged: Ampere executes weight-only W4A16, so our quality numbers are the optimistic bound for native-FP4 rigs — the native-W4A4 quality run on a 5090/GB10 is still the arc's missing datapoint (the new compose header carries the exact 3-step ask). And both unsloth exports ship real fp8 KV scale tensors, but whether they load on the Qwen hybrid is unverified — treat as scale=1.0 until someone (probably us) does the in-container check.

[noonghunna]

KV-scale follow-up — settled, and it closes the loop on the provider question.

We said above to treat the unsloth checkpoints' shipped fp8 KV scales as "unverified, assume scale=1.0." We've now probed the running engine directly (in-worker apply_model, reading each attention layer's _k_scale/_v_scale): the calibrated scales LOAD and apply — all 16 full-attention layers of the 27B carry the shipped values byte-exact (0.023–0.271, none at the 1.0 default). The hybrid's KV-calibration disable only blocks runtime calibration; checkpoint-provided scales go through fine (maybe_remap_kv_scale_name + the compressed-tensors KV-cache method).

Which means the provider A/B we posted was, unknowingly, the exact experiment: calibrated scales (unsloth, 109/150, NIAH clean to 240K) vs scale=1.0 (nvidia, 110/150) — a tie. Calibrated fp8-KV scales are worth ≈nothing measurable on this model family at these tests — e4m3 KV is robust either way, which validates the scale=1.0 regime the production tiers run from the opposite direction. BENCHMARKS + the nvfp4-fast compose header are updated to match.