Call for 4090 / 5090 testers — arch-aware KV dtype A/B (#246 Phase 1) #571
Replies: 7 comments 5 replies
|
Completed update.sh 246-ab-e5m2_paulp5090_20260704.zip 246-ab-e4m3_paulp5090_20260704.zip Looked like it ran into trouble with nvfp4 |
|
Yeah, here's the vLLM when it reaches NVFP4: [nvlink] NVLINK_MODE=pcie_p2p — forcing PCIe P2P (NCCL_P2P_LEVEL=PHB, custom all-reduce ON) |
|
@paulp83 @guybrush01 — huge thanks, and the nvfp4 crash is not your fault or your hardware's — it's a genuine upstream kernel gap we've now diagnosed and gated. What happened: That was our bug to catch, not yours — our arm guard used a naive "sm ≥ 10.0" check that wrongly let sm_120 through. Fixed in #574 (just merged): the nvfp4 arm now refuses cleanly on consumer Blackwell instead of crashing mid-run, and points you at the right dtype. The good news: your e5m2 and e4m3 arms are exactly the data we wanted — and To wrap up your run: |
|
@guybrush01 — quick course-correction on what's actually worth your time, now that the first data is in. The KV-dtype arms (e5m2 / e4m3) turned out speed-neutral on 5090s — and that's expected, not disappointing. @paulp83's single-5090 run showed e4m3 vs e5m2 = 86.73 vs 86.66 decode TPS (<0.5%). The reason is mechanical: on consumer Blackwell (sm_120), FP8 KV is storage-only — the native FP8 attention kernel (FA3 / trtllm-gen) only exists on Hopper and datacenter Blackwell, so e4m3 and e5m2 store the same bytes and both dequant to BF16 for attention. So the KV arms are low-value on your card; no need to prioritise them. The To run it (2×5090): # 1. update to get the nvfp4 fix (so the runner behaves cleanly)
bash scripts/update.sh
# 2. one-time: fetch the FP8 weights (~29 GB)
WEIGHTS=fp8 bash scripts/setup.sh qwen3.6-27b
# 3. run the fp8w arm (add e5m2,e4m3 too if you like — they're quick, but fp8w is the prize)
bash scripts/arch-ab.sh --arms fp8w
# or, to bundle all three: bash scripts/arch-ab.sh --arms e5m2,e4m3,fp8w
The runner writes one |
|
Here's the result. Had to make two changes to the process. I'd do a pull request, but it's literally like 2 things.
Github isn't allowing the file to be uploaded via Firefox nor Waterfox. So I renamed it to .zip. Change it back to tgz for decompression. |
|
you are a star! thank you so much. |
|
Massive — this is the arm that closes the #246 loop. FP8 weights 134.5 / 165.1 decode on your 5090s vs the int4 arms' ~86 confirms native FP8 GEMM is the real consumer-Blackwell lever. 🎉 Both your fixes are in (merged to master):
So a quick Two honest notes on the data: your MTP hit a perfect 100% accept this run (mean accept length 4.00) — unusually clean, so it may flatter the TPS a touch vs a typical ~2.5–3; and it's cap-marginal at 262K (1531 MB ceiling headroom) — fine for chat, tight for sustained deep-context agents. One ask to fully close the "route Blackwell → FP8 weights" rec — and genuinely no rush, this is the slow part: could you run the 8-pack quality? fp8 is near-lossless by design, but we don't ship a recommendation without the pack confirming it. Ideally both thinking modes, so we catch the domain-split: bash scripts/quality-test.sh --full --no-thinking # our shipped default — the one that gates the rec
bash scripts/quality-test.sh --full --enable-thinking # thinking-on, whenever you have the cyclesHeads-up so it's not a surprise: each of these is a real sit — ~1–2 hours — so please treat it as a free-evening thing, no urgency at all. If you only have time for one, |
|
Here's the results of the nothink test. I'll run the think one later on. ~/work/club-3090 master [quality-test] thinking: disabled for every pack, ignoring per-pack defaults (non-canonical)
TOTAL | 102 / 150 | 68% | | | Failure breakdown:
Warnings:
==========================================================================
|
|
That's the last piece — thank you. 102/150 thinking-off lands within the 8-pack's ±5-7 noise band of our 2× 3090 fp8 (107), so FP8 is near-lossless on native Blackwell too, not just on Ampere's Marlin path. That's exactly the datapoint that closes the "route Blackwell → FP8 weights" recommendation — we now have fp8 quality confirmed on both silicon, alongside your decode numbers. Recorded on your dual-max row (with the harness fingerprint). Seven of eight packs track ours cleanly (toolcall 14 · structout 14 · bugfind 13). The one dip is dataextract 9/15 — but those are mostly No rush at all on the thinking-on arm whenever you have the cycles — but this alone is enough to call the fp8-weights lever confirmed on 5090. Genuinely appreciate you seeing it through. 🙏 |
|
~/work/club-3090 master [runner] timeout scaling active: measured_decode_tps=72.8, reference_tps=100.0, scale=1.37, thinking-budget-multiplier=16384/1024=16.00
TOTAL | 109 / 150 | 73% | | | Failure breakdown:
Warnings:
==========================================================================
|
|
This is the datapoint of the week — thank you. 🎯 109/150 on 2× 5090 = exact total parity with the 2× 3090 primary's 109/150 (#594). Blackwell native FP8 GEMM and Ampere Marlin W8A16 land the same 8-pack score — the fp8 tier's quality is arch-invariant, which is precisely what the scale=1.0 KV theory predicted. With your earlier 102, that's n=2 on-rig runs both inside the ±5–7 noise band. Your row is now the shipped 5090-class bar: Per-pack reads, for what we can honestly say:
Also: all 8 packs ran Two pointers while you're set up:
|
|
Adjacent ask for the same rigs 👋 — NVFP4 slugs just shipped (the nvfp4 arm discussed here is now a first-class catalog tier): 4 authored-blind slugs for Hopper/Blackwell — Qwen3.6-27B dense (MTP-on) and 35B-A3B MoE (MTP-off, deliberately) × single/dual. We can't boot them on sm_86, so first 5090/GB10 boots decide their fate. Details + test recipe: #608. If you're already set up for this thread's KV A/B, both fit in one session. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This thread is the coordination hub for cross-rig testing of non-Ampere (4090 / 5090 / DGX Spark / beyond) architecture support — instructions below, and post your results + notes as comments here. The engineering tracker is #246; verdicts land back there.
The four-phase plan for non-Ampere support
fp8_e4m3on sm_89+ for two pilot configs. First data: no consumer-card perf benefit (storage-only there) — the injection stays as a harmless better-precision default, not a speed lever. Verdict pending the lastfp8wdatapoint.MAX_MODEL_LEN. Independent of the KV question; design starting now.required_sm-gatedOne deliberate scoping choice: no quality testing in early phases. Your runs are TPS + recall only (~25–30 min per arm). Quality gates (the full 8-pack) run once, at promotion time, on the winning configuration — not N arms × N rigs. The NIAH recall ladder is included per arm because it catches "allocates fine, silently craters recall."
What to run
Update to latest master, make sure your usual Qwen weights are present, then one command — the runner detects your rig and picks the pilot config:
bash scripts/arch-ab.sh --arms e5m2,e4m3bash scripts/arch-ab.sh --arms fp8w· or--arms e5m2,e4m3,fp8wfor all threefp8wis the priority — native FP8 weights. ~1 h forfp8walone; needs a one-time 29 GB download (below)Each arm is a fresh boot +
rebench-full(verify → bench n=5 → NIAH ladder; soak/quality skipped).--dry-runshows the plan first;--resumepicks up where an interrupt left off;--helphas the details.Arm notes:
e5m2/e4m3(KV dtype): a fast baseline. On consumer cards these come back near-identical by design (FP8 KV is storage-only there). Still worth the ~1 h as a clean control + a NIAH-recall sanity check on your silicon.fp8w(dual rigs — 2× 5090 or 2× 4090): runsvllm/qwen-27b-dual-maxstock — FP8 weights + int8-PTH KV, no dtype pin. FP8 weights run through Marlin dequant on our 3090s but native FP8 GEMMs on your Blackwell, which is exactly the lever we can't test here. On Ampere the dual→dual-max decode margin is +17% / +16%; if yours is meaningfully wider — especially on TTFT / prefill (native FP8 GEMM helps the compute-bound prefill most) — that's the "recommend FP8 weights on Blackwell" signal. Needs the FP8 weights (~29 GB):WEIGHTS=fp8 bash scripts/setup.sh qwen3.6-27b. 2× 4090 works too — Ada (sm_89) has native FP8, so it's a valuable second consumer arch (does the lever generalize beyond Blackwell?); at TP=2 the weights are ~14.8 GB/card, leaving ~7 GB/card for KV — enough to boot 262K but tighter than a 5090, so the ceiling-stress rung may not fully clear (fine — the bench + prefill numbers are the point). A single DGX Spark can't run this arm (it's one GPU;fp8wis TP=2). Spark's 128 GB could host a single-card FP8 config, but that's a separate future add — ping here if you have a Spark and want it.nvfp4: skip it — datacenter-Blackwell-only (the runner refuses it on your card, cleanly). See the update box.What to send back
The runner ends by writing one file:
results/rebench/246-ab-bundle-<host>-<date>.tgz. Attach it to a comment here. It contains the per-arm bench logs, NIAH ladders, machine-readable summaries, and an auto-generated rig report (PCIe lanes, driver, power caps — paths/hostname/user are redacted by default).Two things as valuable as the numbers:
Decision rule (from #246's acceptance criteria)
e4m3vse5m2): ≥ ~15% → expand the injection; within CV → close it as a harmless precision default. First data says within-CV (storage-only on consumer), so this is heading toward "close with the measurement" — thefp8wdatapoint is the last thing outstanding.fp8w(FP8 weights): if native FP8 weights give a meaningful margin on Blackwell (watch prefill/TTFT), we write down "route Ada+/Blackwell users to the FP8-weights slug."nvfp4: parked until NVIDIA ships a consumer-Blackwell FP4 KV kernel (TRT-LLM #10241) — not testable on any card in this thread until then.No deadline — whenever you have a free evening. New volunteers very welcome (any 4090 / 5090 / DGX Spark) — just grab the command for your silicon above. Thanks for lending the hardware 🙏
All reactions