Replies: 10 comments 4 replies
-
|
As a comparison here is the bench output from the exact same system with a RTX 3090ti: bench.sh outputbench output (3 warmups + 5 measured per prompt) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I'd love some suggestions on tweaks to the compose file that I can test to make better use of the 5090 and remove some of the needed workarounds that are in place from the 3090. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Massive thanks @apnar — first 5090 (Blackwell sm_120) data point on club-3090. Quick read on what your numbers show, then concrete tweaks to chase. What the data says (5090 vs 3090 Ti, same system)
~1.6× speedup is roughly the memory-bandwidth ratio (5090's 1.79 TB/s vs 3090 Ti's 1.01 TB/s = 1.77×) which is the right shape for an INT4-quantized 27B that's bandwidth-bound on weight reads. You're getting most of what the hardware promises out of an unmodified-for-Blackwell config — meaning the workarounds we ship for Ampere aren't catastrophically counterproductive on Blackwell, but several of them can be removed or replaced for further gain. Concrete suggestions in priority order: Tweak A (likely +5-15%): drop the Ampere-specific Marlin RO-mountThe compose mounts Try: # In your override compose, comment out or remove the two RO mounts:
# - ../patches/vllm-marlin-pad/marlin.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/marlin.py:ro
# - ../patches/vllm-marlin-pad/MPLinearKernel.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/MPLinearKernel.py:roBoot. If it boots clean and Tweak B (likely +5-10% accuracy, neutral TPS): swap KV formatYour current config inherits Try: - --kv-cache-dtype
- fp8_e4m3e4m3 has 1 more mantissa bit than e5m2 → less rounding noise on KV writes. On Blackwell you get the accuracy benefit AND the native compute path. Run a verify-stress probe at 60K + 90K needle (you'd need to raise Note: also worth testing Tweak C (lots more headroom): raise max-model-lenYou have 48K cap but the 5090's 32 GB VRAM has ~3 GB of unused headroom (29.8 used / 32 avail) at your current load. The KV pool is 5K-ish tokens worth bigger than what 48K needs:
Try Tweak D (probably no-op but worth flagging): Genesis Blackwell auto-detectionYour boot log will show
After your bench, would be useful to grep the boot log for docker logs vllm-qwen36-27b 2>&1 | grep -E "is_blackwell|\[OFF\]|\[ON \]" | head -30Some of those On the soak-test docker errorYour In the meantime, the soak-helper update I shipped today (commit f32d8a6) detects "silent-empty" turns (HTTP 200 + 0 completion tokens) automatically — useful gate even when you can't run it via the docker entrypoint. If you can run the helper directly: python3 scripts/soak-helper.py summary <turn-log.csv> /tmp/summary.md <boot_vram_mib> 200 0 5(But you'd need to feed it a turn-log.csv from a manually-orchestrated soak run, which on microk8s means writing a small wrapper. Probably not worth it until I soften soak-test.sh's docker check the same way setup.sh got softened.) What we'd love to see nextIn rough priority for adding to BENCHMARKS as a new "5090 single-card" row:
Genuinely thanks for being the first Blackwell data point on this stack. The 5090 sub-class is where most of the hardware roadmap goes; having someone running it through our verify-stress harness is a real contribution. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thanks for the suggestions. For A, I'm only running a single card so I've been using the default docker-compose.yml, the suggestion in A appears to only apply to the dual configurations as I didn't see those lines in the default one. I made the changes suggested in B and C and here are the bench results: bench.sh outputbench output (3 warmups + 5 measured per prompt)Additionally, here is the grep for is_blackwell: I'll look to give the suggested model a try later but am happy to try any other config tweaks you'd like to suggest. |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
@apnar — three observations, plus a Genesis bug to flag. Tweak A — you're right, NA for single-cardConfirmed: The Genesis sm_120 detection gapThis is the most interesting line in your boot log: Genesis correctly detects This is a Genesis bug, not a club-3090 issue. sm_120 is RTX 5090 / 5080 (consumer Blackwell). Genesis's platform detector probably checks for sm_100 (datacenter Blackwell) but not sm_120 (consumer). Worth filing on Sandermage/genesis-vllm-patches — title something like "Add sm_120 (consumer Blackwell, RTX 5090/5080) to is_blackwell detection". That filing alone would unlock a bunch of Blackwell-aware patches for you. Would expect non-trivial TPS uplift if it works (PN26b, P67 multi-query, FlashInfer NVFP4 paths, etc., are all Blackwell-relevant). Your B+C results — surprising and useful
Both fp8_e4m3 and 96K ctx slightly REGRESSED on your rig. Counter to my recommendation. Two likely reasons:
Recommendation: revert to fp8_e5m2 for now, keep max-model-len at whatever ceiling matches your actual workload (48K for short prompts, 96K+ if you regularly send long prompts). The fp8_e4m3 win on Blackwell is real in theory but needs the vLLM/Genesis stack to catch up. NVFP4 — the real Blackwell experiment5090 has native FP4 tensor cores — that's the marquee Blackwell feature. The closest community quant for our model is If it loads cleanly and runs, it should ship a real Blackwell win because the FP4 weights run on native FP4 tensor cores end-to-end (not dequant-to-fp16 like we'd see on Ampere). vLLM's NVFP4 support landed in late 2025 — should be in the nightly you're running. To try:
Expected: meaningfully faster than fp8 paths if it works. Possibly broken if vLLM nightly's NVFP4 path has issues — that'd be useful upstream signal too. Action items I owe you
Will land those when I do the next pass on docs. In the meantime, the Genesis sm_120 filing is the highest-leverage thing you can do — it'd benefit everyone running 5090 on this stack. |
Beta Was this translation helpful? Give feedback.
-
|
Filed the Genesis sm_120 detection bug as genesis-vllm-patches#20. Fix is a 1-line change in |
Beta Was this translation helpful? Give feedback.
-
|
Tried running the kaitchup/Qwen3.6-27B-autoround-nvfp4-linearattn-BF16 as suggested. First run: Then I try changing the data-type value from float16 to bfloat16 and it died with the following log output: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Tried changing Quantization to 'compressed-tensors' which got further but died with the following: |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@apnar — progress!
Try
|
Beta Was this translation helpful? Give feedback.
-
|
I got much further this time the language-model-only. Here was the output: I tried lowering the 96000 max-model-len back down to 48000, but it still failed with similar error. Any thoughts on where I can find a bit more memory? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@apnar — short answer: the OOM isn't where the easy memory is; the variant you picked is doing what its name says and it's structurally tight on a 32 GB card. Three concrete tweaks below that should boot it, plus an honest take on whether it's worth chasing. What the boot log showsThat 25.29 GiB load is expected, not a vLLM bug. The quant is So at Three tweaks (in order of how much memory they free)1. Drop MTP — likely +1.0-1.5 GB- --speculative-config
- 'null'MTP draft weights live in GPU memory (BF16 head). On 35B-A3B-FP8 we measured ~1 GiB per GPU; on 27B with BF16 head it's a similar order of magnitude. You lose the +30-40% TPS that MTP provides, but you regain the headroom that lets the model boot at all. This is the highest-yield single change. 2. Raise gpu-memory-utilization — likely +0.5-1 GB- --gpu-memory-utilization
- '0.96' # from 0.92The boot log explicitly told you the equivalent is 0.9214 with cudagraph profiling factored in, and 0.96 is well within RTX 5090 driver headroom. The risk is a brief peak-pressure activation hitting OOM; the reward is enough KV pool to actually fit a request. 3. Lower max-model-len to 16K-24K initially — proves boot works- --max-model-len
- '16000' # from 96000Once you confirm the model boots and serves one request, raise it as far as KV pool allows. Boot-log line Combined first attempt- --max-model-len
- '24000'
- --gpu-memory-utilization
- '0.96'
- --speculative-config
- 'null'If that boots: you've got a working NVFP4 baseline. Then experimentally raise The honest take — is NVFP4 worth chasing here?Probably not as your daily driver, for these reasons: (a) You don't get the Blackwell FP4 tensor core speedup you'd hope for. Native FP4 compute only kicks in on the layers that are FP4-quantized. On this variant 75% of the work is BF16 DeltaNet — running on regular Blackwell BF16 path, no FP4 win. So the speedup ceiling vs your current INT4 AutoRound is bounded by the 25% attention fraction, not the headline NVFP4 number. (b) You're already getting most of the Blackwell win on the standard INT4 AutoRound config. Your earlier numbers (122 narr / 162 code TPS, 29.8/32 GB used) are ~1.6× the 3090 Ti numbers — which is roughly the memory-bandwidth ratio. That means you're near the bandwidth-bound ceiling already. There's not 1.5× more TPS to recover by switching to a partial-FP4 quant; there might be 1.05-1.15× at best, and you give up 12 GB of KV pool to get it. (c) The "real" Blackwell-FP4 win is gated on a future quant that does FP4 on both attention paths. That requires either an NVFP4 DeltaNet kernel (doesn't exist in vLLM today) or a different model architecture without DeltaNet. Neither is something we can shortcut from the user side. (d) NVFP4 + AL>1 spec-decode interactions on Genesis-patched vLLM aren't well-characterized yet. You're a sample size of one on this configuration; we're discovering things like "PN8 silent-empty interaction" only this week through stiggy's bisect on a different config (issue #43). NVFP4 surface area is even thinner. What I'd suggest insteadStick with the AutoRound INT4 config you've already validated (122/162 TPS at 60K verify-stress all green) as your daily driver. If you want to spend cycles experimenting, the higher-yield Blackwell exploration is:
Those three are likely +10-20% TPS combined and unlock real new data points. NVFP4 with a partial-quant variant is a science experiment with a low ceiling on this hardware/model combo. If you boot NVFP4 anyway, what we'd love to seeIf you do get it booting with the three tweaks above:
That's the experiment that closes the question. Even a "no, INT4+MTP wins" result is real signal worth posting. Side note: your boot log shows the Genesis platform detection picking up |
Beta Was this translation helpful? Give feedback.
-
|
Managed to get it running with the changes you suggested. Actually didn't need to drop MTP. Numbers don't look good though. Here is the bench output: These are the command line options I used: Here are the logs from that run: As an aside, might be worth adding the '--root-user-action' flag on the pip command in the entrypoint for compose files just to get rid the the initial warning. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@apnar — thanks for sticking with the NVFP4 attempt and getting clean bench numbers out the other side. Honest read: 32 TPS is the expected envelope for that quant variant on this hardware right now, not a config issue you can tune around. Worth explaining why before you decide whether to keep it or fall back. Why the numbers are this slowCompare against your earlier INT4 AutoRound run on the same 5090:
The kaitchup variant's name is precise: The CV 0.1% you observed is the giveaway — that's a textbook bandwidth-bound path: 32 TPS × ~26 GB residency reads per token ≈ 0.83 TB/s sustained, which is 46% of the 5090's 1.79 TB/s peak. Reasonable for a path that's neither using Marlin's packed kernel (which AutoRound INT4 has) nor Blackwell's native FP4 tensor cores (which a hypothetical full-NVFP4 quant would). For context on what AutoRound INT4 is doing differently: it goes through vLLM's Marlin kernel which fuses dequant + GEMM in a packed format optimized for consumer GPU bandwidth. NVFP4 + compressed-tensors loader doesn't have that fusion path yet on consumer Blackwell — partly because of the genesis-vllm-patches#20 SM 12.0 detection bug we filed (Genesis can't fully recognize sm_120 as Blackwell, so platform-specific paths don't engage), and partly because vLLM's NVFP4 support is newer / less optimized than Marlin INT4. Recommendation: AutoRound INT4 is the daily-driver on Blackwell consumerFor now:
NVFP4 on Hopper (datacenter) is more mature today — the Hopper FP4 path has been in use longer and the loader is better tested on H100/H200. On consumer Blackwell (5090), it's preview-quality. What would be useful from your sideIf you have cycles for it:
Either of those gives you a more useful "5090 owner" data point than the NVFP4 path can today. We'd love to add a 5090 single-card row to BENCHMARKS at the AutoRound INT4 numbers — that's a config we can confidently recommend to other 5090 owners. Cross-link to genesis bugFiling reference for #20: the SM 12.0 detection issue is at Sandermage/genesis-vllm-patches#20. Once that lands, several Blackwell-specific paths in Genesis (Marlin tuning, P40 grouping kernel) will start engaging cleanly on your rig, which should help even the AutoRound INT4 path. Worth watching that issue. Genuinely thanks for being our 5090 cross-rig — the NVFP4 attempt was a worthwhile preview-class data point even though the numbers didn't justify daily-driver use. We're collecting these pin to the upstream tree as the Blackwell consumer envelope sharpens. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
A small meta-note as we close out today's loop on this thread, @apnar: The back-and-forth here on the 5090 NVFP4 attempts is genuinely high-signal — first Blackwell consumer cross-rig data on club-3090, surfaced the Sandermage/genesis-vllm-patches#20 sm_120 detection bug, gave us real numbers to anchor "what does the AutoRound INT4 path look like on a 5090". All useful. But we noticed the thread has accumulated several rounds of full-container-log dumps + Genesis dispatcher banners + bench output — content that's structurally a bug-tracking conversation, not the kind of open-ended hardware-questions discussion this channel is shaped for. We just sharpened the routing in disc #17 and CONTRIBUTING.md (PR #61) to make the convention explicit going forward:
Future iterations on this NVFP4 / 5090 work — would you mind taking them to a
This thread is fine as the design / hardware-class discussion for what 5090 ownership of club-3090 looks like — high-level "should we ship a 5090 default config?" sort of questions. The structured cross-rig bench data row (AutoRound INT4 daily-driver numbers) belongs in a Numbers from your rig issue when you have it. The bug reports (NVFP4 boot OOMs, slow throughput diagnoses, KV pool admission failures) belong in regular bug-report issues. No judgment on what's already posted — this is a "going forward" convention, not a retroactive cleanup. We're not migrating this thread or asking you to re-file anything you've already shared. Genuinely thanks for the depth of cross-rig data here; it's been load-bearing. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
As requested in one of the other discussions here is some initial output from a rtx 5090. I'm running on microk8s but using a container with the same parameters as:
models/qwen3.6-27b/vllm/compose/docker-compose.yml
Output of scripts/report.sh --full:
club-3090 rig report
Generated: 2026-05-04 17:02:23 UTC
Redacted output (paths, host, user, tokens). Re-run with
--no-redactfor full data.System
CPU + RAM
Disk
GPU hardware
NVLink
No NVLink detected (PCIe-only)
Topology
PCIe / GPU topology matrix
Full nvidia-smi
Full nvidia-smi output
Display / desktop state
Container runtime
Stack version
011d4cc(branch:master)git statusto inspect)2db18df(per scripts/setup.sh)Active container
No vLLM container running. Start one with
bash scripts/launch.shand re-run for the full report.verify-full.sh output
verify-full output
verify-stress.sh output
verify-stress output (7 boundary checks incl. Cliff 2 needle recall)
soak-test.sh (SOAK_MODE=continuous) output
soak-test stdout (5-session × 5-turn ramping conversation, ~25 min)
bench.sh output
bench output (3 warmups + 5 measured per prompt)
Generated by
bash scripts/report.sh. Flags:--verify(verify-full),--stress(verify-stress 7/7 incl. Cliff 2 needles),--soak(SOAK_MODE=continuous, catches Cliff 2b),--bench(canonical TPS),--full(all four, ~35 min). Use--no-redactto disable redaction (internal sharing only).Beta Was this translation helpful? Give feedback.
All reactions