Happy 4090 user checking in #62
laurimyllari
started this conversation in
General
Replies: 17 comments 11 replies
|
With the 450W limit, power use seems to go up to ~300W, and there's only a small change in the benchmark results. Hopefully this provides a clean 4090 data point. club-3090 rig reportGenerated: 2026-05-05 15:37:22 UTC Redacted output (paths, host, user, tokens). Re-run with System
CPU + RAM
Disk
GPU hardware
NVLinkNo NVLink detected (PCIe-only) TopologyPCIe / GPU topology matrixFull nvidia-smiFull nvidia-smi outputDisplay / desktop state
Container runtime
Stack version
Active container
Container Python / CUDA versions
Boot log highlightsKV pool sizing: Engine config (CLI flags + engine init): Recent warnings/errors (last 5): Full boot log (first 200 lines)First 200 lines of docker logsverify-full.sh outputverify-full outputverify-stress.sh outputverify-stress output (7 boundary checks incl. Cliff 2 needle recall)soak-test.sh (SOAK_MODE=continuous) outputsoak-test stdout (5-session × 5-turn ramping conversation, ~25 min)Soak summary ( Soak test summary
Recommendation
bench.sh outputbench output (3 warmups + 5 measured per prompt)Generated by |
0 replies
Power limit testingI checked the min power limit with 450W bench resultVerifying minimum power limitI ran a benchmark and full/stress verifications at 150W power limit to see if the GPU is (at least somewhat) stable still. Performance takes quite a hit here :) 150W bench result150W verify-full150W verify-stressBenchmark runs for every 10W from 150W to 300WI ran to do a quick bench run, and to take the mean narrative TPS values. 150W160W170W180W190W200W210W220W230W240W250W260W270W280W290W300WResultsLooks like the 230W recommendation for a 3090 works well here too: |
0 replies
|
@laurimyllari — welcome 🙌 great first-time post. Two things to call out, one practical, one cross-rig data: 1. You're on stale master — pull updates todayYour
cd ~/repos/club-3090
git pull origin master
bash scripts/setup.sh qwen3.6-27b
bash scripts/switch.sh vllm/minimal # or one of the better configs below2.
|
| Workload | Variant | Ctx | Why |
|---|---|---|---|
| chat / agents w/ images | vllm/long-vision |
145K | vision tower + tools + thinking + MTP n=3 |
| IDE-agent / RAG (text) | vllm/long-text |
180K | dropped vision tower for ctx + MTP n=3 |
| one-shot >50K text prompt | vllm/long-text-no-mtp |
200K | max ctx, slower decode, no MTP |
| bulletproof, no cliffs | llamacpp/default |
262K | different engine, cliff-immune, slower |
⚠️ Caveat: there's an open Cliff 2b onlong-text*/long-vision(single-prompt prefill >~50K may OOM on 24 GB single-card configs) — we filed Sandermage/genesis-vllm-patches#22 today and Sander's already on it. For prompts >50K today, route tollamacpp/default.
3. The 4090 power-curve sweep — genuinely useful cross-rig finding
Your sweep (230W = 52.85, 290W = 52.85, 300W = 52.97, 450W ≈ 52.97 narr) confirms what we observed on 3090s: decode is memory-bandwidth bound, not compute bound, so power-capping past ~230W gives diminishing returns. The 4090's GDDR6X bandwidth (~1 TB/s) is similar enough to 3090's that the curve shape carries over.
If you re-bench after the v7.72.2 uplift on long-text.yml or long-vision.yml, those numbers would be a great cross-rig data point — second 4090 in our matrix (snoby's was the first, but on dual-card so allreduce-bottlenecked). Drop them via Numbers from your rig when you have them and we'll add a row to BENCHMARKS.md.
For now, thanks for showing up with structured numbers and a chart out of the gate — that's exactly the bar club-3090's matrix is shaped around. Welcome to the club (which despite the name has gone happily multi-vendor — see disc #17).
0 replies
|
Thank you for the guidance! I'm testing now on 29718ca and switched to vllm/long-vision. With the defaults, I got I decreased the max_model_len to 140000. Trying to run the full report, stress crashes with the following: I think this is the issue that made me switch to minimal originally as I saw it discussed in the open issues. I also tried vllm/long-text, and there I had to adjust the max_model_len to 99072 (I went with 90000 for safety): Since this is about half what the compose has, I suspect there might be an actual issue here. Is this expected or should I file an issue with full details? This one passes stress though, and I've submitted the numbers as #71. I'll run this for a bit and see how it does. The speed is really nice :) |
1 reply
|
@laurimyllari — solid bench numbers in #71 + great diagnostic detail. Both halves of your question: "Is this expected or should I file an issue?"Expected, but the docs underspecify it for 4090. Two things compound on your rig that don't apply to the 3090s I calibrated the composes against:
So: not a bug to file separately, but a real cross-rig data point we should fold back into the docs. Your 90K override on Your numbers are excellent and going into BENCHMARKS102.96 narr / 103.09 code TPS on A few specific things stand out:
Concrete suggestion for your setupDrop On Cliff 2b for 4090The OOM you saw on Adding the BENCHMARKS row direct to master shortly — your rig context fills in a real gap. |
|
@noonghunna thank you again for the help! The env overrides are really handy. I tested with I tried combining that with a lower max_model_len ( I tried to do a full verify run, but hit the Cliff 2b.
I tried I lowered the max_model_len further to 80000 and got it to run full report until the soak test, and again hit cliff 2b. Maybe I got lucky with that one report run - now I'm getting the same OOM even at 0.93 and 90000. I will wait for v7.73.x (you and Sander are moving at impressive speed 👍) unless any further testing at this point is useful. I'm using this for a coding agent so llamacpp/default is probably most appropriate (vllm/minimal works with 64k context but it's pretty limiting). |
1 reply
|
We are pushing hard for the same reason, please bear with us while we figure this out. Thank you for the cross rig data, that's what keeps us going. |
|
@laurimyllari — solid debugging methodology and totally the right call. Quick close-out: Confirming your routing
One unrelated lever worth knowing aboutRecent cross-rig finding from @syangsao on issue #58 (water-cooled 3090, llama.cpp + Qwen3.6): power capping at 330W instead of stock 388W sits at peak TPS/W efficiency and only loses ~5% TPS. On a 4090 the math is similar — capping at ~370-400W (vs 4090's 450W stock) likely lands at the same efficiency knee with much cooler/quieter operation: sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 380 -i 0 # 4090 sweet spot estimateWorth a sweep on your rig if you're running long coding-agent sessions and want a lower thermal/acoustic footprint at near-zero TPS cost. Numbers anchored at What you're waiting for (status)Sander pivoted from a surgical PN59 patch to a full memory-management stack rework for v7.73.x (per disc #19, 30 min ago). His framing: solving it on single-card 24 GB structurally yields the same optimization on 32/48/72/96/142 GB tiers. He's targeting a 0.85-0.92 mem-util band as the new safe default and waiting for tomorrow's nightly to land the necessary upstream improvements. We'll validate the moment v7.73.x publishes across all 5 single-card variants ( Cross-rig matrixWhile we wait, your 4090 + Cliff 2b confirmation at 90K under multiple mem-util settings is itself useful data — you've validated that the cliff isn't 3090-specific (rules out "this is just a single-rig curiosity"). Adding to the validation pool we're feeding upstream at Sandermage/genesis-vllm-patches#22. Sincere thanks for the patient debugging. |
0 replies
|
A test limiting the 5090's power consumption to the absolute minimum would be incredibly helpful. I want to figure out the lowest possible TDP threshold for the 5090 (300|320|340|360|380W) and what the performance penalty looks like in that scenario. I'm toying with the idea of picking up an ASUS 5090 and doing a 48GB VRAM mod without any extreme board modifications. Luckily, I have a tech guy who can pull this off... It's a shame there's no way to source a bare PCB to solder on 96GB... and taking a blind gamble on Alibaba really isn't very appealing. |
0 replies
|
@Sandermage — great question, and an actionable one. Two 5090 owners on the cross-rig matrix who'd be the right people to ping:
@efschu — would you have time to run a power-cap sweep on your 5090? Sander's specifically interested in the lowest viable TDP threshold for the 5090 + the performance penalty curve, ahead of a potential 48 GB VRAM mod project. Same three-point sweep methodology @syangsao ran on his water-cooled 3090 which set our 330W production-default recommendation, just extended to five caps: # One-time setup (persists across driver reload):
sudo nvidia-smi -pm 1
# Sweep (test each cap):
for pl in 300 320 340 360 380; do
sudo nvidia-smi -pl $pl -i 0
bash scripts/bench.sh > /tmp/5090-pl${pl}.log
grep -E 'wall_TPS|GPU state' /tmp/5090-pl${pl}.log
done
# 5090 stock TDP is 575W; capping at 300W is ~52% of stock.
# Reset:
sudo nvidia-smi -pl 575 -i 0Caps requested by Sander: 300 / 320 / 340 / 360 / 380W (his framing: "lowest possible TDP threshold and what the performance penalty looks like"). Drop the resulting numbers (narr + code wall TPS, plus the GPU state line which captures actual measured power draw + temp at each cap) in a fresh issue or as a comment here. We'll fold the data into @apnar — if @efschu doesn't have bandwidth, would you be able to pick this up? Same methodology, no rush. Per @syangsao's data, the 3090 efficiency knee was 330W (5% TPS loss vs 388W stock for ~15% power reduction). Educated guess on 5090: knee likely in the 340-380W range (Blackwell is more thermally efficient than Ampere, scales better past lower caps), but the empirical sweep is the only way to know. Sander — re: the 48 GB ASUS 5090 VRAM mod: this is exactly the kind of consumer Blackwell direction the cross-rig matrix has been hoping for. 48 GB single-card consumer Blackwell is the natural successor to modded 3080 20 GB / 3090 24 GB territory. If you ship that rig and start testing v7.73.x on it, we'd love to track the 48-GB modded 5090 row as a first-class entry alongside @troymroberts' modded 3080 20 GB and the stock-VRAM tiers. |
0 replies
|
@Sandermage thank you for your great work! Sorry I can't help you with the 5090 numbers. Here's what my 4090 looks like with llamacpp. Raw data below the graphs. 
power-bench-llama-150-450-50.txt And a more detailed view (range is a bit too high, I think around 280 might be the sweet spot; max tps/W probably a bit lower even): Looks like it needs a higher power limit than with vllm/minimal. I can test again with the long variants when the new version lands. |
9 replies
|
I did both runs, results below. Considering the velocity of the project, it might be good to include the git commit in the output. These were on aa99173. Power-cap sweep — NVIDIA GeForce RTX 4090 (GPU 0)GPU: NVIDIA GeForce RTX 4090 VRAM: 24564 MiB Stock TDP: 450.00W Cooling: air
Detected boost-clock plateau(s):
Reset: left at last cap (--no-reset) Notes:
Power-cap sweep — NVIDIA GeForce RTX 4090 (GPU 0)GPU: NVIDIA GeForce RTX 4090 VRAM: 24564 MiB Stock TDP: 450.00W Cooling: air
Detected boost-clock plateau(s):
Reset: left at last cap (--no-reset) Notes:
|
|
@noonghunna friendly ping in case you missed the last runs - I realized I didn't tag you. Let me know if you'd like any other numbers. |
|
@laurimyllari thanks for the ping — and apologies, your earlier message slipped past me without a tag. These are exactly the runs we needed. Headline reads from your sweep:
What we'll do with this:
On further runs — if you've got cycles for one more, I'd love a |
|
Quick follow-up — landed all three:
|
|
@noonghunna here you go Power-cap sweep — NVIDIA GeForce RTX 4090 (GPU 0)GPU: NVIDIA GeForce RTX 4090 VRAM: 24564 MiB Stock TDP: 450.00W Cooling: air
Detected boost-clock plateau(s):
Reset: left at last cap (--no-reset) Notes:
|
|
@laurimyllari — that's an exceptional sweep, way more granular than what we asked for and the 10W-resolution zoom around the knee is the kind of data we hoped someone would generate. Thanks for the detailed graphs + raw txt files, both attached and downloadable. Headline: 4090 efficiency knee = 260-280W on llama.cppPulled the wall_TPS line from each cap in your two files and computed TPS/W:
Your hunch was spot-on — peak TPS/W at 260W, with the knee plateau extending up to ~290-300W (within 4% of stock TPS for ~33% less power). Past 300W, every additional watt buys diminishingly less throughput. Cross-rig pattern emergingComparing yours against @syangsao's 3090:
So 4090 is proportionally more aggressive in its efficiency — Ada thermal/power scaling lets you cut 33% of stock TDP for only 5-7% TPS loss, vs 3090's 15% TDP cut for 5% loss. That's a meaningful generational win. Your "needs higher power limit than vllm/minimal" observation is interesting too — engine matters because the kernel mix differs (llama.cpp's GDN forward kernel is more compute-bound than vLLM's GEMM-dominated AutoRound path), so the optimal cap shifts with engine. Worth noting in the cross-rig table. Folding into docs/HARDWARE.mdWe'll add your data as the canonical 4090 + llama.cpp anchor in Plus a "cross-rig pattern" line: efficiency knee lands at 60-85% of stock TDP across consumer Ampere/Ada — start there for any new card class and zoom in. On the 5090 askNo worries on not having one. Sander's 5090 ask still goes to @efschu / @apnar. But your methodology is now the gold standard — your 10W-resolution sweep around the knee gave us much sharper insight than the 5-cap sweep we'd outlined. We'll point any future power-cap contributors at your txt-file format as the template ("here's what a clean cross-rig power sweep looks like"). Re: re-testing on long variants when v7.73.x landsYes please. Once Sander tags v7.73.x and we re-validate Sincere thanks for the patient debugging across the whole thread — bug repro → mem-util sweep → llamacpp routing decision → power-cap data. Top-tier contributor experience from your side. |
0 replies
|
No luck here. I'm on Ubuntu 26.04 running NVIDIA 580 open drivers. If I try to set the power level into the 300s I get the following error: |
0 replies
|
@Sandermage In case if this helps https://x.com/snixtp/status/2052076464330154249?s=46&t=R6INmM76eUmUqb7vLzkRUg and |
0 replies
|
I ran to sweep of the 400s instead of the 300s, and there is basically no difference in output at the different wattages so I'll likely keep the card at 400 going forward. first last |
0 replies
Tooling update —
|
0 replies
Quick update — script now auto-derives the cap range per cardFollow-up to my post above: my original Pushed
So @efschu / @apnar — the literal command for your 5090 sweep is just: git pull origin master
sudo bash scripts/power-cap-sweep.sh --cooling <air|water|aio>No Same applies to anyone else picking this up on a different card class — the script figures out the right range from the hardware itself. |
0 replies
|
Ran your new script with modified power caps to keep above 400 as noted above. Even though there really isn't any variance I thought you may want the output in the same format. Power-cap sweep — NVIDIA GeForce RTX 5090 (GPU 0)GPU: NVIDIA GeForce RTX 5090 VRAM: 32607 MiB Stock TDP: 600.00W Cooling: air
Reset: auto-reset to 600.00W stock Notes:
|
0 replies
|
@apnar — really useful data, thanks for running it. The most informative finding isn't the TPS column — it's the actual-power column. What your data says
The 5090 is compute-saturated, not power-saturated on Qwen3.6-27B. The card maxes at ~430W draw regardless of cap because compute is the bottleneck, not power. So:
This is genuinely important data for Sander's 48 GB mod project — it tells him that the 600W stock TDP is overkill for current model sizes, and he can probably budget a 450-500W mod-cooler thermal envelope instead of designing for 600W. Two things would sharpen the picture further1. Re-run on the updated script (you ran the older 6-cap version, with stale "3 warm + 5 measured" footer text). Master is now at
For your 5090 specifically the literal command is: git pull origin master
sudo bash scripts/power-cap-sweep.sh --cooling air(no sudo bash scripts/power-cap-sweep.sh --cooling air --caps 300,310,320,330,340,350,360,370,380,390,400,420,440That'd test exactly the range Sander asked about (300-380W) plus a few above to confirm the plateau. 2. Try a heavier workload on a follow-up sweep — the 5090's compute saturation on Qwen3.6-27B is partly because that model is small relative to 32 GB VRAM. If you can run Gemma 4 31B (we just shipped MTP + DFlash on master — see disc #67) or any larger model that actually fills the 5090's compute, the knee position will likely shift higher and we'd get a more representative picture for the 48 GB mod use case. Moving the conversationCross-rig power-cap discussion has outgrown laurimyllari's "Happy 4090 user" thread (this one). Just spun up a dedicated home: disc #86 — Cross-rig power-cap efficiency matrix. Your data is already in the matrix there as the 5090 anchor. Future power-cap sweeps + follow-ups will land in #86 going forward. Sincere thanks for the data — the compute-saturation finding is exactly the kind of insight that changes how we frame the 5090 + larger-model story going forward. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I am fairly new to local LLMs, but after very encouraging results with just a 6700XT, I repurposed a gaming PC w/ a 4090. It's Ubuntu on bare metal, currently running the minimal.yml with 64k context and 0.95 mem util. I set a 230W power limit as a starting point. I'm using it with pi.dev and so far quite pleased with the results - I really appreciate the work that has gone into making this so capable!
Full report on this setup (I'll follow up with the report for 450W power limit):
club-3090 rig report
Generated: 2026-05-05 14:59:33 UTC
Redacted output (paths, host, user, tokens). Re-run with
--no-redactfor full data.System
CPU + RAM
Disk
GPU hardware
NVLink
No NVLink detected (PCIe-only)
Topology
PCIe / GPU topology matrix
Full nvidia-smi
Full nvidia-smi output
Display / desktop state
Container runtime
Stack version
224ca71(branch:master)git statusto inspect)2db18df(per scripts/setup.sh)nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8digestsha256:7923b48047be655ab8a18f2d152f7a67a03acb80866ee7cae298fea13c7ac9c7(7 days ago)Active container
vllm-qwen36-27b-minimalvllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8Container Python / CUDA versions
torch=2.11.0+cu130 torch_cuda_build=13.0 cudnn=919000.20.1rc1.dev16+g7a1eb8ac2Boot log highlights
KV pool sizing:
Engine config (CLI flags + engine init):
Recent warnings/errors (last 5):
Full boot log (first 200 lines)
First 200 lines of docker logs
verify-full.sh output
verify-full output
verify-stress.sh output
verify-stress output (7 boundary checks incl. Cliff 2 needle recall)
soak-test.sh (SOAK_MODE=continuous) output
soak-test stdout (5-session × 5-turn ramping conversation, ~25 min)
Soak summary (
results/report-soak-20260505-150551/summary.md):Soak test summary
Recommendation
bench.sh output
bench output (3 warmups + 5 measured per prompt)
Generated by
bash scripts/report.sh. Flags:--verify(verify-full),--stress(verify-stress 7/7 incl. Cliff 2 needles),--soak(SOAK_MODE=continuous, catches Cliff 2b),--bench(canonical TPS),--full(all four, ~35 min). Use--no-redactto disable redaction (internal sharing only).The exact changes in my repo:
All reactions