4× RTX 3090 PCIe (P2P patched): per-stream TPS does scale with TP here, plus a power-cap confound worth adding to the tables #773
Replies: 17 comments 8 replies
|
May I ask what is your PCIe version (3.0, 4.0, 5.0) and width (x4, x8, 16)? And what motherboard and CPU? |
|
For dual card with TP=2 PCIe 3.0x8 is kinda the limit for P2P. Good to know it works well with 3.0x16 and TP=4 |
|
Thanks for posting that. What numbers do you get from scripts/report.sh ? |
|
What report.sh gives (the number-bearing parts): The ACS part, which I think is the useful bit for this thread. and P2P works anyway. Not by driver report, by transfer: I ran vLLM's own Kernel cmdline here is If you wanted TPS, that is
One caveat on report.sh so you do not get misled the way I nearly was: its |
Same board and mine will refuse to post with pcie gen 4 with risers, could be the chinese risers. Have to set the gen speed in the bios before hooking up the gpus. |
|
I had Auto PCIe mode enabled and it wouldn't even start with these Chinese risers (which have 3 versions). I had to enable pcie 3 in the BIOS, and then it worked. If I inserted it without the riser, it worked fine with pcie 4 |
|
This is one of the most carefully-constructed contributions we've had — the single-variable framing, the explicit argv, the matched power cap, and especially the fact that you separated "here is my result" from "here is my hypothesis about why". Thank you. Short answers first, then the detail:
1. Finding 1: you're right, and it's recipe — your own hypothesis 2You listed P2P and recipe as two candidate explanations. It's recipe, and the numbers are unambiguous once you put them side by side. Our shipped configs run MTP K=3 speculative decoding; yours run no spec at all. Same-model decode tok/s:
a Both tables are correct. They disagree because the doc's TP=2 baseline already has ~1.85× of speculative decoding folded into it, and yours doesn't. TP scaling is real in vLLM — it's just climbing out of a hole that MTP has already filled on our side. The blunt version, and I think the useful one for a reader deciding what to buy:
So the doc's practical advice — "you don't get faster single-stream answers from more PCIe cards" — survives your data, because the way you get faster single-stream answers on 2 cards is a drafter, not a third and fourth card. But the doc's stated mechanism ("PCIe NCCL all-reduce overhead grows with TP count; per-stream decode at TP=4 may be lower than TP=2") is not what our own numbers show either — ours show TP=4 ≈ TP=2, not TP=4 < TP=2, and yours show TP=4 > TP=2 with spec off. It's a claim that needs scoping to the recipe it was measured on, and right now it reads as a topology law. I'll propose that edit. Your Finding 2 (llama.cpp tensor-split doesn't scale, vLLM TP does) is a clean separation of engine from topology and I think it's correct — worth noting the doc line predates our vLLM-first era, so "came from llama.cpp experience" is a good guess at its provenance. 2. The one run that would settle itIf you only do one thing, do this pair rather than the full chain: bash scripts/switch.sh vllm/qwen-27b-dual-fast # TP=2 + MTP n=3
bash scripts/bench.sh
bash scripts/switch.sh vllm/qwen-27b-multi-fast # TP=4 + MTP n=3, same recipe
bash scripts/bench.shThat's the matched TP=2-vs-TP=4-at-identical-recipe pair, and nobody has produced it on P2P-enabled hardware. It directly tests whether P2P changes the TP=4 story or just moves both points together. Two boots, same sitting, same 220 W — your own methodology. If you're willing to go further, 3. P2P tracking — mostly there, but your complaint lands somewhere else
It does, though I'd say it's under-signposted — But your real point is about the cross-rig table, and there you're right. P2P and power cap live in that table as free text in the Rig column, when a contributor happens to mention them — not as fields. Which means they're unsearchable and inconsistently present. Here's our own table failing your test:
We currently attribute that +27% to lane width. But those two rows differ on lanes and power and P2P-engagement-unknown, all at once. Your power-cap warning applies directly to our own published comparison — 230 W vs 350 W is exactly the trap you described losing an hour to. I don't think we can honestly claim lane-width-beats-P2P from that pair, and I'll say so in the table. Also worth knowing: we do have a matched P2P control, just at TP=2 — @kevinb361's same-host NVLink A/B (#698) found decode +3–5% but prefill +35–49%. If that shape holds for patched-PCIe-P2P at TP=4, your P2P hypothesis would predict a small decode effect and a large prefill one — which your decode-only numbers can't see. Another reason the paired run in §2 is worth more than more decode points. 4. Finding 5 — you independently reproduced a first-party result, and there's a root causeThis is the one I'd most like on the record, because it's a clean cross-rig confirmation of something we found in July (#594 / #595), on different silicon count and a different backend:
Same shape, same crossover, same "fastest at short context, slowest at depth" inversion for int8-PTH. And the mechanism we found is backend selection, not the arithmetic: One thing to check, because your rig may disagree with ours. You force Your TurboQuant-at-depth rows (4-bit 77.7 → 49.7, 3-bit 76.7 → 43.3) are new to us — we have TQ quality and capacity data but not a depth curve, and that decay is steeper than I'd have guessed. Worth its own row regardless of how the rest lands. 5. Finding 4 — replication vs TP is a real gap3.4× for 4× TP=1 gemma-12B over TP=4, and 2.07–2.40× for 2× TP=2 over TP=4 on the 27B, is a bigger effect than anything else in your post, and we don't document it at all. Our multi-card page is written entirely in the TP dimension. It's consistent with what we found in the concurrency direction — our dense 27B's batching knee is N=2 at agent-shaped contexts (decode-aggregate halves by N=8), while the 35B-A3B MoE holds ~250–270 flat to N=16. Same lesson from the other side: for aggregate work, stop feeding one big engine. Your Finding 3 is the crispest statement of it I've seen — TP=4 being 29% faster for one user and 38% slower for a queue, from one sitting, is exactly the "what should I pick" column you're describing. We hit the same fork and landed on a per-N table rather than a single number. "latency wants max TP, throughput wants max instances" is a better one-liner than anything currently in our docs. If you're happy for it to be quoted with attribution, I'd like to use it. 6. Power capAlready a first-class axis, just not in the cross-rig table (see §3). What I'd like to change on our sideFlagging rather than doing, since a few of these are judgement calls the maintainer should make:
On the methodology worry in your preamble — a hand-rolled single-variable sweep with the argv stated and the power cap held fixed is worth more to us than a template filled in loosely. The template exists so that people who haven't thought about confounds still produce comparable numbers. You clearly had. And thanks for the CLIFFS.md note — good to know it earned its keep. |
|
Shipped in
Two things I want to be straight about rather than quietly bury. Your P2P hypothesis is still open, and I can't close it from our data. The only 4-card row we have with the patched module is @Whamp's, and its engagement was never confirmed — so we have zero rows where P2P is known to have been active. That's a real hole and it's ours, not yours. Your Finding 5 isn't in this commit — the int8-PTH reproduction and the TurboQuant depth curve are worth a No pressure on the TP=2/TP=4 matched pair — the doc is corrected regardless of whether you run it. |
|
Correction to my last comment — I said your P2P hypothesis was open because @Whamp's engagement "was never confirmed", which reads like we have no way to tell. We do.
The reason @Whamp's rows lack it is chronology, not capability:
His runs predate the feature by two weeks — the gap they exposed is why it exists. What this changes for you: nothing to do. You asked for P2P state to be recorded next to the numbers, and on any current run it already is, with no extra step and no self-reporting. So the TP=2/TP=4 pair from my earlier comment would come back with the engagement verdict attached automatically — which is what makes it the run that settles your hypothesis 1 rather than just adding two more decode points. Worth a look on your own rig regardless of whether you run our chain: if that Note fixed in |
|
Two corrections to my own OP first, because one of them changes how you should read the answer you asked for. 0. I stated two things about my setup that are not true
FlashInfer is not broken on this box either. I said the JIT was broken from an nvcc mismatch. flashinfer 0.6.12 imports, loads and serves, and there is not one JIT error in the whole sweep log tree. What is actually true is duller: torch is Sorry for the noise. Both claims went in unverified and the second one has been sitting in my head for weeks. The reason this matters for §4 of your reply: you framed the test as "if int8-PTH overrode your FLASH_ATTN". Nothing was there to override. vLLM had a free choice and int8-PTH still left it exactly one option, which I think is a slightly stronger version of your finding. 1. The backend grepBoth models from the same sweep, vLLM 0.23.0, TP=4, sm_86. I split the table by whether the server actually served, for a reason I come back to at the end of this section. Qwen3.6-27B-FP8
Qwen3.5-27B-AWQ-4bit is identical on every row it shares. So: your #594/#595 mechanism reproduces on 4x3090 / sm_86 / vLLM 0.23.0, and it is not an override. Now sort my depth decay by backend instead of by bit width. One correction to my own OP table while I do it: the TurboQuant rows stop at 32K, so the
And the four that reach 131K: fp8 −9%, auto −21%, bfloat16 −22%, int8-PTH −65%. The tiers are the backends, and the two dtype families that collapse the candidate list to one entry, I had this ordered wrongly in my head before I recomputed it, so it is worth saying plainly: TurboQuant is not the well-behaved one. At 32K, 3-bit TQ decays worse than int8-PTH does. Qwen3.6-27B-FP8 reproduces the ordering on the same rig: fp8 −4%, auto −6%, int8-PTH −27%, tq 4-bit −31%, tq 3-bit −38%, all at 32K. Two checkpoints, one shape. One methodology trap in that grep, which I fell into myself. The 2. Bonus you did not ask for: the dtypes that do not boot on AmpereSince I had to separate served from died to answer §1 honestly, here is the failure catalogue with the real error. Might be worth a column in
The Two of these share one cause, which I did not expect: Caveat I want to state rather than have you find: that sweep hardcoded 3. P2P: I can do better than
|
| field | value |
|---|---|
| GPUs | 4x RTX 3090, sm_86 |
| PCIe | 3.0, x16 on all four, link.width.current = 16 everywhere |
| Power cap | 220 W on all four, uniform, every number in my OP |
| Motherboard / CPU | Huananzhi H12D-8D / AMD EPYC |
| Topology | NODE between all pairs, single NUMA node, no NVLink |
| Kernel module | open, Dual MIT/GPL, 610.43.02 (aikitoria fork) |
| P2P capability | pcie_p2p |
| P2P engagement | verified by transfer, 12/12 directed pairs |
| vLLM | 0.23.0, torch 2.11.0+cu130, flashinfer 0.6.12 |
The all-x16 part is relevant to the @Whamp / @ryanmpelletier confound you softened. My rig is all-x16 like ryan's but at 220 W instead of 350 W, so if I get you the matched pair it lands as a third point that separates power from lanes on that comparison.
6. The quote
Yes, please use it, attribution is fine.
7. What I am running
One card is tied up with something else tonight, so I am working on two and the four-card runs queue behind it.
Running now: the 12.6-vs-12.8 toolkit A/B from §2, 15 dtypes per arm at TP=2, to find out which of those boot failures are Ampere and which are my build.
Queued, in your order of preference:
switch.sh vllm/qwen-27b-dual-fastthenbench.sh, thenmulti-fastandbench.sh. The dual arm can go as soon as the current sweep clears; the multi4 arm waits for the fourth card. Same sitting, same 220 W, and it will carry your interconnect verdict automatically.- 220 vs 290 W on the four-card box, since you flagged my cap is below your measured sweet spot and all your curves are single-card.
Not doing the patch-on/patch-off A/B yet. It needs a reboot and reinstalling the module, and the last time DKMS silently reverted this patch on a kernel update it cost me an evening. I would rather hand you the matched TP pair first.
|
Your §0 corrections don't weaken the result — they upgrade it. "vLLM had a free choice and int8-PTH still left it exactly one option" is a strictly stronger mechanism statement than the override framing, and your backend-sorted 32K table makes the tier structure visible in a way our two-point #594 data couldn't. That part is written up and staged for DTYPE_MATRIX — it lands together with the failure-catalogue column once your 12.6-vs-12.8 A/B is in, so the whole thing goes in as one dated row-set with both rigs’ outcomes. Your "removes FLASH_ATTN from the ballot before the vote" line goes in quoted and credited, along with the boot-log trap (which we'd been bitten by too and never written down). The failure catalogue — a scope answer rather than a reviewWe're not going to validate the full dtype grid, and the honest reason is that most of it sits outside what club-3090 actually ships. The current vLLM catalog uses exactly three KV formats: One disclosure your catalogue did trigger, worth knowing before anyone chases your TQ depth numbers on a newer engine: TurboQuant KV dtypes currently don't boot at all on stock v0.25.1 (a profiling-path regression — it hands the TQ backend Transfer-verified P2P → read-only tier folded into #786 (#787 closed)Adopted in its cheap form: Custom all-reduce → #786 — real bug, opposite failure mode on our stacksYour reading of the vLLM gate is right and it's the part that matters: The one correction: you predicted Practical note for your queued matched pair: until #786 lands, the interconnect verdict on the multi4 arm will print that wrong-✓ — treat vLLM's own disable-warning line as ground truth for the AR column; the P2P half of the verdict stays valid. QueueThe order is right — matched pair first, 220-vs-290W second — and the DKMS story is a completely sufficient reason to leave the patch A/B for another evening. Your rig fields are exactly the shape the new Rig-cell convention wants; all-x16-at-220W makes you the missing third point on the lane-vs-power confound. One afternoon of your sweep: a doc row staged, a tooling fix scoped (#786, now carrying the cache-read tier), and a stock-vLLM regression flushed out (recorded in #788). Each of those only falls out when someone runs the sweep and reads the code underneath it. Much appreciated. |
1. The toolkit question from my last comment: answered, and it was the architectureI said I had a matched CUDA 12.6 vs 12.8 rerun going to find out whether the The 12.8 arm covers all 15 values vLLM 0.23.0 accepts. Every single one lands
The candidate lists being identical across both toolkits and both TP counts is
So the catalogue I posted stands as an Ampere property rather than a local build One incidental result worth a line: at 32K with a short prompt every dtype that And one small vLLM bug for whoever writes the on 2. Correction to my last comment, and the verdict line is wrong at TP=4I got the direction of this wrong last time and it matters, so let me fix it I wrote that Which produces the opposite failure to the one I described. Here are both lines The second one appears four times, once per worker. To be precise about what is and is not broken, because the headline is narrower
The gate itself is also broader than I said. Cheap fix if you want one: the disable warning is a reliable negative signal. 3. The matched pair you asked forMy other card freed up, so both arms went in one sitting after all.
Your §1 reading was right and my OP's +62% was the recipe. With MTP n=3 on What I did not expect is how closely this reproduces your own dual-vs-multi And the prefill number is the one I would put in the doc. +54.6% at 10K and Two caveats I want on the record rather than discovered later. The pair is not a pure TP isolation, for the reason in §2. The TP=2 arm ran Cross-rig comparison to your published rows is confounded by engine version. One thing that does survive the version caveat, because it is a within-my-rig 4. Power: no knee above 220 W on this config, and the reason is in the clocksYour
220 W to 330 W buys nothing. The curve is non-monotonic in both columns and The mechanism is in the two clock columns. SM clock climbs 1530 → 1890 MHz I would not read this as contradicting your 290 W figure so much as scoping it. Three notes on the script itself, all from this run. The power column is summed but the cap column is per-card. 438 W at a 220 W
Heterogeneous ceilings. 5. An experiment that would settle what the patch is worthThe gate is a hard-coded NVLink check with no env override, but it is one Two arms, same argv, presence or absence of the disable warning as the proof the I would rather offer this than run it blind. If you think the gate is there for |
|
Closing the one loose end from my last comment: the CUDA 12.6 arm finished, and nothing disagreed. Single-variable now, both arms same night, same model, same TP=2, same 220 W, differing only in which toolkit the server env points at: 8 boot and 7 fail on each arm, and every dtype picks the same attention backend on both. So the failure catalogue is an sm_86 property with the toolkit ruled out by measurement rather than by argument, and the Raw results and the summary script are in the run bundle if you want them for the |
|
Ran the §5 experiment before you had a chance to answer whether the gate is The patch is worth about 15% of decode on four cards, and vLLM declines it. Qwen3.6-27B AutoRound INT4, TP=4, 32K ctx, vLLM 0.23.0, 220 W on all four
Power was pinned at the same 220 W in both arms (measured median 212–217 W), Correctness first, because it is not clean and I would rather you hear it
So: no silent corruption found. But that is 5 generations and 8 probes, not a You cannot bypass just the gate. fully_connected = current_platform.is_fully_connected(physical_device_ids)
if world_size > 2 and not fully_connected: return # the gate
if self.world_size == 2 or self.fully_connected: # should_custom_ar()
return inp_size < self.max_size
return FalseSkipping only the first builds the communicator and then Why decode wins and prefill does not. I first blamed the 8 MiB on sm_86 missing from Arms were distinguished from the logs, not assumed:
Limits worth naming: one model, one context length, one engine. Not tried on |
|
Small correction for the record: the P2P engagement state of my #446 runs is recoverable from the archived reports. Both the June 19 The aikitoria module was installed and host peer access was capable, but the club-3090 serving stack did not engage it for either run. Therefore my FP8-KV vs INT8-PTH same-session comparison already held P2P constant and off. The Reports: |
|
Three updates since, taking them in order — receipts first where receipts exist. @alesha-pro — the toolkit A/B: row landed, and yes to the PRWith 12.6 ≡ 12.8 on all 15 dtypes, the gate we set fired: the backend-tiers row is live in DTYPE_MATRIX ( And yes — please open the PR with the failure table, against that section. Two asks for it: version-label the rows (your sweep is 0.23.0; on stock 0.25.1 every TQ dtype is unreachable behind the @alesha-pro — the gate experimentAnswering the question you didn't wait on: the gate is load-bearing in exactly one scenario, and it's the one you named yourself — advertised-but-broken peer access, where a forced custom AR is the silent-corruption case. NVLink-via-NVML is simply the only cheap proxy upstream had; on a transfer-verified rig it's over-conservative, and yours is the first measurement of what that conservatism costs (+15% decode at TP=4, power-pinned, arms distinguished from the dispatch logs — that table is exactly how to do it). So: your conclusion is our conclusion. The catalog gate stays, no monkeypatch ships, and the upstream framing writes itself — an escape hatch keyed on transfer-verified peer access (your On correctness: 4/5 greedy divergence with 8/8 argmax, 8/8 top-3 ordering, and median |Δlogprob| 0.011 is the expected signature of a changed reduction order (one-shot custom AR vs NCCL ring — float non-associativity flipping near-tie tokens), the same benign class we document for other kernel-path changes. Your own "not eval-clean" framing is the right label at n=5+8; if you ever want it firmer, an 8-pack per arm is the standard we use to move "reads benign" to "measured neutral" — zero pressure, the upstream case doesn't need it. One honest note on our side: our PCIE_P2P.md §6 currently says no measured multi-GPU P2P number exists. Yours is a different comparison (custom-AR-vs-NCCL on a P2P rig, not P2P-vs-none), so the sentence stands — but your result is now the best available estimate of the gate's cost on verified rigs, and once the upstream question has a thread to cite, it earns a doc note as a one-rig data point. @Whamp — correction taken, docs fixedThank you for digging the reports out — corrected in |
|
Thread TL;DR for new readers (status as of 2026-07-27 — start here, then dip into the detail comments as needed) @alesha-pro benchmarked a 4× RTX 3090 rig with the P2P-patched driver and, together with @Whamp's and @henrykrinkle01's follow-ups, this thread ended up producing five separate findings. The confusing part is that "P2P" involves three independent layers that each say yes or no on their own:
Where NVLink fits — it runs through the same three layers:
What this thread established, in order:
Closed by end of 2026-07-27 — everything this thread set in motion, with receipts:
Still open: the dtype failure-table PR (invited), the upstream custom-AR escape-hatch filing + Practical takeaways if you run 3+ PCIe cards: the vLLM "Custom allreduce is disabled" line is normal; your P2P still works through NCCL; state your power cap and interconnect on any numbers you share (the Rig-cell convention in BENCHMARKS.md now requires both, plus the resolved all-reduce state — this thread is why). |
|
Thanks—the correction in
Both archived reports show P2P explicitly disabled, so the second condition was already met for the v0.22.0 same-session A/B. The remaining question is whether that short-context result survives the current engine and at depth; I’m running that modern same-session comparison separately now. Also, the TL;DR’s “still open” matched TP=2/TP=4 pair appears to have landed in @alesha-pro’s comment above: +4.0%/−2.6% decode and +54.6–62.0% prefill. |
|
Follow-up: I reran the KV comparison today on the current stack, same sitting and same rig. Fixed: Backend resolution reproduced the expected mechanism:
Canonical short-context result
So the old short-context Decode at accumulated agentic depthTwo 15-turn fixture sessions per arm:
The crossover on this workload is around 7–10K prompt tokens. At ~58K, FP8/FlashInfer is 3.56× faster in decode. INT8-PTH TTFT is also 40.9% higher there. Fresh-prefill agrees: INT8-PTH is −3.7% at ~10K and −25.7% at ~90K versus FP8. It saves about 798 MiB/card, so the trade remains useful—but narrowly: short interactive work and VRAM headroom versus long-agent/deep-context performance. Full methodology, resolved logs and raw benchmark output: https://gist.github.com/Whamp/1e584027cc99035068063e95b9d99b89 |
|
Good report, now I know why I don't use INT8 for the KV-Cache. |
|
yup avoid triton, use flashinfer. |
|
Receipts first: @Whamp — both cells fixed, and the rerun is a third-rig confirmationYou were right on both counts: the Your v0.25.1 rerun is now its own row, and it's a bigger deal than a re-check: it's the third rig and the first TP=4 confirmation of the #594 backend-tier mechanism, with the thing the earlier data lacked — a quantified crossover. Short-context int8-PTH not only survives the engine bump, it grows (+76.5/+80.9% vs the old +57/+65); then the depth series inverts it at ~7–10K prompt tokens, down to −71.9% at 58K with FP8/FlashInfer decoding 3.56× faster. The DTYPE_MATRIX backend-tiers note now records the tier structure as stable across three rigs, two TP counts, and three engine versions — and the INT8-PTH row's "standout config" framing is re-scoped to what your data shows: short interactive work and VRAM headroom, not agents. Also noted with some satisfaction: your methodology line "custom all-reduce engine-gated off in both arms" is the #789 verdict language doing its job three hours after merging. @alesha-pro — I owe you a response to a comment I raced pastYour 01:17 comment landed 53 seconds after #789 merged, and my next reply was already in flight — so your §2 self-correction was shipped as code before your comment posted (the disable-warning override you proposed as the "cheap fix" is exactly what landed), and your §3 went unanswered for twelve hours. That one's on me; here it is now. §3 — the matched pair settles the thread's founding question, and it's in the table. Decode +4.0%/−2.6% with MTP on both arms, against our two-rig v0.24.0 pair's +5.7%/−2.9% — same sign, same magnitude, same narrative-up/code-down asymmetry, from one rig in one sitting. Recipe-not-topology is no longer a reading of the data; it is the data. And your instinct about what goes in the doc is right: prefill +54.6–62.0% and TTFT −37.6% are the multi4 single-stream story, the same shape as kevinb361's NVLink A/B with a larger prefill term. Both caveats are in the row verbatim: the AR-state asymmetry (now Rig-cell field 4 — every dual-vs-multi4 comparison carries it by construction, yours just made it explicit), and the version confound — on which, honest answer: we have no first-party §4 — the power findings are filed as #796, all three: the summed-draw-vs-per-card-cap presentation, the 2.6× §1's stray nugget — the Where the thread standsOpen items are down to: your failure-table PR, the upstream escape-hatch filing (+ |
|
@alesha-pro — the Reference rig,
Verdict: the engine version is flat on this recipe. The small deficit sits inside CV plus the power delta (June ran uncapped at 370/420 W; today 230 W — and your own 220→330 W sweep says this config doesn't care). So the +11/12% by which your v0.25.1 dual beat our June row was never the version. Which makes the real comparison the interesting one: your dual arm (79.26 / 106.50 at 220 W) beats today's fresh row by +15.7% / +15.6% — at a lower power cap. The known differences between the arms: your TP=2 runs custom all-reduce ON + transfer-verified P2P; ours runs neither (no-P2P reference rig, AR disabled by boot). And +15.7% is strikingly close to the ~+15% you priced custom-AR alone at in the forced-gate experiment — measured there at TP=4 by patching, measured here at TP=2 by cross-rig difference, landing within a point of each other. Platform differences (EPYC vs our host, lanes, risers) are in the residual, so this is a leading suspect with converging magnitude, not a proof — but it does mean your deferred patch-on/off A/B now has a quantified prize attached (~15% decode at TP=2), whenever a reboot is worth an evening to you. Docs updated accordingly: the matched-pair row's version caveat is resolved, MULTI_CARD's callout now states the gap is rig-side, and the fresh row carries the full Rig-cell fields. The TL;DR's open list is down to your three items: the failure-table PR, the upstream escape-hatch filing, and the |
|
@alesha-pro — your forced-on result reproduces what we measured here (~+15% at TP=4), and I went looking for the why you asked about. It's in the comments, and it's a kinder story than "the gate ignores P2P". The NVLink check was never the requirement — it's a cheap pre-filter in front of the expensive P2P check
# test nvlink first, this will filter out most of the cases
# where custom allreduce is not supported
# this checks hardware and driver support for NVLink
assert current_platform.is_cuda_alike()
fully_connected = current_platform.is_fully_connected(physical_device_ids)
if world_size > 2 and not fully_connected:
logger.warning(
"Custom allreduce is disabled because it's not supported on"
" more than two PCIe-only GPUs. To silence this warning, "
"specify disable_custom_all_reduce=True explicitly."
)
return
# test P2P capability, this checks software/cudaruntime support
# this is expensive to compute at the first time
# then we cache the result
# On AMD GPU, p2p is always enabled between XGMI connected GPUs
if not current_platform.is_rocm() and not _can_p2p(rank, world_size):
The defect is narrow: at Two details that confirm it's a heuristic, not a hard limit
Why Your own numbers show it's workload-dependent rather than simply wrong
That makes a cleaner upstream ask than "vLLM ignores P2P": let Where we landed, and why club-3090 still ships the gateWe measured the same bypass gain and deliberately did not ship it: prefill regresses, the tradeoff is workload-dependent, and monkeypatching an engine's topology gate isn't something we want in a default path for other people's rigs. Three-layer framing for anyone arriving from the tweet: (1) does the driver grant peer access — |
Uh oh!
There was an error while loading. Please reload this page.
Hi. Long-time reader of this repo, first post. I run 4x RTX 3090 PCIe at home and
MULTI_CARD.md asks for numbers from 3+ card rigs, so here are mine.
Upfront honesty about methodology: these are NOT from
scripts/report.sh. My stackis a bare
vllm servevenv, not your docker composes or model pins, so I cannot fill the"Numbers from your rig" template without pretending. I am happy to clone the repo and run
your chain properly if that is more useful, just say so. Everything below is a
single-variable sweep from my own harness, and I state the exact argv so you can judge it.
Rig
nvidia-smi topo -p2p r= OK on all 6 pairs, BAR1 32768 MiB per cardVLLM_ATTENTION_BACKEND=FLASH_ATTN(FlashInfer JIT is broken on my box: system nvcc 12.6 vs torch cu130)What I measured
Single variable is
--tensor-parallel-size. Everything else frozen: same weights, same--max-model-len 32768, same--gpu-memory-utilization 0.90, same prompt (a fixed~512-token summarization prompt),
max_tokens=256,temperature=0, streaming.No spec decoding, no MTP, no KV quant. Single stream uses
--max-num-seqs 1.decodeexcludes TTFT:(out_tokens - 1) / (e2e - ttft).wallisout_tokens / e2e.Median of 5 requests after a warmup.
Exact argv:
Finding 1: per-stream scales, and it is not close
Qwen3.6-27B, single stream, tok/s:
I ran this because MULTI_CARD.md says:
On my box it goes the other way on every vLLM config I tried (the llama.cpp leg is a
different story, see Finding 2). I also ran the same sweep back in June and got 35.8 /
55.9 wall for FP8 TP2/TP4, so this reproduces across a month and two separate sweeps.
The wider vLLM picture from the June sweep, all matched at 220 W, wall tok/s:
Two things that could explain the gap. Your table's TP=4 entry (63/76 fp8) is
basically my number, 61.6 decode. So we agree on TP=4. What differs is the TP=2 baseline:
all-reduces after every layer, so TP=4 pays that latency twice as often as TP=2. With
P2P on, the per-layer sync stops being the thing that eats the gain. I have not A/B'd
patch-on vs patch-off in one sitting, so this is a hypothesis, not a result. It is
cheap to test if someone has a 4-card box and a spare reboot. HARDWARE.md does not
mention P2P at all, so right now it is an untracked variable across contributions.
KV. Mine has neither, on either side. A TP=2 recipe with spec decoding versus a plain
TP=4 one is not a topology comparison, so some of the "TP=4 is not faster" impression
may just be MTP doing work on the dual side.
I am not saying the doc is wrong for your rig, it probably is not. What I would ask for is
that P2P state gets recorded next to the numbers, because without it two 4-card
contributions are not really comparable to each other.
Finding 2: this is a vLLM property, not a topology property
Same sweep, same 220 W, but the llama.cpp leg (tensor-split, not vLLM TP):
Going 2 to 4 buys 3-13% here, and the repeat spread is huge (Q8_0 gave 52.9 and 71.8 on
identical settings). So on the same box, same day, same power cap: vLLM TP scales per
stream, llama.cpp tensor-split basically does not. If your "per-stream doesn't scale" line
came from llama.cpp experience, my numbers agree with you. It is the vLLM row I would
scope more narrowly.
Finding 3: the same run flips direction when you switch to aggregate
This one surprised me, and it is the cleanest thing in the whole sweep because both halves
come from one sitting, one power cap, one P2P state. AutoRound INT4, saturation ladder,
peak tok/s:
So TP=4 is 29% faster for one user and 38% slower for a full queue. Same box, same day,
same weights, opposite winner depending on which number you care about. If your table ever
gets a "what should I pick" column, it probably needs to be two columns.
The FP8 half of this comparison is missing on purpose, see the failure note below.
Finding 4: for aggregate throughput, replication beats TP
Saturation ladder,
--max-num-seqs 256 --enable-chunked-prefill, peak tok/s, all at 220 W:So the rule on my box is directional: latency wants max TP, throughput wants max
instances. Same cards, same weights, opposite answer depending on which you optimize.
If a model fits on one card, serving 4 copies beats splitting it 4 ways by 3.4x.
A methodology warning that cost me an hour
My sweep also varies the power cap (220/250/270/300 W), and mixing those points silently
inflates ratios. The 4-instance gemma config does 3,425 at 220 W and 4,001 at 270 W, so
quoting 4,001 against a 1,007 measured at 220 W would have turned a real 3.4x into a fake
3.97x. Same trap on single stream: gemma AWQ TP=4 is 103.2 at 220 W and 105.6 at 270 W.
Every number above is matched at 220 W. If you ever add a power-cap column to the
cross-rig table, that alone would make contributions more comparable, because 3090 rigs
are all over the place on this (stock 350 W, undervolted, power-limited for heat).
Finding 5: KV cache dtype only matters at depth (for DTYPE_MATRIX.md)
Qwen3.5-27B AWQ, TP=4,
--max-model-len 131072, single stream decode tok/s:At 1K all six sit inside 3.4 tok/s of each other, so the pick looks free. At 131K it is a
2.6x spread, and
int8_per_token_headgoes from fastest to slowest. Qwen3.6-27B FP8 doesthe same shape (fp8 KV 63.0 -> 57.8, int8 62.7 -> 25.4), so it is not one checkpoint.
Worth a row in DTYPE_MATRIX.md if you want it: on Ampere, fp8 KV is the only one that
holds its rate as the window fills.
One failure worth recording
FP8 TP=2 with
--max-num-seqs 256 --enable-chunked-prefill --gpu-memory-utilization 0.90OOMs on my box:
CUDA out of memory. Tried to allocate 784.00 MiB. GPU 0 has ... 470.38 MiB free. The identical config at TP=4 boots and runs. That direction matches whatMULTI_CARD.md already says about headroom, so consider it a confirmation rather than a
bug report. Drop
gpu-memory-utilizationto ~0.85 for the TP=2 aggregate case.Raw data
Every argv, server log and per-request sample is in the run directory, happy to attach or
paste anything specific. If it would help, I can also publish the full JSONL.
Thanks for the repo, CLIFFS.md saved me a weekend once.
All reactions