Replies: 1 comment 1 reply
|
Fantastic insights, thanks Paul! I was looking to migrate to Threadripper to support a 3rd 5090, and maybe my spare non-blackwell GPU (3090) to use for display. Right now, I'm using the Ryzen's iGPU for display so I can dedicate the full 64GB to LLM use. It seems like I might just stay on my current platform rather than a previous gen TR like I was planning, at least until the bubble pops or at least offers a mild deflate. Thanks again for the detailed analysis, it saved me a lot of investigation and money! |
|
@guybrush01 don't spend your budget on something else yet and keep those items in the shopping cart. Viability is still emerging with this platform template. Wasif is still in the kitchen cooking with GLM and Qwen3.8 Flash-Next recipes. In the interim, https://github.com/FlashML-org/FreeToken/tree/main has been pretty successful, today, hosting 'Flast-Next RAM+ GPU 0 while I run a single gpu slug on GPU 1. I'm trying to determine how to best tell the story as a follow up to this discussion. |
Uh oh!
There was an error while loading. Please reload this page.
WRX80 + 3975WX (Zen2) + dual 5090: a CPU-bound decode platform
Situation: A write-up of what happens when a high-end
platform (251 GB, PCIe ×16, 8-channel) with fast Blackwell GPUs is paired with a
Zen2 Threadripper PRO — and why the box ends up CPU-bound on decode despite
having two 5090s. All numbers are the my own measurements on this rig
and its B850 predecessor; each table names the A/B it came from.
Discussion drafted using AI. Purpose is to share a formatted evaluation of hardware, software, and performance at the intersection of changing hardware. I hadn't found much discussion along this combination and other may find it informing.
TL;DR
WRX80 (3975WX, Zen2) — made prefill faster but decode ~2× slower
(measured on Qwen3.8-27B, AutoRound INT4, vLLM TP=2).
driver, not the interconnect, not NUMA, not the GPUs.
(schedule → launch → sync → sample), and per-round-trip latency is set by
single-core CPU performance. The 3975WX (Zen2) has roughly half the
single-core of the 9950X3D (Zen5 + 3D V-Cache) → maps almost exactly onto the
2× decode gap.
Its per-step CPU cost (draft forward + verify) starves an already host-gated
decode. Run
SPEC_N=0(drafter off) for both single-stream and concurrent.is bandwidth-bound, not host-latency-bound, so its steady-state decode TPS
survives the CPU swap — but TTFT still shows the ~2× host penalty. The
diagnosis is workload-specific, not a blanket "this CPU is slow."
The platform
A strictly-better platform (more RAM, more channels, wider PCIe) that is slower
on half the workload is the counterintuitive hook — and the answer is that the two
halves of inference are gated by different things.
1. The split result: prefill up, decode down
Model: Qwen3.8-27B (AutoRound INT4 weights, fp8 KV). Driver-matched A/B (both
610.57.04, vLLM v0.27.1, same compose
dual-superfast=dual/autoround-int4/dflash2-fp8.yml, dflash n=7, TP=2, seqs=1, thinkingOFF). B850 arm = issue #1079 (08-21); WRX80 arm = issue #1107 (08-26).
Prefill is the platform's real win (more GPU power + memory bandwidth). Decode
is the casualty — and it is the CPU, not the GPU.
2. Why it's CPU-bound (the determination)
max_num_seqs=1is host-latency-bound. Each step is a chain of hostround-trips; per-round-trip latency is set by single-core CPU performance, not
core count or memory bandwidth (there is no CPU offload, so the 92 vs 32 GB/s
memory difference is irrelevant to the GPU decode path).
L3, 96 MB of it 3D V-Cache) has ~1.7–2.2× the single-core of the 3975WX (Zen2,
~4.2–4.4 GHz, 128 MB L3 over 4 CCDs, no V-Cache) — an IPC×clock estimate, not a
benchmark — and lands almost exactly on the measured 2.05× decode gap.
prefill is trivial at that length) → the ~74 ms delta is host-side first-step
latency.
built-in/fused drafting) = 133.69 vs 92.46 = 1.45×; the external block-drafter
(dflash n=7 = K autoregressive host round-trips/step before the target verify) =
2.05×. The external drafter multiplies host round-trips/step → amplifies the
host-latency difference. Corollary: B850 jumped +43% MTP→dflash, WRX80 only +1%
(its slow host can't exploit the better drafter — it's host-gated).
Ruled out (all measured / matched)
One minor, quantified confound: the two A/B arms are ~70 commits apart; in that
window
dual-superfastgained a volume-mountedvllm-flashinfer-decode-pinpatch(WRX80 has it, B850 didn't). Its own c=1 A/B is −2.1% narr / −1.3% code —
cost-neutral, so a ~2% effect cannot explain a ~2× gap. Noted, not attributed.
3. The drafter consequence: once it's CPU-bound, MTP is the enemy
Knowing the box is host-gated, the MTP drafter's per-step CPU cost becomes the thing
that starves it. Two A/Bs on this rig (both
dual-fast, INT4, fp8 KV, TP=2):Concurrent — 8 streams @ 2K ctx, greedy,
ab-spec-n.sh:No-drafter wins aggregate, per-stream, and GPU utilization; deeper spec is
worse (more drafter CPU overhead, not better amortization). The bottleneck moves
from CPU (64% SM, single-stream MTP) to GPU (89.9% SM, concurrent no-drafter) —
the box is now GPU-capped, so raising
max_num_seqsfurther buys little.Single-stream — 1 request at a time,
bench.sh,ab-mtp-single.sh(both armsself-verified drafter OFF / ON mtp n=4):
The drafter is a mixed bag single-stream: it helps code (structured, predictable
→ good drafter acceptance) but hurts narrative (prose → poor acceptance, so you pay
the full draft+verify CPU cost for few extra tokens). This inverts the 3090
reference rig, where MTP helps both (the
qwen3.8-27bdualmtp.ymln-sweep:narr 62.59→76.40, code 57.45→105.60, n=0→4) — its no-drafter baseline is much lower,
so even a mediocre drafter wins. Provenance caveat: that sweep was measured
2026-08-16 on the then-current Avuja export;
BENCHMARKS.mdflags the knee as"inherited, re-measure before quoting" after the 08-20 Frozenlock export swap, so
treat the 3090 baseline as directional. Both regimes point the same way:
SPEC_N=0.4. Corollary: the quant tier doesn't rescue decode either
With the box now GPU-bound under concurrent no-drafter load, the natural question is
whether NVFP4's real-FP4 GEMM (W4A4 on sm_120) pushes it higher than INT4. It does
not —
dual-fast(INT4) beatsdual-nvfp4on every decode-side metric(aggregate 430 vs 413, per-stream 105 vs 81, SM 90.6% vs 85.0%, KV 1.08M vs 0.99M;
a separate run from §3's 437/104 — ~1–2% run-to-run, same ranking).
NVFP4's only edge is prefill (TTFT 1762 vs 2058 ms, −14%) — and its weights are
heavier (21 GB vs 18 GB), which shrinks the KV pool. Decode is bandwidth-bound, and
"4-bit" NVFP4 is not lighter than INT4.
5. The counterpoint: the 35B-A3B MoE partially escapes the CPU penalty
The dense Qwen3.8-27B is the clean CPU-bound case. The
qwen3.6-35b-a3bis a different animal — a MoE with 35B total but only 3B active per token — and
it changes the picture in a way that confirms the diagnosis rather than refuting it.
Cross-rig, same NVFP4 dual compose (TP=2,
max_num_seqs=1, fp8 KV, MTP off):WRX80), the a3b decodes at the same rate on both CPUs. The reason is the decode
regime: with only 3B active weights per forward, the step is memory-bandwidth-bound
(weights streaming from VRAM), not host-latency-bound — so both the Zen5 and the
Zen2 host can keep up with the GPU. (Caveat: these two arms are not driver-matched
— B850 on 610.43.03, WRX80 on 610.57.04 — so read the tie as directional, not a
matched A/B. The dense Qwen3.8-27B comparison in §1 is driver-matched.)
host-schedule/launch event with almost no GPU work, so it still exposes the ~2×
host-latency gap (119 vs 57 ms). For a light MoE the CPU-bound signature moves from
slow decode to slow first token.
dense model.
Why this is a confirmation, not a contradiction: the one workload shape you'd
predict to be least host-sensitive — a light, bandwidth-bound MoE — is exactly the
one that keeps its decode TPS across the CPU swap. The penalty survives precisely where
the step is heavy (dense decode) or the event is a bare host round-trip (TTFT). So the
"CPU-bound" label is really "host-latency-bound workloads are CPU-bound," and the
a3b is the clean example of a workload that mostly isn't.
MTP is off here for a second, independent reason. On the 35B-A3B the built-in MTP
drafter is net-negative at vLLM TP=2 (−45% / −51% wall TPS despite AL 2.4–3.4 —
measured on the 3090 reference rig, not re-measured on the WRX80; applied here as
an engine-level vLLM TP=2 property): the MoE draft forward pays inter-GPU TP=2
sync that the acceptance can't amortize — a vLLM-scheduler-specific problem that
ik-llama single-card doesn't pay. So on this rig+model the drafter is off both
because the host is the bottleneck (general, §3) and because MoE×MTP carries its own
inter-GPU sync tax (model-specific).
Practical takeaway (if you're on a similar CPU-bound rig)
suspect single-core host latency — not the GPU.
NUMA the way §2 does.
SPEC_N=0). It is net-negative when the host is thebottleneck — in both single-stream and concurrent.
host latency; on a Zen2 box that's a hardware ceiling, not a config one.
improved.
Caveats
inference; the drafter's acceptance rate is not logged by
bench.shat thisverbosity, so the accept-len itself is not measured.
rig inverts the single-stream MTP result (MTP helps both there) — this is a
your-rig-is-the-outlier finding, not a reason to change the shipped compose's
default.
and cost-neutral (§2), but a same-SHA re-run of both arms would make it airtight.
WRX80 on 610.57.04) and not a same-session A/B — so the "decode tie, TTFT 2×"
read is directional. A driver-matched a3b re-run of both arms would tighten it.
The −45%/−51% MTP-on-MoE numbers in §5 come from the 3090 reference rig's
BENCHMARKS.mdrows, not a WRX80 measurement.the airtight version is a per-step host-latency profile (nsys) on both rigs.
Repro / provenance
results/ab-spec-20260828-192847/(harnessab-spec-n.sh—local-only, untracked; the results dir is in the repo, the harness is not)
results/ab-mtp-single-20260828-221323/(harnessab-mtp-single.sh— local-only, untracked)results/ab-arms-20260828-203711/(harnessab-spec-n.sh—local-only, untracked)
2x5090-wrx80-…-a3b-nvfp4, 08-28,610.57.04, NVFP4 dual, MTP off — a local rig report, not in the repo); B850 arm =
BENCHMARKS.mdqwen3.6-35b-a3b/dual/nvfp4row (610.43.03). MTP-on-MoEnet-negative:
BENCHMARKS.mddual/…/fp8.yml+preview-mtp.ymlrows(−45%/−51% at vLLM TP=2, 3090 reference rig).
All reactions