Any love for 4x3090 peeps in the future?

[Whamp]

I love the work you're doing here. I run a 4x3090 rig though and would love to leverage some of the best ideas being developed here and contribute some of my own. Happy to use my rig for testing some of these things too to provide answers for others. Just wanted to check to see if this would be something you're interested in adding to club-3090

[noonghunna]

@Whamp — yes, definitely interested. 4× 3090 unlocks things 1×/2× can't, and we have a wishlist of bench data points we can't generate ourselves:

What 4× 3090 can do that 2× can't

• TP=4 splits Cliff 2. DeltaNet GDN forward state buffer is per-rank — on TP=2 it doesn't help (Cliff 2 fires at 50-60K single-prompt regardless), but on TP=4 each rank handles half as much state, so single-prompt 100K+ becomes plausible. We've never been able to test this.
• Concurrent serving at scale. dual-turbo.yml hits 269 TPS aggregate at n=4 streams on 2× 3090 (4.67× KV concurrency). On 4× 3090 with TP=4, the same KV-pool math should land at ~8× concurrency — meaningful for multi-tenant agent farms.
• DFlash N=5 + bigger draft head. Our dual-dflash.yml is capped at 185K because the draft model takes ~1.75 GB/card on 2× 3090. With 4 cards the draft splits cheaper and ctx unblocks.
• MoE-class models. This codebase is single-model today (Qwen3.6-27B) but Qwen3.5-235B-A22B-class MoEs need 4×24 GB minimum. Adding it would mean a new models/qwen3.5-235b-a22b/ tree alongside the existing 27B one — repo structure already supports it.

What we'd love your help with

In rough priority order:

1. Boot the existing dual.yml at TP=4 instead of TP=2 — change --tensor-parallel-size 2 → 4 in the compose. We'd love to know what max_model_len + max_num_seqs the engine pre-check accepts. If it boots cleanly, that's a dual4.yml we can ship.

2. TP=4 + DFlash bench. Same prompt set we use (scripts/bench.sh) — narrative + code TPS, MTP AL, VRAM per card. Compare against our published 2× 3090 dual-dflash numbers (78 narr / 127 code TPS).

3. TP=4 single-prompt Cliff 2 boundary. verify-stress.sh probe 7 (60K + 90K needles) is the canonical Cliff 2 test. We expect it passes on TP=4 — would be the first published data we have for that.

Best path forward:

git clone https://github.com/noonghunna/club-3090
cd club-3090
bash scripts/setup.sh qwen3.6-27b
# Then either: edit dual.yml to TP=4, or open a PR adding a dual4.yml variant

Happy to merge any working compose variant + bench data as a PR. The hardware-mention docs (docs/HARDWARE.md) explicitly call out NVLink-vs-PCIe; 4-card setups need to know which they have for TP performance, so any boot log you share would help us add that guidance.

Welcome aboard. The cross-rig data is exactly what makes this stack honest.

[Whamp]

Fantastic. I'm on it.

[noonghunna]

@Whamp — 🙏 take your time. Even a partial first pass (just the boot log from dual.yml at TP=4 showing the engine-pre-check estimated max + Maximum concurrency line) would be useful before any benches. That alone tells us whether our dual4.yml projection is sane.

If you hit anything weird during boot, paste the relevant docker logs excerpt as a new discussion — first 4×3090 boot data we have. Worst case we co-debug; best case it's clean and we ship dual4.yml with credit.

[Whamp]

PR opened: #44

Validated the TP=4 fp8/MTP baseline from this thread on my 4× RTX 3090 PCIe rig:

• tensor_parallel_size=4, max_model_len=262144, max_num_seqs=4
• vLLM pre-check accepted the config
• GPU KV cache size: 483,200 tokens
• max concurrency for 262K requests: 6.77x
• verify-full.sh: PASS after warm retry (first post-boot Paris request hit the script's 30s cold-path timeout; immediate rerun passed)
• verify-stress.sh: PASS 7/7
  • Cliff 2 probe 7 recalled both large needles: 58,569 tokens and 91,070 tokens
• bench.sh: 63.01 narrative / 76.25 code wall TPS, peak 23,494 MiB/card

Interpretation: TP=4 gives the expected Cliff 2 margin and full-context concurrency headroom, but single-stream TPS is lower than TP=2 on PCIe-only allreduce. I framed dual4.yml as a 4-card capacity / Cliff 2 boundary variant rather than a faster short-prompt path.

[Whamp]

Follow-up: I added the TP=4 DFlash experiment to the same PR rather than splitting it out: #44

vllm/dual4-dflash results on my 4× RTX 3090 PCIe rig:

• TP=4, max_model_len=262144, max_num_seqs=2
• DFlash N=5 with z-lab/Qwen3.6-27B-DFlash
• GPU KV cache size: 207,264 tokens
• max concurrency for 262K requests: 2.27x
• verify-full.sh: PASS
• verify-stress.sh: PASS 7/7, including 58.6K + 91K Cliff 2 needles
• bench.sh: 64.00 narrative / 104.40 code wall TPS
• DFlash AL during code bench: 4.43 / 4.37 / 4.35
• peak VRAM during bench: 21,960 MiB/card

This looks shippable as a 4-card full-262K code-heavy variant: slower than the 2-card DFlash variants for raw single-stream TPS, but much faster than dual4.yml on code while preserving 262K admission and two full-context streams.

[noonghunna]

Quick update for anyone landing here from search:

@Whamp's PR (#44) is in review and now adds TWO 4×3090 variants:

• dual4.yml — TP=4 fp8/MTP baseline. 63 narr / 76 code TPS, 6.77× concurrency at 262K, ~23.5 GB/card peak.
• dual4-dflash.yml — TP=4 + DFlash spec-decode. 64 narr / 104 code TPS, 2.27× concurrency at 262K, ~22 GB/card peak.

Both pass verify-full.sh and verify-stress.sh 7/7 — including Cliff 2 needle recall at 58K and 91K tokens. That's the first published evidence on this stack that TP=4 escapes Cliff 2 on single-prompt the same way TP=2 does, with more headroom.

PR is not yet merged — waiting on the canonical rig report (scripts/report.sh) and optionally a v2 continuous-soak result against the new variants. Once those land we'll merge and close this discussion.

Variant picks (current best understanding):

• short-prompt single-stream code → dual-dflash.yml (TP=2, fastest)
• full-262K code single-stream → dual4-dflash.yml (Whamp's new variant)
• max concurrent capacity at full ctx → dual4.yml

Massive thanks @Whamp — cross-rig data of this rigour is what makes club-3090 honest.

[clort81]

Github makes it hard to follow these conversations once started.

[noonghunna]

@clort81 — fair point. If you have GitHub Copilot enabled on your account, the discussions view has a "summarize this discussion" feature that's designed for exactly this — the spark icon at the top of the discussion (or /copilot-summarize in a comment box) rolls up where the thread landed without you scrolling through every comment. Worth a try if you have access; it handles long threads better than the native UI.

If you don't have Copilot, the TL;DR for this thread:

• @Whamp shipped TP=4 baselines for 4× 3090 PCIe in #44 (merged e3f8f02). Two new variants:
  • vllm/dual4 — TP=4 fp8/MTP. 63 narr / 76 code TPS. 6.77× concurrency at 262K. ~23.5 GB/card peak.
  • vllm/dual4-dflash — TP=4 + DFlash spec-decode. 64 narr / 104 code TPS. 2.27× concurrency at 262K. ~22 GB/card peak.
• Both pass verify-full.sh and verify-stress.sh 7/7 including 91K Cliff 2 needles — first cross-rig confirmation that TP=4 escapes Cliff 2 on single-prompt the same way TP=2 does.
• Switch with bash scripts/switch.sh vllm/dual4 or bash scripts/switch.sh vllm/dual4-dflash (latter requires the gated z-lab/Qwen3.6-27B-DFlash draft — WITH_DFLASH_DRAFT=1 bash scripts/setup.sh qwen3.6-27b).

The canonical state lives in docs/MULTI_CARD.md and BENCHMARKS.md — those track what's currently shipped on master rather than the historical thread.