Qwen Flash Next - multi gpu results (broken wddm?)

[pepedombo]

set "CUDA_VISIBLE_DEVICES=0,1,2,3" && E:\llama\llama-bin\llama-server.exe --host 127.0.0.1 --port 8080 -m H:.lmstudio\models\unsloth\Qwen3.8-Flash-Next-GGUF\UD-IQ4_XS\Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf -ngl 999 -sm layer -ts 1.45,0.80,0.85,0.90 -c 90000 -np 1 -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024 --fit off --load-mode none -fa on -t 8 -tb 8 --jinja --reasoning-format deepseek --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --lazy-mode off --n-cpu-moe 9

Basic cmd as entry point. Just curious why it sucks as it's moe? Anyone can explain me all these results from ddr5/ddr4 that goes over reddit?

I can fully offload q3_xxs.gguf (the example above is q4_xs) which starts 30tok/s at decode but goes somewhere below 20tok/s (full pp is around 150-200 (300+ for xxs) depends on the fit on or manual fit). My setup of 4gpus is quite retarded in terms of pci-lane bottleneck as its x8/x4/x1/x1 for 5070/2x5060/4060 but still people with single 16gigs vram report crazy numbers with decode/pp.

As far as I feel quite confident at stating x1 doesn't influence decode (for 27b dense at least) then every time I push ram offload it goes crap. My results of pp 170 and decode of 12tok/s at 50k ctx seems lower to what people state at their ddr5 multi-chan memories.

UPDATE:

I did some testing and the main problem seems to be Windows, not DDR4 or PCIe x1 itself. On the exact same hardware and same llama.cpp commit I faced huge drop in decode when 4th gpu was added.

I've figured it out quick with llama-bench, example below (tested on 27b-q6):

Windows (3-gpu baseline): ~1100PP / 20 tok/s
Windows (4-gpu): 900PP / 9tok/s
Ubuntu (3-gpu baseline): 1300PP / 20tok/s
Ubuntu (4-gpu): ~1300PP / 20 tok/s

Docker/WSL2 on Windows gave basically the same bad result as native Windows (~913 PP / ~10 tok/s), so this does not look like a Windows build issue- it points more towards the Windows/WDDM multi-GPU path. GPU power draw is also noticeably lower on Windows, which looks like the cards are stalling/waiting rather than being compute-limited.

On Linux Flash-Next behaves completely different:
Q3_XXS (full offload 80k ctx): starts 45tok/s and drops to 30-35 at 50k ctx (no mtp, acceptable result), PP jumped from windows 350tok/s to 550tok/s.
Q4_XS (partial offload, autofit): stable 20tok/s at 40k ctx instead of 10, slower due to offload (ddr4 3200 dual-chan memory)

The flaw is found but I have no idea if it used to be like that or it just happened lately.

[Volunteer-1]

qwen4exp architecture is not fully supported yet, therefore it's slow.

[helloe365]

Two separate things are stacking in your results.

The first is what the comment above says, and it's worth taking seriously: the qwen4exp arch only landed in tree recently — it's implemented (src/models/qwen4exp.cpp, including the hybrid block-sparse attention memory it needs), but the fused kernels that make the established architectures fast take longer to arrive. Follow the qwen4exp-tagged PRs over the next releases; the same GGUF will get quicker without you changing anything.

The second is your bus, and no amount of software maturity fixes that. With -sm layer, every token's activations hop through all four cards, and two of your links are x1 — that's ~1 GB/s each way on Gen3. At tiny contexts each hop is small enough not to matter, which matches your cold-start 30 tok/s; as the context grows to 50k, the KV plus activation traffic per token grows with it until the transfers themselves serialize and decode sinks. The decay you see with context size is the signature of that.

The DDR5 numbers people post are a different hardware class: those are --n-cpu-moe builds with the experts resident in system RAM on quad/octa-channel (Threadripper/EPYC, 200+ GB/s), where RAM is nearly as fast as a mid GPU. Your X570 is dual-channel DDR4 — ~50 GB/s on a good day — so any RAM offload has a hard ceiling several times lower. It's not a flaw in llama, it's arithmetic: similar weight bytes over a quarter of the lanes and a fraction of the RAM bandwidth.

What I'd try on that box, in order:

• measure a clean single-GPU baseline: the xxs quant fully on the 5070, nothing else connected — that tells you what the arch is worth without bus contention
• if the split is unavoidable, bias -ts harder so the x1 cards carry as few layers as possible, or test dropping the x1 pair entirely if a quant exists that fits on two cards — two well-fed GPUs often beat four where two are starved
• keep -ctk/-ctv q8_0 as you have it; that's already the main lever against context growth

If any of this untangles the numbers, marking the answer helps the next person with mixed-width lanes find the bandwidth explanation.

[Volunteer-1]

@helloe365 Thanks for elaborating on what I meant.
With -sm layer, prefill speed will be bottlenecked by the slowest compute among the chain of cards, and decoding speed will be bottlenecked by the slowest VRAM bandwith among the chain of cards.
With -sm tensor, decoding speed will be bottlenecked by the slowest PCIe bandwith among the chain of cards, and when using drafting, this effect will be much stronger.

[pepedombo]

I decided to ask this question based on my experience with Qwen 27B dense and how it behaves on a mixed PCIe setup. Qwen Flash-Next being MoE and having additional per-layer token embedding / n-gram-related components makes its scaling much harder to predict.

I'm not using -sm tensor for obvious reasons: it is considerably more PCIe-intensive, gives poor PP on my setup, and decode performance is not particularly impressive either.

I'll start with Qwen 27B + MTP as a baseline. These are code-generation workloads using KV16 and -sm layer.

For an equal layer split, the effective memory bandwidth can be roughly approximated using the harmonic mean:

5070 Ti + 2×5060 Ti: ~538 GB/s
5070 Ti + 2×5060 Ti + 4060 Ti: ~442 GB/s
5070 Ti + 5060 Ti: ~597 GB/s
2×5060 Ti: 448 GB/s

This simplified model also explains why 5070 Ti + 5060 Ti is measurably faster than 2×5060 Ti in real-world testing.

As a separate PCIe test, I compared a 5060 Ti running at PCIe x8 vs x1 with a 9B dense model on a single GPU. Decode started at about 66 tok/s on x8 and 63 tok/s on x1, so the x1 link caused roughly a 4.5–5% decode loss. I have not yet measured the full PP penalty in that configuration.

With Qwen 27B, fewer GPUs are clearly better on my system as long as the model fits. On 3 GPUs, fresh-context decode is ~39.56 tok/s. Adding the 4060 Ti as a fourth GPU drops it to ~26.72 tok/s, a ~32.5% loss, despite the simplified equal-split bandwidth estimate dropping by only ~18%.

At ~60k context:

3 GPUs: ~716.8 tok/s PP, ~33.35 tok/s decode
4 GPUs: ~583.7 tok/s PP, ~24.14 tok/s decode

So adding the fourth GPU costs about 18.6% PP and 27.6% decode at ~60k context.

On the 3-GPU setup, decode drops from 39.56 tok/s fresh to 33.35 tok/s at ~60k context, which is only a ~15.7% loss.

Now for Qwen Flash-Next UD-IQ3_XXS: 80k context,KV8_0,-sm layer,all main layers on GPU, per_layer_token_embd.weight=CPU

Results:

Fresh decode at ~1k context: 30.35 tok/s
Decode at ~60k context: 15.56 tok/s
Full ~60k PP: 326.42 tok/s

That is a ~48.7% decode loss at only ~60k context.

So the interesting part for me is the direct 4-GPU comparison. On the same 4-GPU topology, the dense 27B model drops from ~26.72 tok/s fresh to ~24.14 tok/s at ~60k context, which is only about a 9.7% decode loss. Flash-Next, on the other hand, drops from 30.35 tok/s to 15.56 tok/s at roughly the same context length, a 48.7% loss.

The configurations are not perfectly identical — 27B uses KV16 while Flash-Next uses KV8_0, and the architectures are fundamentally different. On the same mixed-PCIe 4-GPU setup, Flash-Next shows a much stronger context-length-dependent decode penalty than the dense 27B model.

I will not provide more data from next-q4-xs because as you've already stated - my 80gigs ram are running at 35gb/s at average (3000mhz 2x32+2x8) and its a solid brick in the setup that causes huge loss in every area when I apply various offloads.

[Volunteer-1]

Let me repeat myself:
With -sm layer, decoding speed will be bottlenecked by the slowest VRAM bandwith among the chain of cards.
Because every token generation must flow through all layers, that means through the slowest VRAM.

[pepedombo]

The slowest card clearly matters, but it does not solely determine decode speed. With an equal layer split, my measurements track the harmonic-mean bandwidth approximation surprisingly closely.

llama-bench, Qwen 27B Q6_K, -sm layer, -ts 1/1, TG1024:

5070 Ti + 5060 Ti: 23.21 t/s
5070 Ti + 4060 Ti: 16.71 t/s
5060 Ti + 4060 Ti: 13.86 t/s

For qwen-flash I expected somewhat better results, that's all. There are more numbers involved and I'm aware of 4060 which must disappear from my setup.

[Volunteer-1]

@pepedombo With the Qwen3.6-27B Q6_K, two RTX 3090 cards achieve 68 t/s TG1024 on Windows (LM Studio quant) and 94 t/s on Linux (Unsloth quant). The RTX 3090 has 936 GB/s VRAM bandwidth, which is nearly identical to the 5070 Ti's 896 GB/s. You are massively underutilizing the 5070 Ti.

[pepedombo]

@Volunteer-1 I think you're mixing different test assumptions here.

[Volunteer-1]

@pepedombo What do you mean? I issued a test prompt to Qwen3.8-27B Q6_K that produced 3627 tokens at 58 t/s on Windows. https://ibb.co/0j0qf0M8