Experiment: fixed GPU expert working set with Native CPU miss execution for Qwen sparse MoE (16 GiB GPU, 16K-262K context)

[zhaoyilun]

I've been experimenting with tensor-structured sparse-MoE placement in llama.cpp and put together a result package I'd like feedback on. The starting point was a simple observation: for a sparse MoE that's way bigger than VRAM, the question isn't "which layers go to CPU" but "which tensors". I keep everything that suits CUDA (attention, norms, router, shared/dense FFN) on the GPU and leave routed expert weights in host RAM. At model load, a fixed subset of packed experts gets materialized in a GPU pool; graph construction splits the selected expert IDs into resident hits (GPU) and misses (Native CPU expert), then merges the weighted outputs.

My test box: RTX 4080 Super 16 GiB, i9-14900KF, 96 GiB DDR5-4000, Windows, Qwen3.8-Flash-Next 176.94B IQ4_XS-family GGUF (QWEN4EXP arch), pinned llama.cpp revision 4e97ac8.

What I measured at 16K context (three 500-token runs each):

• same-path Native expert-only CPU control: 15.135 tok/s
• fixed cache 4 GiB: 16.096 tok/s
• fixed cache 6 GiB: 17.275 tok/s
• fixed cache 8 GiB: 18.753 tok/s

So +23.91% over the control at 8 GiB. Short-context repeats: 19.531 tok/s at 2K, 18.321 at 8K. One thing that surprised me: CUDA Graph is doing a lot of heavy lifting here — the same 8 GiB config with graphs off drops to 13.567 tok/s.

I also ran the context ladder out to 262K with mixed KV quantization (q8_0 K / q4_0 V):

• 262K, q8-q4 KV + 4 GiB pool: 5.776 tok/s (p50 170.7 ms / p95 197.2 ms), peak 13,611 MiB
• q4-q4 KV + 6 GiB pool: 5.649 tok/s — the q8-q4 + 4 GiB combo is slightly faster while using less VRAM
• retrieval at 262K: 5/5 with q8-q4 (4/5 with q4-q4, and that miss was an output-budget truncation, not a retrieval failure)
• restoring a 262K session takes about 14 s

Being honest about this part: the cache gain shrinks from +23.9% at 16K to about +4.6% at 262K. Attention/KV dominates per-token time out there, so the expert working set stops being the bottleneck.

The most painful lesson was a WDDM one. q8-q4 KV with a 6 GiB pool at 262K fits on paper — a short 5+5-token run of it looked fine at 5.508 tok/s with 637 MiB headroom — but a 250-token run collapsed to 0.71 tok/s mean with p95 at 7.5 s, because total demand crossed physical VRAM and Windows started paging through shared memory. And it's nondeterministic: whether you collapse depends on residency. That's why I landed on keeping at least ~1 GiB of allocator headroom as a hard rule, with the pool stepping down 8 GiB → 6 GiB → 4 GiB as context goes 128K → 192K → 262K.

I also tried adjacent-token selective H2D promotion (packed 48×10 route observation tensor). Hit rate went from ~31% to 61.78%, and every threshold setting made throughput worse — up to 114.45 MB/token moved and 22.89 ms/token inside the promotion routine alone. Cache optimization is about end-to-end service time, not hit rate. That path stays in the tree but disabled by default (threshold=-1).

On prefill: physical ubatch 2K → 8K took the same 16K prompt from 287.387 to 524.003 tok/s (three-run mean; 617.342 single-run at 32K). At 262K cold prefill is still 786.9 s (331.8 tok/s), which is the biggest single wait in the system. I'm actively working on expert bucketing + grouped GEMM to push past 1000 tok/s — the first implementation round hasn't landed yet, and measured results will go into the repo as they arrive.

Pieces I think could be upstreamed, and where I'd genuinely like feedback:

1. expert tensor placement override
2. fixed packed expert working set
3. public hit/miss/fallback counters
4. packed route observation for diagnostics

Before turning any of this into a PR, I'd rather hear where people think a fixed expert working-set policy belongs in llama.cpp, whether the API shape makes sense, and how to generalize beyond this one model.

To be clear about lineage: this builds on llama.cpp itself as the base; the expert-slot streaming idea came from reading Slotstream (carloslfu — same model on a Mac, streamed from SSD); the CPU-executes-missed-experts trade is straight out of Fiddler (efeslab, arXiv 2402.07033); KTransformers was my heterogeneous-execution comparison. No source copied from any of them.

Everything above ran on the full llama.cpp path, and every number has per-run JSON + stderr logs behind it: https://github.com/zhaoyilun/moe-working-set-16gb (patch, runners, ~1150 evidence files, build/repro docs). Limitations are listed there too — one machine, one model family, and a few matrix cells still unmeasured.

[zhaoyilun]

Quick update on the prefill item from the original post: the first round of the expert-staging prefill runtime measured 720.8 tok/s on the 16K prompt at physical ubatch 6144 — +27.3% over its same-condition control, with numerics agreement at cosine 0.9999928.

The shape that works is: an eval-callback barrier between graph stages, shared VRAM staging windows, and a pinned pack/DMA arena — full expert bucketing + grouped GEMM is still the road to 1000+, that part continues. Also apologies for the broken title earlier (a scripting artifact on my side) — fixed now.

Run data and the design notes are in the repo under evidence/results/prefill-grouped-gate/, driver scripts in evidence/profiling-scripts/.

[zhaoyilun]

Formal gate results are in — superseding the quick numbers in my previous comment (the +27.3% there was against an earlier baseline; the gate report re-ran everything).

• 16K prefill at physical ubatch 6144: 725.667 tok/s (three runs, CV 0.13%). Same-build same-ubatch control: 516.9 → +40.4%. Best historical config (ubatch 8192): 566.7 → +28.1%.
• Numerics: first-token argmax and the full 8-token greedy sequence are identical across all three runs. Logits cosine 0.9999928 for a single ubatch, 0.9948 across the whole 16K prompt (three ubatches of KV propagation amplify the drift — the greedy-sequence equality is the stronger check).

The mechanism: router-post anchor node → eval-callback barrier → per-layer unique-expert pack into a pinned arena → bulk DMA into shared VRAM windows (6 windows, 512 slots, ~2.8 GiB) → mapping rewrite → grouped MMQ on a large-batch pure-GPU graph.

Two negative boundaries from the same gate, since they matter for anyone generalizing this:

1. 262K staging is a NO-GO. Staging windows + KV measured 15,840 MiB with 536 MiB headroom — under the 1,024 MiB safety line this project treats as hard — and the attempt degraded into WDDM shared-memory crawling (terminated after 40 min at full GPU load, pathological throughput). 262K prefill stays on the control path at 331.8–364.2 tok/s.
2. The obvious next step toward 950+ (persisting expert weights across ubatches instead of re-staging) is structurally dead on 16 GiB: per-layer independent windows would total 22–41 GiB.

Decode regression check: no binary-level regression — 2K decode 18.979 vs 19.531 tok/s on the old binary (-2.8%, environment-level); a 16K plan-mode reading of 15.0 was taken after ~20 h of continuous load and is attributed to environment state, with a cold-restart re-measure queued before citing decode numbers from this build.

Full gate report (Chinese, measured-labeled) and all per-run data: evidence/results/prefill-grouped-gate/.

[zhaoyilun]

Update — second model, stock llama.cpp, same machine. A data point for the "how does this generalize beyond one model" question at the end of the main post.

I put the same 16 GiB host to daily use with a much smaller hybrid-attention MoE: Qwen3.6-35B-A3B (Q4_K_M, 21.2 GB, 40 layers = 10 full attention + 30 linear attention, 256 experts / 8 active per token, 262K native context) on unmodified llama.cpp b10666 — --n-cpu-moe 24 (16 layers of experts in host RAM), q8_0 KV, -ub 2048. None of my patches involved.

• 63K cold prefill went from 184.6 to 1,460 tok/s (7.9×) purely by raising the physical ubatch 128 → 2,048, with decode unchanged. This is the consumer-side version of the per-(layer, ubatch) expert transfer cost in the main post: CPU-resident expert tensors get copied once per layer per ubatch regardless of ubatch size, so more tokens per ubatch is pure amortization.
• Full 260K-context cold prefill (260,000 prompt tokens, greedy): 238 s = 1,091 tok/s, and 512-token decode at 34.3 tok/s at that context. Decode from 2.4K → 260K falls only 57.5 → 34.3 tok/s because attention is 10 of 40 layers; the dense Qwen3.8-27B on the same card goes 66 → 5.4 tok/s by 45K.
• The WDDM headroom rule from the main post held here too: the 260K run sat at ~1.0 GiB minimum free VRAM for 4 minutes under full load without collapse, while a 4,096-token-ubatch variant (323 MiB free) and an --n-cpu-moe 12 variant (453 MiB free) both degraded pathologically.

Per-run JSON and the bench script: evidence/results/a3b-262k-context/ in the repo.

To be clear about what this does and doesn't validate: a 21 GB model with static layer split + a big ubatch is already "good enough" on a 16 GB card, so the fixed working-set policy isn't needed at this size. The overlap that matters is the transfer-cost model — whether you use per-layer unique-expert staging windows or a plain large ubatch, per-(layer, ubatch) expert traffic is the quantity to drive down, and it dominates prefill on both a 176.94B model and a 35B one.

[zhaoyilun]

Follow-up on the second model: we grafted the MTP/NextN head from the official Qwen3.6-35B-A3B GGUF onto a community fine-tune of the same base (uncensored merge) at the tensor level, since fine-tunes ship without the head and can't use draft-mtp at all. Two non-obvious bits: the head is exactly 20 blk.40.* tensors (pure addition), and qwen35moe.block_count must be rewritten 40→41 or the loader fails looking for blk.39.nextn.*.

Measured on the same 16 GiB host, official b10819, --n-cpu-moe 24, q8_0 KV, -ub 2048, 128K: decode 62.2 → 79.4–79.7 tok/s (+28%), prefill 2,328–2,427 tok/s, draft acceptance 0.851–0.891 / mean 2.70–2.77 with the stock-paired head running against an abliterated merge — i.e. the MTP head tolerates substantial weight drift in the target, so heads are portable across fine-tunes of the same base (speculative decoding stays lossless regardless; a mismatched head only lowers acceptance).

One more field note: on current master, --n-cpu-moe + mmap costs ~60% of prefill unless --load-mode none is set (the server warns about tensor overrides with mmap; measured 1,293 → 2,145 t/s at 22.7K on the same binary). Might be worth making that combination warn louder or default differently.

Method, graft script, per-run JSON and the negative results (WSL2 decode tax −12…−21%, MSVC self-build regressions on both this project and ik_llama.cpp): https://github.com/zhaoyilun/moe-working-set-16gb (docs/a3b-mtp-deployment.md).

[mladvocacyteam-lab]

just wanted to say thanks