🐣 DeepSeek-V4-Flash-0731 (284B MoE) on 2× 24 GB — CPU expert offload, and why we are publishing no TPS numbers

[noonghunna]

We just shipped llamacpp/deepseek-flash-dual-q8 and llamacpp/deepseek-flash-dual-iq2 — a 284B MoE running on two 24 GB cards, by keeping 137 GiB of routed experts in host RAM and pinning a few expert bundles back onto the GPUs.

It's DeepSeek-V4-Flash-0731, GGUF quants and the DSpark drafter by @unsloth. Stock upstream llama.cpp — zero patches. All credit for the model goes to DeepSeek; unsloth made the quants and the drafter, and the DSpark runtime is ggml-org#25784.

A third slug, llamacpp/deepseek-flash-multi4-q8, ships for 4-card rigs — authored but never booted by us, because we have two cards.

---

🐣 Why this is Incubating, and why there are no TPS numbers

We are deliberately publishing no performance or quality figures. Not modesty — we don't think ours would predict yours.

Everything else in the catalog is GPU-resident, so the GPU is the variable and our numbers travel reasonably well. These slugs are not GPU-bound. Under expert offload the GPU sits idle most of the time — we measured SM at ~12.8% — waiting on ~40 sequential GPU↔CPU handoffs per token. What you get is set by:

• Host RAM bandwidth — how many channels are actually populated, and at what speed. A 4-channel box and an 8-channel box are different machines here, and populating 7 of 8 slots is different again.
• Core count and thread behaviour — this path is synchronization-bound, not compute-bound. On a sibling model we measured throughput inverting past the knee: -t 48 burned 94% CPU to deliver 31% of peak. More threads made it worse.
• PCIe topology and how the expert stream interleaves with everything else.

So a decode number from our rig would be close to meaningless on yours, and publishing one invites exactly the wrong comparison. We'd rather ship the config, tell you honestly what we don't know, and collect real numbers from real rigs.

Also genuinely untested: quality. No 8-pack has been run on either tier — and IQ2_XXS carries ~2.6-bit experts, where it matters most.

⚠️ Host RAM is a hard gate

slug	host RAM (worst case)
deepseek-flash-dual-q8	~146 GB
deepseek-flash-dual-iq2	~86 GB

This is not a recommendation — below it the box thrashes or OOMs. Preflight refuses to launch and tells you the number. The IQ2 tier exists precisely so the model fits a smaller box.

Why this model

Two tiers, and they are different capability profiles, not a speed/quality slider:

• Q8_K_XL — the quality tier. MXFP4 experts + BF16 attention.
• IQ2_XXS — the reach tier. 60 GB less host RAM, which is the whole point: it's what makes a 284B model fit a constrained box at all.

Three levers compose: CPU expert offload + partial residency (bundles pinned back onto the GPUs, sized automatically from your detected free VRAM) + the DSpark drafter.

Ships at 200K context, not 262K. At 262K it boots, reports healthy, answers a trivial prompt — and then dies on the first real prefill with a CUDA OOM. Which is its own lesson: on this class of config, a clean boot and a passing smoke test are both insufficient.

How the offload actually works

Worth spelling out, because there are two very different ways to run a model that doesn't fit, and they behave nothing alike.

The common approach is layer-granular — -ngl N puts N whole layers on the GPU and leaves the rest entirely on the CPU. Simple, and the engine can size it for you. But "the rest" includes those layers' attention, which is the latency-critical part.

These slugs are expert-granular instead:

-ngl 99                                          every layer's attention stays on GPU
-ot blk\.[0-9]+\.ffn_..._exps\.weight=CPU        only the routed experts page out

On this model the routed experts are 137 GiB — about 91% of the file — against a 11.7 GiB dense stack. So offloading experts alone moves nearly all the weight off the GPU while keeping all 43 layers of attention resident and fast. Layer-granular offload would banish attention for roughly half the model to get the same VRAM saving.

Then residency puts some experts back. Whatever VRAM is left after the dense stack, KV, compute buffers and the drafter gets filled with whole expert bundles, pinned to specific cards:

-ot blk\.(0)\.ffn_..._exps\.weight=CUDA0        pinned back onto GPU 0
-ot blk\.(42)\.ffn_..._exps\.weight=CUDA1       pinned back onto GPU 1
-ot blk\.[0-9]+\.ffn_..._exps\.weight=CPU       everything else

Two details that matter if you hand-edit this:

• -ot is first-match-wins. The device rules must come before the =CPU catch-all. Reverse them and every expert silently lands on CPU — no error, no warning, just a config that quietly isn't what it claims.
• A bundle must sit on the card that owns that layer's dense tensors, or every token pays a cross-PCIe hop. The launcher picks from each card's outer edge for exactly this reason, since the split point isn't observable from outside the engine.

You don't tune any of this. The launcher reads free VRAM per device and emits the rules, so a 24 GB card and a 32 GB card each get what they can hold — and a rig with more cards gets more experts resident, which also means less host RAM needed, since every pinned layer is a layer not in CPU memory.

Why this makes your numbers differ from ours

The trade is that a token's expert weights now stream from host RAM, and the GPU waits for them: ~40 sequential GPU↔CPU handoffs per token, with the GPU at ~12.8% SM utilisation (versus ~43% when fully resident). The card is idle, not busy.

Which is the whole reason we're not publishing throughput: the bottleneck has moved off the GPU and onto your memory subsystem and core count.

Getting it

# Reach tier (IQ2, ~85 GB) — the default, plus the required DSpark drafter
bash scripts/setup.sh deepseek-v4-flash-0731

# Quality tier (Q8, ~151 GB) instead
WEIGHT_KEY=deepseek-v4-flash-0731:unsloth-q8-kxl bash scripts/setup.sh deepseek-v4-flash-0731

It defaults to IQ2 deliberately — it's the tier most rigs can actually run, and guessing wrong costs you a 151 GB download. Either way setup.sh also fetches the DSpark drafter, which is not optional: both slugs pass -md and won't boot without it (the main GGUF carries no embedded prediction head, so DSpark is the only speculative path).

Engine floor: server-cuda-b10236 or newer. DSpark landed in ggml-org#25784 and is absent from earlier pins — the flag simply isn't understood. The catalog pin already satisfies this.

Run it

Command line — 🐣 incubating means non-functional-by-default, so --force is required:

bash scripts/switch.sh --force llamacpp/deepseek-flash-dual-q8    # :8030
bash scripts/switch.sh --force llamacpp/deepseek-flash-dual-iq2   # :8031

Expect 8–10 minutes to load (--no-mmap, and it's 151 GB). Served model name: deepseek-v4-flash.

In the c3 cockpit — they show in the Catalog tab with a 🐣 glyph, no toggle needed, and the catalog now carries a host RAM column plus offload = n-cpu-moe and spec = DSpark, so you can see the gate before you commit to a download (h is for deprecated / hardware-incompatible slugs, which these are not). Select one and launch with f — force, because incubating is non-functional by default.

You don't need to size residency yourself: the launcher reads free VRAM per device and injects the expert-bundle placement, so a 24 GB card and a 32 GB card each get what they can hold.

Run the evals — this is the ask

Two non-overlapping passes. Operational health:

bash scripts/report.sh --full        # verify + stress + soak + bench + agentic

Behavioural quality — the 8-pack, both reasoning modes:

pip install git+https://github.com/noonghunna/benchlocal-cli.git
git clone https://github.com/noonghunna/benchlocal-cli && bash benchlocal-cli/tools/build-sandboxes.sh

bash scripts/quality-test.sh --full --no-thinking
bash scripts/quality-test.sh --full --enable-thinking

⚠️ Two traps we hit, so you don't have to:

1. This is a thinking model. Probe it with a small max_tokens and content comes back empty while the reasoning trace eats the budget — it looks like a failure and isn't. Give it room.
2. A short probe proves nothing. Use a ≥10K-token prefill before you believe a config works. Ours passed boot and a trivial decode at 262K, then died on the first real request.

If a long generation times out on the operational leg, raise VERIFY_LONG_TIMEOUT — the default is sized for GPU-resident models and these run far slower.

What'd help

Your numbers, with your RAM configuration attached — channels populated, speed, core count, and the -t you used. That last part matters more than usual: we expect the thread knee to move with core count, and one rig can't map that curve.

Specifically useful:

• Decode and prefill on both tiers — and whether IQ2 or Q8 wins on your box. We're not confident the ranking is the same everywhere.
• A 4-card boot of multi4-q8. Nobody has run it. Residency should be much better with four cards, and host RAM should fall as card count rises — every layer pinned to a GPU is a layer not in host RAM. Untested.
• The 8-pack on either tier, especially IQ2 — quality on ~2.6-bit experts is the open question.

Credits

• Model — DeepSeek, DeepSeek-V4-Flash-0731
• GGUF quants + DSpark drafter — @unsloth
• Engine — ggml-org/llama.cpp; DSpark runtime #25784

[tomByrer]

I think publishing a range of T/s is still helpful; if folks are way off noting that someone else has better numbers might make them see what else they can tweak (like RAM speed in BIOS).

& if the T/s is too slow, some folks won't bother with it, like me with 1 RTX3090.

[noonghunna]

very difficult ask Tom! i'm not releasing my own numbers just because they won't hold on user rigs at all due to everyone having a different blend of cpu, ram and bus speed.

I hope some clarity will start coming in as folks report in their numbers, that was the idea.

[henrykrinkle01]

I have a range
0-40 tps 😁

[noonghunna]

that 0 i agree, but hey at 40 you are under-selling 😄

[clort81]

I do not understand how you get requirements of 150+GB SDRAM for the full weight model: Full size model runs fine with 128GB CPU + 48GB GPU here.

I have AMD Ryzen 7900X3D with 4x32GB DDR5 SDRAM, (G.SKILL DDR-6600 downsped to 3600mhz due to 4-sticks)

llama-cli -mli -t 6 --parallel 1 -fit on -b 4096 -ub 2048 --ctx-size 156000 --jinja -fa on --no-mmproj --load-mode mlock \
-m /media/sd/Models/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf

Approx 30k tokens into a run I am seeing
[ Prompt 108.7 t/s | Generation: 8.8 t/s ]

Experimental boosting of SDRAM up to 4400mt/s gets me to:
[ Prompt 160.5 t/s | Generation: 9.9 t/s ]

I'm still not understanding the layer loading you do, and why on earth you put the download logic into your run-script.

Is there a simple command to try that does NOT download anything??

[noonghunna]

Good questions, and your numbers are genuinely useful — that's the first external Q8 datapoint anyone has posted. Taking them in order:

Why we say ~146 GB when your 128 + 48 runs it fine: both numbers are right, because host RAM need is config-dependent, not model-fixed.

The routed experts are ~137 GiB. Host need ≈ 137 − (whatever your VRAM holds of them) + overhead. Your llama-cli line is leaner than our slug in three ways that all free VRAM for experts:

• no draft model (ours attaches the DSpark drafter — ~17 GB of VRAM across two cards)
• 156K context vs our 200K
• -ub 2048 vs our 4096 (roughly half the compute-buffer reserve)

So on your box, -fit on gets to park ~30 GiB of experts in VRAM, and host need lands around ~105 GiB — fits 128. In our slug's config on 2× 24 GB, after the drafter + KV + buffers there's room to pin back only a couple of expert bundles, so nearly all 137 GiB stays in host RAM → ~146 worst case. Same weights, different placement budget.

The layer loading is the same idea as your -fit on, made explicit: -ngl 99 keeps all 43 layers' attention on GPU, one -ot rule pages only the routed experts to CPU, then the launcher pins expert bundles back onto specific cards, sized from your detected free VRAM. We do it explicitly rather than letting the engine choose because a bundle must sit on the card that owns that layer's dense tensors (or every token pays a cross-PCIe hop), and because the drafter + 200K KV get budgeted first.

Download logic is not in the run script. setup.sh downloads (with SHA verification) and switch.sh runs — the run path never fetches anything. Preflight only checks the files exist and refuses with the exact path it expected.

Running it with the weights you already have — yes:

export MODEL_DIR=/media/sd/Models
mkdir -p "$MODEL_DIR/deepseek-v4-flash-0731-gguf/UD-Q8_K_XL"
# move or hardlink your 5 shards in there
# (a symlink pointing OUTSIDE $MODEL_DIR won't survive the Docker bind mount — preflight detects that and says so)

One file you don't have yet is genuinely required: the DSpark drafter (11 GB). Both slugs pass -md and won't boot without it — the main GGUF carries no embedded draft head. It's the repo-root Q8_0 file, not the BF16 inside the repo's dspark/ folder:

hf download unsloth/DeepSeek-V4-Flash-0731-GGUF dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf \
  --local-dir "$MODEL_DIR/deepseek-v4-flash-0731-gguf/dspark"
bash scripts/switch.sh --force llamacpp/deepseek-flash-dual-q8   # serves on :8030

The honest caveat for your box specifically: with 128 GB, the Q8 dual slug's preflight will refuse — its config (drafter + 200K ctx) genuinely needs more host RAM than your leaner run does on the same cards. Your options: the IQ2 slug (~86 GB host, different capability tier), or keep your llama-cli config — which is a well-matched setup for that hardware, not a workaround. And if that 48 GB is a single card: our shipped slugs assume 2+ CUDA devices, so your direct run is currently the right tool, full stop.

Two things in your post that match our findings exactly, for the record: the 3600→4400 RAM lift moving prefill +48% / decode +12.5% is the memory-bandwidth-bound story from the top post playing out live, and -t 6 on a 12-core is the same nproc/2 knee we ship as the default. If you're willing: bash scripts/report.sh now captures RAM channels populated + measured bandwidth — posting that with your numbers is exactly the data this thread exists to collect.

[clort81]

Good news: After more DRAM tuning (around 4400mt/s with tighter timings) I get:

At beginning
[ Prompt: 173.6 t/s | Generation 12.4 t/s ]

Later in gen, depending on what's being done
[ Prompt: 135.9 t/s | Generation 11.4 t/s ]

Invocation is naive llama-cli with -fit on as above.

Will run report.sh once i've studied it.

[noonghunna]

12.4 t/s generation from naive llama-cli -fit on on tuned DRAM is a solid result for zero config. Whenever you get to report.sh, the bundle will tell us your channel count and PCIe layout, which is what we'd need to say how much the slug's drafter + explicit placement would add on your box. No rush — the numbers you're posting are useful as-is.

[clort81]

Now down to 8-9.5 t/s and no idea what changed. LoL. Doesn't matter. I'm a slow person anyway. I getting good answers :)
Cheers.

[mgabor3141]

A first 4-card Q8 baseline for the configuration you said nobody had booted yet. This is not yet a recommended production recipe; we are still iterating on placement, batching, and speculative decoding.

Rig

• 4x RTX 3090, 96 GiB total, 220 W each, PCIe 3.0, no NVLink
• Threadripper 1950X, 16C/32T
• 128 GiB DDR4-3200, all 8 DIMM slots populated, 4-channel UMA
• 51.4-52.0 GB/s STREAM Triad
• Unsloth UD-Q8_K_XL, 150.75 GiB, 43 layers, 256 experts, top-6

This used plain llama.cpp with CPU expert offload, not the multi4-q8 launcher, and without the DSpark drafter. It therefore should not be treated as a benchmark of the shipped slug.

Configuration

-ngl 999 --n-cpu-moe 26 --tensor-split 28,5,5,5
-b 4096 -ub 4096 --no-mmap --flash-attn on --jinja
--ctx-size 131072 --parallel 1

Club report result

One canonical report.sh --full run, with ENABLE_THINKING=1, produced:

• verify: PASS
• stress: 7/8; check 6 failed at the client’s hardcoded 600-second timeout
• soak: PASS
• bench: PASS
• agentic: PASS
• overall exit: 1, because of that stress timeout

The completed run measured approximately:

• decode: 11.59-14.78 tok/s, depending on workload
• TTFT: 222-297 ms
• prefill at 10K: 248.9 tok/s
• prefill at 90K: 221.4 tok/s
• context ceiling test: 120,477 tokens
• soak VRAM growth: 0 MiB

The failed stress check was then run separately with STRESS_REASONING_TIMEOUT_S=2400; it completed 7,630 reasoning tokens and exited 0. That is a separate diagnostic rerun, not part of the full-suite result.

The stress script should make this timeout configurable, matching the existing VERIFY_LONG_TIMEOUT pattern:

- --max-time 600
+ --max-time "${STRESS_REASONING_TIMEOUT_S:-600}"

The important practical finding so far is that -ub 4096 is essential for CPU-offloaded MoE. A short exploratory comparison measured 39.65 tok/s prefill at -ub 256 versus 256.70 tok/s at -ub 4096, but those figures came from a separate local benchmark, not the club bench, and should not be compared directly with the report values.

This is an exploratory baseline only. We are now using short iteration tests before running another long report, and will publish a final usable configuration after placement and speculative-decoding tests converge.

[noonghunna]

This is the first 4-card Q8 datapoint anyone has posted — thank you. And your rig makes it unusually comparable: your 51.4–52.0 GB/s Triad is within noise of the reference rig's own 4-channel-era measurement (51.9), so your no-drafter decode of 11.6–14.8 slots directly into the bandwidth model we've been building. That's a controlled experiment neither of us had to design.

Three quick returns on your findings:

• The stress check-6 timeout: you were right, and it's fixed — fix(verify-stress): make probe-6 reasoning timeout configurable #934 ships STRESS_REASONING_TIMEOUT_S with your exact env name, so the rerun workflow you already used works unchanged against master.
• Your -ub 4096 finding independently replicates the shipped default — the multi4-q8 slug carries -b 4096 -ub 4096 for exactly the reason you measured (6.5× prefill vs -ub 256 in your probe). Good to have it confirmed from a second rig.
• Ceiling caution that bit us: on this model, graph/compute allocation scales with declared context while KV stays nearly free — which means a config can boot clean at a high ctx and die on its first deep prefill (our 262K dual-card config did exactly that). Your 120,477 pass at 131K declared with ~6 GiB spare is the honest way to establish the ceiling.

When your placement iteration converges, two arms worth comparing against your --tensor-split 28,5,5,5: the shipped multi4-q8 slug (contiguous layer assignment + launcher-sized residency, fixed for 4-card after the first community boot — and your 128 GiB clears its gate), and a DSpark drafter arm (11 GB download; on the dual-card Q8 config it's the single largest decode lever we've measured, though the gain is content- and sampling-dependent). Curious what drove the 28,5,5,5 skew — if you have numbers vs an even split, that's a placement datapoint nobody has.

[noonghunna]

First canonical numbers for both shipped dual slugs — from the reference rig, at last.

These were owed since the slugs shipped: both went out 🐣 Incubating with "no performance numbers published," because our host was quietly running 7 of 8 memory channels (the reason we also deferred the moe-cache RFC measurements — a bandwidth-starved host doesn't just make offload numbers low, it biases them). The DIMM is replaced and verified (STREAM Triad back to ~116 GB/s), so here is what the shipped composes actually do on the reference 2×3090.

Rig: 2×RTX 3090 (24 GB, PCIe-only), EPYC 7543 guest (32 cores), 8-channel DDR4-3200, 196 GB RAM. Stock llama.cpp b10236, launcher defaults (DSpark drafter + auto residency), 200K ctx, -t 28 (tuned for this host's topology — the launcher's auto value is correct for typical desktop CPUs; don't copy the 28).

llamacpp/deepseek-flash-dual-q8 — quality tier (~146 GB host RAM gate)

3-boot medians, repo-canonical sampling (temp 0.6 / top_p 0.95):

metric	result
decode, narrative	17.1 t/s (draft acceptance ~0.33)
decode, code	26.9 t/s (acceptance ~0.70)
TTFT (short prompt)	169 ms
prefill @10k / @90k	369 / 287 t/s (CV ≤0.6% across boots)
VRAM	23.3 + 22.8 GiB, no swap
greedy protocol (temp 0, /completion)	35.2 t/s — different protocol, never compare to the sampled rows

llamacpp/deepseek-flash-dual-iq2 — reach tier (~86 GB host RAM gate)

Same protocol, same host:

metric	result
decode, narrative / code	15.4 / 24.3 t/s (acceptance mean 0.46)
TTFT	~165 ms
prefill @10k / @90k	436 / 311 t/s
greedy protocol	32.2 t/s

What the pairing tells you

• Q8 is both the quality AND the decode-speed tier here (~10% faster than IQ2): the drafter accepts far fewer tokens against 2.6-bit experts, and that costs more than the smaller expert reads save. IQ2's case is the RAM gate (86 vs 146 GB) and prefill (+18% @10k — less weight volume over PCIe).
• Decode is strongly content-dependent through the drafter (17 narrative vs 27 code on the same boot) — when comparing rigs in this thread, compare like prompts.
• One honest asterisk: in 2 of 3 Q8 boots, a host-side contention mode we're still chasing halved sustained code decode for a few consecutive runs (narrative, greedy, and prefill never affected; acceptance normal; zero guest-visible cause). Numbers above are from clean boots; the anomaly is documented in the repo and under investigation.

Both slugs stay 🐣 Incubating until the quality 8-pack lands — that's next. BENCHMARKS.md rows 2–3 carry the full detail as of today's commit.

@mgabor3141 — your 4-channel Triad (51.4–52.0) is within noise of what this rig measured in its own 4-channel era, so your no-drafter Q8 numbers and these drafter-attached ones now bracket the DSpark contribution on near-identical host bandwidth. That's a comparison nobody planned and it's the most useful accident in this thread.

[clort81]

What this tells me is i want noonghunna's 8-channel DDR4-3200 :D

That's nominal 200GB/s RAM vs my tweaked 58GB/s. So my experts are sitting on memory 28% as fast as the 8CH DDR4. At my safe (no speculative) decode of 8.8 vs 27 t/s that's close to linearly scaling with memory speed.

However if i go with buun-llama-cpp I'm pulling 12.5 t/s until a few k context, then it crashes. So I'm hoping to get that running someday. Will share more then. Delete this if it's a waste of space. Thanks and cheers.

[shaxs]

DeepSeek V4-Flash on dual RTX 3090s — a tuning campaign with real numbers

Sharing our experience tuning DeepSeek V4-Flash (the 284B MoE, Q8-class) on a
two-3090 box, since most of what's posted is either "it works" or raw
spec-sheet numbers. This is a full controlled sweep with measurements,
including the parts that surprised us.

Hardware and baseline

CPU:          Intel Core i9-7900X (10C/20T, Skylake-X)
Memory:       256 GB DDR4-3200 (quad-channel)
GPUs:         2× RTX 3090 24 GB (PCIe 3.0, no P2P)
STREAM:       ~49 GB/s host memory bandwidth

Where we started: ~7.2 answer-tokens/s in real use.

Where we ended: 16.93 TPS median (code workload) after tuning, and a clear
picture of what matters and what doesn't.

Engine and model

• Club's llamacpp-club3090 MoE expert-cache fork
• DeepSeek-V4-Flash-0731, Unsloth UD-Q8_K_XL (284B MoE, ~161 GB model)
• DSpark Q8_0 drafter, CPU-resident (-devd none)
• MoE expert cache with hot experts in unused VRAM, cold experts in system RAM
• Served context 204,800, q8_0 K/V

What we swept and what we found

1. Speculative depth: 2 is the sweet spot

Depth	Answer TPS	Acceptance	MoE hit
2	14.42	77.6%	56.6%
3	14.37	70.4%	57.2%
4	13.73	67.0%	57.2%
5	12.76	58.0%	58.3%

Depth 5 had the highest MoE cache hit rate but the lowest throughput —
the extra speculative work cost more than the cache gains saved. Depth 2 wins
(+13% over depth 5).

2. Code vs narrative: acceptance is the differentiator

Prompt	Median TPS	DSpark acceptance	MoE hit
Narrative	13.77	73.3%	59.4%
Code	15.86	93.4%	58.3%

Code is 15% faster despite a slightly lower cache hit rate. The drafter
simply predicts code better than prose. Cache hit rate is not the whole story.

3. CPU threads: the club default is safe but not optimal

Threads	Median TPS
5	10.87
10 (nproc/2 default)	15.56
14	16.45
15	16.93
16	16.83
20	11.22

The real knee is 15–16 threads. Thread 15 beats the nproc/2 default by
+8.75%. Using all 20 logical CPUs collapses throughput by 33% (SMT contention
on a memory-bandwidth-bound workload). On this MoE-with-CPU-offload setup,
threads matter a lot — it's not a "leave it alone" knob.

4. Batch/ubatch 2048 vs 4096: a prompt-heavy profile, not a universal win

Metric	2048	4096	Delta
Decode TPS	16.74	15.78	−5.76%
MoE hit	57.9%	53.2%	−4.7 pts
Uncached 8K prefill	122.9 tok/s	176.0 tok/s	+43.25%
TTFT (8K uncached)	66.7 s	46.6 s	−30.15%

4096 is a legitimate prompt-heavy profile: big uncached prompts, short
answers, TTFT-sensitive. For sustained decode, 2048 is better. We kept 2048.

The honest caveats

• Our absolute numbers are lower than the reference rig's. The Club 3090
  reference machine posts ~22 TPS narrative / ~34 TPS code with 116 GB/s host
  bandwidth. Our Skylake-X host measures ~49 GB/s STREAM. Across every metric
  we land at roughly 42–45% of reference — which tracks the host memory
  bandwidth ratio, not the engine.
• This workload is CPU/RAM-bound, not GPU-bound. The GPUs sit mostly idle
  while llama-server spins CPU feeding experts from RAM. More VRAM isn't the
  answer; faster host memory bandwidth is.
• PCIe 3.0 and no P2P between cards add to the transfer cost.

What we'd tell others

1. Sweep thread count on your host. The nproc/2 default is a safe floor,
   not an optimum, on memory-bandwidth-bound MoE CPU-offload. We gained ~9%
   just by finding the knee.
2. Don't over-tune speculative depth. Higher depth ≠ faster; depth 2 beat
   depth 5 here despite worse cache behavior.
3. Measure code and prose separately. Acceptance rates differ by 20 points
   and dominate the difference.
4. Check your host memory bandwidth first. If your machine has <100 GB/s
   STREAM, expect MoE CPU-offload decode to sit well below reference rigs no
   matter what you do in llama.cpp.

The fork itself is solid — Club's MoE cache integration matched the exact
leloch commit in a controlled A/B (within 0.4%). If you're on a high-bandwidth
platform, this stack should fly.

NOTE: Written with Deepseek Flash v4 :P

[shaxs]

I should also note that I just bought a WRX80 Threadripper Pro 3975WX system and will be attempting the same experiment with higher memory bandwidth to see how much the needle moves.

[paulp83]

I got the same motherboard and CPU on the way. What CPU cooling solution are you going with?

[shaxs]

Ive got the Supermicro M12SWA-TF on the way and will be using a Noctua NH-U14S TR4-SP3 for cooling.

[paulp83]

See #1105
report --full & quality on/off