Field notes from pure-CPU MoE deployments on two Windows machines (dual-socket Xeon / Threadripper PRO): MinGW+OpenMP inversion, -ngl 0 ≠ no GPU, a decode regression between builds, and what did — and didn't — predict speed across ~10 checkpoints

[geer1895]

Hello everyone,

We recently spent about two weeks deploying large MoE models as pure-CPU inference backends for an agent workload (llama-server + OpenAI-compatible API), first on an older workstation and then on a larger server. The project succeeded largely because of this codebase — thank you for building and maintaining it. We kept careful measurements along the way and wanted to give some observations back to the project.

A few caveats up front: we are application engineers, not kernel developers; each machine is a sample size of one and these observations may not generalize; and it is quite possible we are misusing some flags — corrections, or "this is already documented, see X", are very welcome.

Setup

• Machine A (most numbers below): 2× Xeon Platinum 8369B (Ice Lake, AVX-512), 64 physical cores across 2 sockets; 1 TB DDR4-3200, 16 channels (a simple custom membench measured ~158 GB/s sequential-read peak); Windows 10; no GPU. Build: recent master, w64devkit (MinGW-w64 GCC), -DGGML_NATIVE=ON -DGGML_OPENMP=OFF, Release.
• Machine B (several findings below): Ryzen Threadripper PRO 3975WX (Zen2, AVX2 only, 8 CCDs), 128 GB DDR4-2666 4-channel (~47 GB/s measured peak), Quadro P2000 5 GB (Pascal, sm_61), Windows Server 2019. Mostly prebuilt Windows binaries (an b11160-era CUDA build, plus MinGW self-builds for comparison).
• Workload: production model on machine A is a ~73 GB MoE GGUF (≈177B total / ≈3B active, Qwen3.8-Flash-Next UD-Q2_K_XL), llama-server at 128K context, -t 24 -tb 56 --numa distribute -lm none -fa on -ctk/-ctv q8_0. Screening before settling covered 3 model families × 10 quantization variants on 2 engines; ~1 TB of checkpoints across ~6 families were downloaded and evaluated overall.
• Method: single-process guard before every run, alternating A/B repeats (noise floor ~7% on machine A, up to ±15% on machine B), client wall-clock always cross-checked against the server's timings object, and prefill numbers taken from long prompts only (short-prompt prefill is dominated by first-graph initialization and misled us more than once).

1. MinGW + OpenMP thread inversion (first seen on machine B, quantified on A)

With OpenMP enabled (the CMake default) under a MinGW self-build, decode throughput decreased as thread count rose, and single-threaded prefill was severely degraded (~19× worse); it applied to non-MoE models too. -DGGML_OPENMP=OFF fixed it — differences of one to two orders of magnitude in the worst configurations. We suspect libgomp behavior on Windows; if this is already well understood, a one-line note in the Windows build docs might spare others the same confusion. (For completeness: MSVC builds were also slower than OpenMP-free MinGW across all 16 pp/tg × thread combinations we tried on machine A — decode as low as ~55% of the MinGW numbers — with the caveat that this was a default CMake MSVC build, no /GL / /fp:fast.)

2. Batch threads (-tb) matter a lot for prefill — and have a cliff

On machine A, sweeping prefill batch threads at -t 24: -tb 56 was best at every context depth (e.g. 84.6 vs 63.0 t/s at depth 0), while -tb 64 collapsed to ~5 t/s (−92–94%), reproducibly. Our (possibly wrong) explanation: at 64 batch threads the working set spans both sockets — our NUMA microbenchmark shows remote-node bandwidth at ~41–42% of local, and aggregate memory bandwidth drops ~31% around 56+ threads. On machine B the same knob was the single biggest prefill win we found: --threads-batch 24 gave +23%/+31% on short/long prompts. Tooling nits on both machines: llama-bench rejects -tb ("invalid parameter"), and llama-cli given -tb exits and then hangs waiting on stdin — llama-server was the only workable harness for -tb sweeps.

3. Prompt processing and generation scale oppositely with -np

• Decode: -np 4 aggregated ~2.06× throughput on machine A (each request individually slowing to ~52% of solo), as expected.
• Prefill: a single ~5.8K-token cold request ran ~88 t/s solo but only ~6.8 t/s (−92%) under -np 4. Prompt processing appears to be partitioned rather than parallelized — presumably because prefill already saturates the cores that decode leaves idle.

For interactive agent traffic (long shared prefix, cache_prompt=true), -np 1 + FIFO was strictly better in our tests: the first client ran at solo speed and the second simply queued without degrading it. This may be well understood by the team, but "more parallel slots = more throughput" reads fairly naturally from the docs, and for prefill-heavy workloads our data says the opposite — maybe worth a line in the server documentation. (It also matches what other CPU-inference projects report: extra cores mostly help prompt evaluation, not decode.)

4. Model screening: bytes-per-token predicted speed; per-byte efficiency and kernel coverage did not

Since we tried quite a few models, here is the most transferable thing we learned. Across ~10 checkpoints, the cleanest predictor of decode speed was simply how many bytes each token reads (active weights) — not parameter count, nominal bitrate, or per-byte efficiency. All on machine A:

checkpoint	type	GB/token	effective BW	decode
Qwen3.8-27B NVFP4	dense	14.85	82–87 GB/s	~5.6–5.9 t/s
MiniMax-M2.5 MXFP4	MoE	6.71	86.9 GB/s	12.95 t/s
UD-Q3_K_XL	MoE	7.70	74.8 GB/s	9.72 t/s
UD-Q2_K_XL (final)	MoE	5.70	62–63 GB/s	~11.3 t/s
DeepSeek-V4-Flash IQ2XXS	MoE	9.89	60.4 GB/s	6.11 t/s

(All numbers single-machine and approximate — treat the last digit as decorative. A 190 GB GLM-5.3-Flash checkpoint was also evaluated but was simply too large to be useful here: 4.25 t/s decode.)

Three sub-findings, each of which cost us a wrong turn before we measured properly:

• Every MoE checkpoint landed in a narrow 11.5–16.5 ms-per-GB band, regardless of model family, quant format, or kernel coverage. The access pattern (expert-block scatter across a 70+ GB weight set) seemed to dominate everything else. The one MoE with notably larger expert blocks reached ~87 GB/s effective — essentially dense-class streaming.
• Per-byte efficiency was actively misleading. UD-Q3_K_XL is ~18% more efficient per byte than our final UD-Q2_K_XL but reads 35% more bytes per token — net −12.7% decode. The NVFP4 variant had the best per-byte efficiency of anything we measured (~87–95 GB/s on our two measurement passes) yet still lost ~12% overall to a +61% increase in bytes/token. speed ≈ effective bandwidth ÷ bytes per token; watching only the numerator inverts the conclusion.
• Kernel-coverage reasoning from source code was falsified twice in our data: a format with good kernel coverage (q2_K-class) was our slowest candidate at comparable scale, and NVFP4 — which looks expensive per-nibble when reading the source — was among the fastest per byte. Related small observation, offered with humility: the x86 repack GEMV path appears to be AVX2-guarded only (ggml-cpu/arch/x86/repack.cpp) even on AVX-512 hardware. We deliberately did not modify it, since kernels were not the dominant factor in our measurements. (Separately: on machine B, a newly-introduced format with only scalar/limited CPU kernel paths measured prefill −76% despite being smaller — "quant format × kernel coverage" as a joint property, never the format alone.)

Dense vs. MoE on the same machine, for completeness: the dense checkpoint streamed at ~1.33× the per-byte efficiency of our MoE, but the MoE was ~1.96× faster in absolute decode because it reads ~1/2.6 of the bytes per token.

5. Fastest decoder ≠ fastest agent: one checkpoint's prefill collapsed with context depth

MiniMax-M2.5 had the fastest decode of everything we tested (12.95 t/s, +16% over our final choice) — but its prompt processing degraded much faster with context depth: −38% from depth 0 to 8K vs −14% for the model we picked (same protocol), and a deeper sweep dropped ~70% from 2K to 10K. Since agent workloads are overwhelmingly prompt processing, the "faster" model was meaningfully slower per agent turn. Decode benchmarks alone would have picked the wrong model for this use case.

6. A cautionary engine-comparison data point (and a compliment to mainline)

We also benchmarked a well-known performance fork: its prefill was 1.88–2.47× faster, and its perplexity matched mainline to within 0.117%, so we nearly adopted it. But at temperature=0 generation, mainline passed 4/4 of our fixed-prompt checks while the fork passed 0/4 (finish_reason=length, empty content) — a generation-termination breakage that perplexity cannot see. We share this less as criticism of that fork and more as (a) credit to mainline for behaving correctly, and (b) a reminder to the community that ppl-equivalence does not imply generation-equivalence.

7. Server gotchas that cost us real time

• -ngl 0 does not mean "CPU-only" — KV cache still gets offloaded (machine B). With a 5 GB Pascal GPU present, 32K context worked by luck (KV ≈ 0.4 GB fits), but at 256K the KV offload triggered a CUDA OOM and the failed allocation path dragged system free memory to ~0 GB, briefly taking another production service on the box down with it. The fix was -dev none (or --no-kv-offload). Two thoughts, offered meekly: a docs line saying "-ngl 0 alone does not disable the GPU", and possibly a startup warning when a GPU is present but -ngl 0 with a very large -c. The "it worked at 32K" part is what scares us — small-scale success masked the misconfiguration completely.
• Default mmap collapsed prefill ~10× whenever the model exceeded the page cache (machine B: 84 GB model on 128 GB RAM → prefill 1.62 t/s; -lm none → 16.19 t/s, decode +43%). The same flag also migrated (--no-mmap → -lm/--load-mode), which briefly confused us.
• -c is divided across parallel slots (machine B): we set -c 262144 and got 4 × 64K because the server came up with 4 slots; --parallel 1 was required and we only caught it via /props. Probably documented, but it produced a silent "why is my context short" moment worth a FAQ entry.
• Thinking models + small max_tokens = empty content: the reasoning consumes the entire output budget and the answer is empty (finish_reason=length). We were rescued by the server-side --reasoning-budget flag (a small budget on one machine, 0 as the default on the other — the latter turned a reproducible "empty reply at max_tokens=16" into a clean pass). Also, thank you for exposing reasoning_content separately on the OpenAI-compatible endpoint — our evaluation harness would have scored several correct answers as failures without it.
• --cache-reuse appeared to be a no-op while --cache-prompt is active (its default): 0 vs 256 produced byte-identical results on two machines; as far as we could tell it only becomes relevant for partial-prefix matching. A one-line docs clarification might help.
• A cluster of knobs was speed-neutral for us on both machines (within noise): KV cache quant type (f16/q8_0/q4_0 — we still run q8_0 for memory), ubatch, logical batch size, CPU flash-attention on/off, and mlock. Listing these only as "don't be surprised if these don't move the needle on CPU-bound MoE decode" — we spent real hours on them.

8. A decode regression we noticed between builds (machine B)

Comparing two prebuilt Windows builds 273 builds apart (b11160 vs b11433), same model and flags, alternating runs: prefill improved +17.3% (pp512 49.8 → 58.4) but decode regressed −6.0% (tg128 8.00 → 7.52) — and −11.3% at each build's own best thread count (old best t=20, new best t=16, and the new best is still lower). At low thread counts the two builds were nearly identical, so the regression seems confined to the multi-threaded decode path; both builds exposed identical flag sets, so it is not a defaults change. We did not bisect and the two builds came from different packagers, so toolchain could be a confound — but if multi-threaded CPU decode regression around that range rings any bells for the team, we kept the raw logs and can rerun anything. (Our takeaway was "newer ≠ faster, re-bench before upgrading", which we now apply religiously.)

9. Bonus data point: speculative decoding sign flip — and scenario-dependent acceptance

MTP-style speculative decoding gave +21% on short prompts on machine B but only ×1.34 at 8K of context; on machine A it hurt — four draft configurations tested, the best reached only +3.7% despite an ~82% acceptance rate, and production settings cost −16.6% (256K: −20%). On a bandwidth-bound iGPU mini-PC running the same engine family, the same feature gave +66%. Two smaller notes: acceptance rate varied wildly by scenario on machine B (52% short Q&A, 66% long-document needle retrieval, 87.5% long prompts), and an n-gram-based drafter measured 0% acceptance — worse than useless at our scale. All consistent with "speculative decoding amortizes bandwidth, not per-token latency", and with "benchmark on your own hardware".

10. Minor: extending GGUF metadata

While adding support for a couple of new architectures we learned that LLM_KV key prefixes derive from the canonical enum name, not from any string in the file — so adding a string alias alone does not work; the arch needs an enum slot. Also, probing for missing keys by wrapping get_key_or_arr in try/catch produced an access violation (0xC0000005) in our MinGW build; a non-throwing lookup helper would be a friendlier idiom (we ended up reading array lengths with a get_arr_n-style call first).

We kept raw logs for all of the above and would be happy to rerun anything (thread/depth sweeps, NUMA modes, A/B on specific patches, or additional checkpoints if useful) if that would ever help. Thanks again for a wonderful project — it made unusual deployments (a 73 GB MoE on CPU only) genuinely practical.