[Windows/Zen2 CPU-only] MinGW build decode 5.4× slower than MSVC on qwen4exp (b11160), plus CPU tuning findings, negative results and questions

[geer1895]

Hello llama.cpp maintainers,

First of all, thank you for this wonderful project — it has become the backbone of our local inference setup.

We recently spent several weeks tuning CPU-only inference on a workstation, and collected a set of findings we'd like to share back. The first item looks like a genuine build-related bug and is the main reason for opening this issue; the rest are smaller findings, negative results (documented so others don't repeat them), and a few questions. Everything below is measured on real hardware, with raw JSON/logs available on request. If you prefer, we're happy to split this into separate issues — just let us know.

Environment

• CPU: AMD Threadripper PRO 3975WX, 32C/64T, Zen 2, AVX2 only (no AVX-512)
• RAM: 128 GB DDR4-2667, 4 channels populated (measured ~46.6 GB/s)
• GPU: Quadro P2000, 5 GB, sm_61 — effectively unusable for this model; all measurements below are pure CPU (-ngl 0 -dev none)
• OS: Windows (Server-class)
• Model: Qwen3.8-Flash-Next, architecture qwen4exp, UD-Q3_K_XL quant (83.79 GiB), 176.9B total / ~10B active params, 48 layers, 512 routed experts (10 active per token) + 640 shared, hybrid SSM (36 linear-attention layers + 12 full-attention layers), 256K context
• Reference build: unslothai/llama.cpp b11160-mix-a6922cc, windows-x64-cuda12-legacy variant (MSVC 19.44) — the only release build we found that still covers sm_61
• Self-built: official ggml-org/llama.cpp tag b11160, MinGW-w64 gcc 16.2.0 (w64devkit v2.10.0), CMake 4.4.3

Benchmark method: llama-bench -p 512 -n 128 -r 3, fixed -lm none -lzm off -ngl 0 -dev none. Key conclusions were repeated 2–3 times and sample values are reported.

---

Finding 1 (main): MinGW-w64 build decode is 5.4× slower than MSVC

Same source tag, same model, same machine, same parameters:

Build	Compiler	ISA	pp512 (tok/s)	tg128 (tok/s)
unsloth release	MSVC 19.44	haswell	55.86	6.83
self-built A	MinGW gcc 16.2.0	-march=native (znver2)	52.60	1.27
self-built B	MinGW gcc 16.2.0	haswell (multi-variant build)	51.08	1.28

What we ruled out:

• Not ISA-related: two builds with completely different ISA targets (znver2 vs haswell) both land at 1.27–1.28.
• Not multithreading sync: with the gcc build, single-thread decode is 1.56 and 20 threads is 1.27 — slower than single thread. So the per-thread kernel quality itself is bad, not the barriers.
• Prefill is barely affected (−5.8%), so the problem is specific to the decode path. Decode is single-token matvec traversing the MoE expert-gather path; prefill is large-batch matmul. Our guess is code-quality issues in the MoE gather / lazy-table access path under gcc (vectorization failure, aliasing assumptions, or the lazy gather table handling), but we could not pin it down to a specific code path.

Reproduction:

wget https://github.com/ggml-org/llama.cpp/archive/refs/tags/b11160.tar.gz
tar xzf b11160.tar.gz && cd llama.cpp-b11160

cmake -B build -G "MinGW Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=OFF -DGGML_NATIVE=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_TESTS=OFF
cmake --build build --target llama-bench -j 32

./build/bin/llama-bench.exe -m <qwen3.8.gguf> -p 512 -n 128 -t 20 -r 3 \
  -lm none -lzm off -ngl 0 -dev none
# expected: pp512 ≈ 52 (normal), tg128 ≈ 1.3 (should be ~6.8)

Also reproduces with the multi-variant shared-library build (GGML_CPU_ALL_VARIANTS=ON -DGGML_BACKEND_DL=ON -DBUILD_SHARED_LIBS=ON); the log confirms the ggml-cpu-haswell.dll backend was actually loaded. Results were identical (tg128 ≈ 1.28).

Controls we did not get to, which might help localize the cause: -DGGML_OPENMP=OFF (rule out libgomp vs MSVC OpenMP), -O2 vs -O3, clang/MinGW combinations, and — most valuable — testing a non-MoE model to see whether only the MoE/decode path is affected (we had no other model available locally).

Since MinGW/MSYS2 is a fairly common way to build llama.cpp on Windows, we think this is worth investigating. Any hints from someone experienced with MinGW builds would be appreciated — we're happy to run further experiments on this machine.

---

Finding 2: llama-bench does not support -tb (--threads-batch) — and it hides the biggest prefill lever

llama-bench rejects -tb with error: invalid parameter, while llama-server accepts --threads-batch. On our machine this parameter is the single largest prefill win we found, measured on llama-server (short prompt 68 tok / long prompt 1012 tok):

Config	prefill (short)	prefill (long)
-t 16	28.79	41.50
-t 20	32.74	48.80
-t 20 -tb 24	35.46	54.46

(+23–31% vs the t16 baseline; +8–12% vs t20 alone.) The reason it matters: prefill is compute-bound and decode is bandwidth-bound, so their optimal thread counts differ (24 vs 20 here; both regress at 32 due to SMT contention on Zen 2). Since llama-bench is the recommended tool for exactly this kind of scan — and can test many configs within one model load — not being able to sweep -tb there means most users will never discover this parameter. Any chance it could be added, to align with llama-server/llama-cli?

---

Finding 3: default --spec-draft-p-min 0 is not optimal; 0.30 is +10% here

With MTP speculative decoding on the same server and prompts (-t 20 -tb 24):

Config	decode (tok/s)	acceptance
p-min 0 (default)	8.12	46.9%
p-min 0.30	8.94	54.9%
p-min 0.50	8.69	62.1%
n-max 5	7.10	33.8%
ngram-mod only (no MTP)	6.86	0%

Interesting details: p-min 0.50 gives the highest acceptance but is slower than 0.30 (it filters out drafts that would have been accepted); n-max 5 collapses acceptance to 33.8%, which confirms the documented "<40% → shrink the draft" guidance. Our final production combo (p-min 0.30 + -ctkd/-ctvd q8_0) yields 8.87–9.20 tok/s. Also worth noting: acceptance is highly content-dependent — on long structured outputs we measured acceptance up to 87.5% (21/24, mean accepted len 3.33), so spec-decode benefits long-form/structured generation much more than short Q&A.

We're not necessarily suggesting a default change — just wondering whether the docs could recommend a starting value (0.2–0.3 worked best here) and describe the p-min × n-max interaction.

---

Finding 4: --slot-save-path API is easy to misuse and fails silently

Two sub-items we hit while trying to persist KV across restarts:

1. filename must go in the JSON body, not the query string. POST /slots/0?action=save&filename=x.bin returns 500 with key 'filename' not found — an error message that reads like a wrong parameter name rather than wrong placement. Only POST /slots/0?action=save + body {"filename": "x.bin"} works. A curl example in the docs would prevent this.
2. Saving after the request completes "succeeds" but stores nothing: HTTP 200 with n_saved: 0 and a 0-byte file, because the slot has already been released. To actually persist KV you must call save during streaming generation. Would it be possible to return a non-2xx (or at least a warning field) when n_saved == 0? As-is, "HTTP 200 + empty file" is very easy to misread as success.

---

Finding 5: docs recommend ngram-mod for reasoning models, but it underperforms a tuned MTP

Same server, same prompt:

Config	decode (tok/s)	acceptance
pure MTP + p-min 0.30	8.94	54.9%
MTP + ngram-mod (hybrid)	8.27	46.9%
pure MTP (default)	8.12	46.9%
ngram-mod only	6.86	0%

On this MoE reasoning model, ngram-mod not only loses to tuned MTP — it loses to the untuned MTP baseline. Our read: when a draft model (MTP) is available, ngram-mod's marginal value is limited or negative; it seems better suited to setups without a draft model. It might be worth a sentence in the docs.

---

Finding 6: Vulkan and CUDA release builds differ 12% in pure-CPU performance

With GPU fully disabled on both (-ngl 0 -dev none):

Build	prefill	decode
windows-x64-cuda12-legacy	35.52	8.87
windows-x64-vulkan	31.46	8.28

Since neither touches the GPU, the difference can only come from different CPU-side compilation options between the two release builds. Users who pick the Vulkan build (e.g., no CUDA runtime available) silently pay a CPU penalty — maybe worth unifying or documenting.

(Side note: Vulkan GPU offload on this machine fails outright with error: invalid argument: 1 under -ngl 999 -ncmoe 48 -nkvo 1, although the device is correctly enumerated. We didn't chase it further since GPU offload is a clear loss on 5 GB sm_61 anyway, but the error message could be more specific.)

---

Negative results (documented to save others the time)

All measured on this machine, -t 20, tg128 baseline 6.83:

Attempt	Result	Conclusion
GPU offload (-ngl 999 -ncmoe 48 -nkvo 1)	pp 14.36 (−74%), tg 2.86 (−58%)	per-layer activation transfer over PCIe swamps any gain on 5 GB Pascal
CPU affinity masks + --cpu-strict 1 (6 masks tried)	all below unbound (full mask 5.85, 16-core mask 4.35)	Windows scheduler already does well; strict binding hurts migration
mlock / mmap+mlock vs -lm none	6.84 / 6.80 vs 6.84	no difference; -lm none already holds weights in anonymous memory
KV cache quant (f16/q8_0/q4_0, 9 combos)	6.66–6.82	<2.4%, noise level
ubatch 128–1024, batch 512–4096	6.68–6.81	defaults (512/2048) already optimal
flash-attn off/on/auto	6.85/6.84/6.85	no difference (CPU)
NUMA + prio + poll tuning	6.82	no gain

One byproduct we found genuinely interesting: affinity masks 0x55555555 and 0xAAAAAAAA (16 threads, one per physical core, no SMT siblings) both reach 6.22–6.24 — clearly above the full 32-core mask (5.85), independently confirming that avoiding SMT siblings helps bandwidth-bound decode on Zen 2. This is consistent with the thread sweep: decode peaks at -t 20 (6.83) and regresses at -t 32 (6.21).

---

Questions

1. sparse-fa for qwen4: we noticed this optimization currently ships CUDA/Metal/Vulkan implementations only. Is a CPU implementation planned or possible? For CPU-only qwen4exp users this could be significant.
2. Hybrid SSM layers on x86: are there any current or planned optimizations for the 36 linear-attention (SSM) layers of qwen4exp on AVX2 CPUs, or is that path currently generic?
3. -ncmoe on a tiny GPU: given that full offload is a big loss here, is there a supported/meaningful combination of "a few non-MoE layers on GPU + all experts on CPU" for a 5 GB card, or is that strictly worse by construction?
4. Lazy gather-table readahead: the unsloth fork contains #137 llama: batched readahead for lazily read gather tables. Does official b11160 include an equivalent optimization? (We confirmed official b11160 does support the qwen4exp architecture itself — 11 files match — but couldn't verify whether the performance work is at parity. It might partly explain any difference between official and fork builds on this model.)

---

Thanks again for all the work on this project. We'd be glad to provide raw benchmark JSON/logs, run additional controlled experiments (e.g., the OpenMP/non-MoE controls for Finding 1), or split any of the above into standalone issues with full reproduction steps — whatever is most useful for you.