[Help Wanted] MoE inference 13× slower than llama.cpp — expert weight scatter bottleneck on DeepSeek-V2-Lite

[shifulegend]

🧭 Executive Summary

Project Zero is a from-scratch, dependency-free C inference engine for running large language models on commodity CPUs. We've achieved strong results on BitNet b1.58-2B-4T (beating bitnet.cpp by 1.8× on a Xeon), but have hit a fundamental bottleneck with Mixture-of-Experts (MoE) models — specifically DeepSeek-V2-Lite-Chat and OLMoE.

We are running at ~1 tok/s. llama.cpp runs the same model on the same hardware at 13.79 tok/s.

We're opening this discussion to get community input on the MoE weight scatter problem and any other ideas. All benchmark data, profiling results, and architecture documents are in the repo.

---

📊 Where We Stand

BitNet b1.58-2B-4T — ✅ Solved

Hardware	Project Zero	bitnet.cpp	Advantage
Intel Xeon (Emerald Rapids, 4C)	36.25 tok/s	19.83 tok/s	1.83×
i5-11300H (Tiger Lake, DDR4)	~16 tok/s	~19 tok/s	Within range

Memory bandwidth is the bottleneck here, and we're at ~95% of the theoretical DRAM ceiling on the Xeon. This model is effectively solved.

DeepSeek-V2-Lite-Chat (MoE) — 🔴 13× Behind llama.cpp

Threads	Project Zero	llama.cpp	Gap
T=1	1.10 tok/s	7.73 tok/s	7×
T=2	1.26 tok/s	13.44 tok/s	10.7×
T=4	1.32 tok/s	19.72 tok/s	14.9×

Hardware profiling reveals the root cause:

Metric	Project Zero	llama.cpp
IPC (instructions/cycle)	0.94–1.04	1.61–1.85
L3 cache miss rate	85–86%	~52%
Page faults per run	~2,000,000	~645

IPC < 1.0 and 86% cache miss rate = we are severely memory-stalled. The CPU is spending most of its time waiting for RAM, not computing.

---

🔬 Root Cause Analysis

Problem 1 — F32 dequantization path (confirmed, partially fixed)

We currently dequantize all Q4_K weights to F32 at compute time, incurring 50× more memory bandwidth than llama.cpp's native Q4_K matmul kernels. For dense layers (BitNet), we've replaced this with custom AVX-512 ternary/VNNI kernels. For MoE expert layers, we're still on the F32 path.

Problem 2 — MoE expert weight scatter (primary bottleneck)

DeepSeek-V2-Lite has 64 experts per MoE layer (26 MoE layers total). Each token activates only 6 experts. The experts' weight tensors are stored sequentially in the 8.9 GB GGUF file — so the 6 active experts for a given token are scattered across the file's address space.

What this means in practice:

• 6 independent DRAM streams per MoE layer × 26 layers = 156 scattered reads per token
• Intel's hardware prefetcher tracks ~8–10 streams max
• With 4 threads × 6 streams = 24 streams simultaneously, prefetcher is overwhelmed
• Effective bandwidth drops from ~11.7 GB/s to ~2–3 GB/s
• Theoretical max tok/s at degraded BW: ~2–3 tok/s vs analytical ceiling of ~9.8 tok/s

What llama.cpp does: At model load, it repacks expert weights into a layout where the weights for all experts in a layer are interleaved — so for any activated set of top-k experts, the relevant weights are contiguous. This trades a one-time load penalty for dramatically better cache/prefetch behavior during inference.

Problem 3 — MoE garbled output (fixed, awaiting validation)

Before P8, switching from incorrect dense-mode execution to correct sparse MoE revealed corrupted FP16 scale values in some Q2_K/IQ4_NL expert weight super-blocks. These NaN values cascaded through the residual stream → all logits became NaN → argmax always returned token 0 → endless ãĢįãĢį... output.

Fix applied (P8): NaN guards that zero out corrupt blocks instead of propagating NaN. Needs validation with DeepSeek-Q2_K model file.

---

🙋 What We're Looking For

1. Expert weight repacking strategy

How should we implement the interleaved expert weight layout at model load time for Q4_K_S GGUF files? Specifically:

• What's the right granularity for interleaving — by super-block, by row, by whole expert?
• How does llama.cpp's ggml_tensor repacking work for MoE? (Relevant source: llama.cpp/src/llama.cpp, ggml/src/ggml.c)
• Are there alternative approaches (e.g., on-demand expert prefetch with madvise, huge pages) that could help before a full repack?

2. Native Q4_K matmul kernel

We need an AVX-512 kernel that operates directly on Q4_K super-blocks (32-element blocks with 4-bit weights and FP16 scale + min). The goal is to replace the current dequant-to-F32 → F32 matmul pipeline.

• Any reference implementations beyond llama.cpp's GGML kernels?
• Optimal accumulation strategy for INT4 on VNNI (AVX-512 VBMI + VNNI)?

3. NaN guard validation

If anyone has DeepSeek-V2-Lite-Chat-Q2_K.gguf available, validation of P8 commit cc67445 would be extremely helpful. Expected output: coherent text (not ãĢįãĢį...) at ≥0.63 tok/s.

4. General MoE inference optimization ideas

Any other strategies you've seen work for MoE inference on CPU — expert caching, speculative expert selection, weight format changes — welcome.

---

📁 Key Documents

All docs are in the repo and contain full data:

Document	Contents
BENCHMARK_REPORT.md	Full matrix: threads × SIMD × classifier, perf counters, llama.cpp comparison
docs/PERFORMANCE_CEILING_REPORT.md	DRAM bandwidth math, theoretical ceilings, optimization journal
MOE_RESEARCH_AND_FIX_PLAN.md	Complete MoE research, failed experiments, root cause analysis, fix plan
DEBUGGING_JOURNAL.md	Step-by-step from 1.4 → 16 tok/s on BitNet

---

💻 Reproduce Locally

git clone https://github.com/shifulegend/project-zero
cd project-zero
make release

# BitNet (needs model download — see README)
./adaptive_ai_engine --model models/bitnet-b1.58-2B-4T.bin \
  --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin \
  --prompt "Hello world" --max-tokens 50

# DeepSeek GGUF
./adaptive_ai_engine --model models/deepseek-v2-lite-chat-Q4_K_S.gguf \
  --prompt "What is the capital of France?" --max-tokens 30 --threads 4

Requirements: GCC/Clang, pthreads, libm. No Python, no pip, no CUDA. Single binary.

---

🏷️ Labels suggested

performance, help wanted, moe, deepseek, gguf, avx512