MoE Cache

moe-cache

moe-cache adds a CUDA expert cache, grouped MoE execution, drafting, MTP, overlap, prefetch, and bounded host pinning to llama/dev.

Important

CUDA REQUIRED: Build with -DGGML_CUDA=ON and use an NVIDIA GPU. Do not enable --moe-expert-cache-size on another backend.

Important

WINDOWS USERS: Follow Windows WDDM setup. Windows can reject full pinning even when ordinary RAM remains available.

Current documented checkpoint: 925933801.

What it adds

• GPU expert caching
• Partial host pinning
• Expert prefetch and bounded overflow staging
• Cached prefill
• Grouped target decode
• Grouped drafting and integrated MTP
• CUDA graph reuse
• PLE prefetch and decode overlap
• Cache and grouped-path diagnostics

The cache owns routed expert weights only. Shared experts, dense FFNs, and unrelated tensors keep normal placement.

How it runs

Each routed expert tensor gets the number of GPU slots set by --moe-expert-cache-size. A cache hit uses the resident slab. A miss copies the selected slab from its host source and may evict another resident slab.

The branch has two CUDA execution paths:

1. The grouped path runs certified target decode graphs when the tensor layout, route count, quantization, batch shape, and CUDA kernels qualify.
2. The established cached mul_mat_id path handles supported cache work that does not qualify for grouping.

A grouped-path decline does not disable the cache. Set --moe-expert-cache-size 0 to disable cache placement.

Limits

• Use -fit off; automatic fit does not include expert-cache pools.
• Size slots per cached tensor, not per process.
• Keep the full cold expert source in host memory. GPU slots do not replace it.
• Use layer split or one GPU. The fork does not shard one cached expert tensor with row or tensor split.
• Expect mmap page faults and eviction to affect file-backed sources.
• Treat MTP, PLE prefetch, decode overlap, boundary overlap, and staged Flash Next inputs as opt-in experiments.
• Deploy the executable with the matching fork-built llama, ggml, and CUDA libraries.
• Treat internal candidate and backend helper interfaces as private ABI.

Build

git clone --branch moe-cache https://github.com/GenerelSchwerz/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 20
./build/bin/llama-server --help

Start with the complete flag guide, then choose a measured configuration from Notable runs.

Windows setup - Hardware guides - Feature index - Home