CUDA MoE Expert Cache

CUDA MoE expert cache

The CUDA MoE expert cache keeps a configurable working set of routed expert-weight slabs in GPU memory and transfers cold experts from host memory on demand. It targets models whose complete expert weights exceed VRAM but whose routed working set is much smaller.

Important

CUDA REQUIRED: The NVIDIA CUDA backend implements this cache. Use normal llama.cpp placement on other backends.

Important

WINDOWS USERS: Read Windows WDDM partial-pinning setup before choosing a cache or host budget. WDDM can reject full pinning even when the model fits ordinary system RAM.

Control and placement

--moe-expert-cache-size N allocates N slots per cached expert tensor on its owning CUDA device. Zero disables the cache and restores normal placement. The library equivalent is llama_model_params::moe_expert_cache_slots.

The loader recognizes routed ffn_up, ffn_down, ffn_gate, fused ffn_gate_up, and chunk-expert weights. Shared experts, dense FFNs, and unrelated tensors keep normal placement. The cache override wins over matching CPU-MoE and tensor-buffer placement controls because those same routed weights must have one owner.

VRAM scales approximately as slots x expert slab stride x cached tensors, plus metadata, staging, auxiliary banks, KV, workspace, and graph resources. --fit does not account for these pools, so tight configurations should use -fit off and explicit slot sizing.

The cache has two execution tiers: certified graphs may use grouped MoE decode, while other valid cache workloads use the established cached mul_mat_id path. A grouped decline does not mean the cache is disabled.

Branch implementations

Branch	Loader/API	CUDA implementation
moe-cache	src/llama-model.cpp, include/llama.h	moe-cache.cu, ggml-cuda.cu

The broad grouped architecture was reviewed in PR 56. The direct source above is authoritative for the promoted moe-cache branch.

Feature index - flags and 16 GB setup