moe hot cache 2-4 speedup on big models not fitting vram

[neurall]

If you've got 2-4 GPUs sitting mostly idle running a big MoE model, this is for you.
Stock llama.cpp splits big models across GPUs by layer and tensor needs to sit full in gpu which is scifi these days

So only one GPU computes at a time, the rest sit idle. More GPUs = more VRAM, not more speed.

I forked llama.cpp and built based on excellent csantiago78's PR #27861 a VRAM-filling MoE expert cache instead:

Every GPU caches the experts actually being used, live, updated as you generate,
while the CPU handles cache misses in parallel instead of one-after-another.
Numbers on my box (2x RTX 3090, 128GB DDR4, 3700X), same prompt/temp 0, stock vs fork:
r/machinelearningnews - If you have 2-4 GPUs, prepare for a big smile 🙂 — forked llama.cpp: real multi-GPU speedup for MoE models bigger than VRAM

(Qwen's gain is small honestly — stock's autofit already fit most of it on GPU there, so little room for the cache to help. The big wins are on models where default placement leaves most expert work on the CPU.)

Credit where due:

the expert-cache mechanism builds on u csantiago78's PR #27861
(github.com ggml-org llama.cpp pull 27861) — they had it working first.
I'd independently been chasing the same hot-expert-cache idea and burned real time fighting the scheduler before finding their PR; no point reinventing a wheel that already worked, so I built on top of it

(VRAM-filling auto-sizing, usage-based eviction that only swaps when it pays back, CPU/GPU overlap,
added fused kernels). sdroege independently explored similar prefill-based cache warming in the same PR thread — worth a look too.

GLM-5.3-Flash support is from timkhronos's PRs #27773/ #27917
( github com ggml-org llama.cpp pull 27773).

Repo: github com neurall llama.cpp (Linux CUDA and windows release binary attached,
portable across CPU/GPU variants)

Upped Aditional 10% faster but not necesary
Pre-quantized GLM-5.3-Flash GGUF (10% faster attention weights, experts unchanged):

huggingface co neuralll GLM-5.3-Flash-GSQ-RCO-3.0bit-Q4Kattn-GGUF
If you've got 2+ GPUs and a model too big for one, try it and tell me what breaks — different GPU counts, different models, whatever. Feedback genuinely welcome.

https://github.com/neurall/llama.cpp

looking for job too