RFC: automatic placement of MoE weights on VRAM, RAM and flash, to run models bigger than memory at max prefill and decode speed

[Helldez]

Hi all,

in the last months I worked on BigMoeOnEdge, it's built on top of llama.cpp. It runs MoE models that are bigger then the memory of the device, and it configure itself for the machine where it runs.

It works with any MoE GGUF that llama.cpp can load, any quantization, no code per model.

OS: Android, iOS, macOS, Windows, Linux (arm64 and x86_64).

Backends: CPU, CUDA, ROCm (HIP), Vulkan, Metal, OpenCL, Hexagon NPU. Other llama.cpp backends load in the same way if you build them.

I would like to bring the main part inside llama.cpp, instead of keeping it as a separate layer.

16 GB Apple silicon laptop, Qwen3.6-35B-A3B Q4_K_M (22 GB, so bigger than the RAM). A coding agent made this page from one prompt, nobody touched anything, 35 minutes. Experts read from SSD all the time, prefill on GPU and decode on CPU, both chosen by the planner.

The idea

A MoE model uses only few experts for each token. So you don't need all the model in memory, but you need each part in the right place: the weights that every token reads, the experts used a lot, the experts used rarely. And the right place is different on every machine.

With --auto it measures the machine (free RAM and VRAM, speed of the disk, how fast each device runs a real layer of the model) and the model (size, experts, how it routes), and then it decides:

• where each weight goes: VRAM, RAM, or read from flash when needed
• which device does prefill and which does decode, they can be different
• how big are the caches, and how many threads

The target is the max prefill and decode speed that the machine can give, with the same output as the full model in memory. The settings that change only the speed are tested on the first answer and kept only if they are faster. For every choice it tells you why.

flowchart TD
  M["Machine:<br/>memory, disk, devices"] --> P
  F["Model:<br/>size, experts, routing"] --> P
  P["Plan"] --> W["Where each weight goes:<br/>VRAM, RAM or flash"]
  P --> D["Which device does<br/>prefill and decode"]
  P --> S["Cache size<br/>and threads"]
  W --> R["Run"]
  D --> R
  S --> R
  R --> T["Measure the first answer,<br/>keep what is faster"]
  T -->|"saved for next time"| P

Loading

The rules are simple:

• weights used by every token go in the fastest memory where they fit
• experts too, if they fit. If not, the most used ones stay in a cache (VRAM first, then RAM) and the others are read from flash
• prefill and decode can go on different devices. Every device is timed on a real layer and must give the same result as the CPU
• cache size comes from free memory and from how the model routes, threads from the first answer

So on a phone the experts stay on flash and the prompt goes on the NPU, on a laptop the prompt goes on the GPU, on a desktop the most used experts stay in VRAM (with llama.cpp llama_moe_cache).

Speeds

Here there are only some models that I test often. It runs any MoE that llama.cpp loads, in any quantization: Qwen, Gemma, gpt-oss, DeepSeek, GLM, LFM, Ling, Flint, Kolibri etc. Which tensors are experts is read from the file, so a new architecture don't need new code.

Hardware	Memory	Model	On disk	Experts from	Decode tok/s
Android phone	12 GB	DeepSeek V4 Flash 284B	91 GB	flash	1
		Qwen3.8-Flash-Next 125B	80 GB	flash	2-4
		Qwen3.6-35B-A3B	21 GB	flash	5
iPhone	12 GB	Qwen3.6-35B-A3B	21 GB	flash	3
Apple-silicon laptop	16 GB	Qwen3.8-Flash-Next 125B	76 GB	flash	7-9
		Qwen3.6-35B-A3B	21 GB	flash	18-24
Windows laptop	16 GB	Qwen3.6-35B-A3B	21 GB	flash	5
Desktop GPU (NVIDIA)	16 GB + 32 GB RAM	Qwen3.8-Flash-Next 125B	66 GB	flash, GPU cache	19-21
Desktop GPU (NVIDIA)	24 GB + 96 GB RAM	Qwen3.8-Flash-Next 125B	66 GB	RAM, GPU cache	70
		Qwen3.6-35B-A3B	21 GB	GPU	217
Datacenter GPU (AMD, ROCm)	192 GB	Qwen3.8-Flash-Next 125B	66 GB	GPU	53

Same machine and same file, vs llama.cpp today:

Hardware	Model	BigMoeOnEdge	llama.cpp
RTX PRO 6000 96 GB	DeepSeek V4.1 Flash	6.2-6.6	2.8
RTX 3090 24 GB	DeepSeek V4.1 Flash	6.2-7.0	0.18

Vs Strata (it has its own GPU kernels), both with MTP, Qwen3.8-Flash-Next 125B Q2_0: 153 vs 226-330 tok/s on RTX 5090, 89 vs 150-177 on RTX 4090. The difference is in the GPU compute, not in the placement.

How I measured and all the runs: benchmarks.

What I propose, one PR at time

1. RAM and flash under llama_moe_cache: an expert cache in RAM, filled from the GGUF on flash. So a model bigger than RAM + VRAM can run, without the page cache going crazy.
2. A planner on top of --fit: it puts the weights on VRAM, RAM and flash, chooses the devices for prefill and decode, and sizes the caches from what it measures.
3. Tuning on the first answer: try the settings that change only speed, keep the fastest.

There is already work on this: llama_moe_cache (#29887, #30966) and disk streaming in #25294. I prefer to build on them and not add another thing.

Questions

• Do you want something like this in llama.cpp?
• The RAM and flash part should extend llama_moe_cache or stay beside it?
• Which part is better for a first PR?

The project is Apache-2.0, what goes in llama.cpp I give it under MIT.

Thanks!

[dheiver2]

The dynamic routing strategy is sound, but the bottleneck often shifts from VRAM capacity to PCIe/NVMe bandwidth saturation during expert swapping. When offloading rarely used experts to flash, the latency penalty isn't just the raw read speed; it's the context switch overhead and the serialization of the KV cache if prefill and decode are split across devices. If the planner moves the decode phase to CPU because VRAM is full of active experts, the memory bandwidth of the CPU becomes the new constraint, potentially negating the speed gains of parallel device usage. Have you observed significant variance in throughput when the router's entropy changes, forcing a higher rate of expert swaps than the initial calibration assumed?