RFC+PoC: MoE offload to disk with on-demand paging

[kisasexypantera94]

Running Qwen3-30B-A3B-Q6_K on M3 Pro 36GB, the expert weights alone wire ~27 GiB of memory. Wired memory on Apple Silicon can't be compressed or swapped, which leaves little headroom for other workloads.

This PoC implements on-demand expert loading: instead of keeping all 128 experts resident, we allocate a compact pool of N slots in Metal shared memory and page missing experts from the GGUF file via pread. This is exact inference without fallback to zero/random weights.

With lazy pool allocation this could enable running models larger than physical RAM.
UPD: managed to run Qwen3-30B-A3B-Q6_K on M1 Pro 16GB with 13 tok/s, so the concept works already.

How it works

Each MoE layer gets a compact pool tensor of N slots. A small Metal kernel (kernel_moe_interceptor) copies the selected expert IDs to shared CPU/GPU memory and publishes a request sequence number. A CPU sidecar thread sees the request, resolves expert-to-slot mappings via LRU, loads missing experts from disk with pread, writes remapped slot IDs back to shared memory, and signals an MTLSharedEvent. The GPU encoder waits on this event, then MUL_MAT_ID runs unchanged against the compact pool.

Measurements (Qwen3-30B-A3B-Q6_K, M3 Pro 36GB)

n_slots	wired GiB	saved GiB	tok/s (after warmup)
vanilla	~27.2	-	38.1
128	~27.1	-	30
96	~21.6	~5.6	30
80	~18.9	~8.3	29.1
64	~16.1	~11.1	27.5
32	~10.6	~16.6	22.1
16	~7.8	~19.4	15.7
8	~6.3	~20.9	12

How to run

./build/bin/llama-cli \
  --moe-n-slots 80 \
  --moe-n-layers 48 \
  --no-mmap \
  --no-warmup \
  -m /path/to/model.gguf \
  -ub 10 \
  --temp 0 -p "Explain mixture of experts"

Key constraints:

• -ub must satisfy ub * n_expert_used <= n_slots, otherwise we risk requesting more expert slots than available.

• This feature requires --no-mmap to be able to actually skip allocating tensors for expert weights.
• We also have to set --no-warmup, because warmup allocates a compute buffer for 2048 tokens
  and we get OOM.

Interested in feedback on whether this direction makes sense for llama.cpp, and whether there are cleaner integration points.

[uniment]

super cool. how do you decide which experts to evict?

[kisasexypantera94]

just a simple LRU. Also tried SLRU as suggested here, but saw no improvement, so I omitted it for simplicity

[uniment]

Wouldn't this render "REAP" models obsolete?

E.g. if I have 128GB unified RAM, but I want to run a 160GB model, instead of looking for the REAP specialized for my RAM I'd just let the workload dictate which experts are offloaded based on the RAM available.

Alternatively if I have 96GB VRAM and 48GB system RAM, I could let the workload dictate which experts are loaded into VRAM, LRU offload to RAM, and the remainder LRU offload to disk.

Am I missing anything?

[kisasexypantera94]

I think there are a lot of variables here.

First, it’s not guaranteed that a model’s experts will be sparse. While this was mostly true in my Qwen tests, knowledge in DS4 was distributed more uniformly across experts. And you really want your model sparse to get good performance. However, I think it should be possible to train a model to be intentionally sparse.

There are also many ways to implement expert disk offloading. LRU caching is just one of them. For example, we could stream expert weights layer by layer (with some readahead, of course). I believe that's what flash-moe does btw. The results of this approach would also vary significantly depending on the specific model.

[uniment]

I suppose it would make sense then to have a dedicated flag (which also turns off mmap and warmup) to select offloading strategy, e.g. --moe-offload lru or --moe-offload layer.

[lilydjwg]

I think on Linux --mmap already does this except the model is pre-populated on startup.

[kisasexypantera94]

Yes, the problem is most relevant for Apple machines because Metal has to wire buffers before command execution begins. That's why "trust the OS" approach does not work as good here. Unified memory also plays a significant role.

[kisasexypantera94]

But I guess this approach is also relevant for CUDA/Vulkan/etc., because they also have to pin (wire) memory before execution. So you have to manage your expert weights manually if you want to save memory, run a model on a memory-constrained device and still use GPU acceleration.

[lilydjwg]

I'm using Vulkan with an AMD discrete card and I've observed that the weights can be unloaded from memory when idle and loaded again when I use it; i.e. the memory is not pinned. For example the Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf file is only 73.9% in memory and after running a chat it goes up to 85.9%.

[kisasexypantera94]

Sure, offloading between sessions is not a problem. But during inference the GPU has to either wire the weights or support page faults to fetch paged memory. Vulkan doesn't support this, and CUDA only does via cudaMallocManaged, which llama.cpp doesn't use by default. It might offload between layers though, but that depends on whether llama.cpp splits each layer into separate submits, and from what I've seen it tends to pack everything into a monolithic pass where possible.

[Atomic-Germ]

The point is to keep more important pieces in faster retrival.

[koren1712]

Independent Windows/CUDA implementation + measurements (incl. negative results)

I independently built disk-backed expert streaming on Windows/CUDA before finding
this RFC — posting measurements in case they're useful to the design discussion.

Code + full report:
https://github.com/koren1712/llama.cpp/tree/expert-streaming-win

Hardware: i7-8700K (PCIe 3.0), RTX 4080 16GB, 32GB DDR4-2666, NVMe
(2.98 GB/s seq measured). Model: GLM-4.5-Air Q4_K_M merged GGUF
(73.5 GB file, 67.6 GB expert payload). Base commit: c8ae9a7.

Variant	t/s
baseline mmap (experts on CPU)	0.43
persistent async read pool, 12 workers	0.66
+ in-op IO/compute overlap	0.68
+ expert blocks 0-3 pinned in VRAM	0.75
16 GiB zero-copy slab cache	0.60 (less IO, slower)
24 GiB RAM tier on 32GB machine	0.12 (memory-pressure collapse)

Findings possibly relevant here:

1. 16 MiB overlapped expert reads sustain 2.80 GB/s inside live generation on
   PCIe 3.0 (drive seq capability: 2.98 GB/s) vs 377 MB/s via mmap paging.
2. Cache designs that reduce disk bytes can still lose end-to-end: a 16 GiB
   slab cut reads 132 -> 76 GB per run and was slower; an earlier heap-LRU lost
   to memcpy/eviction. This supports the zero-copy slot-buffer direction.
3. On 32GB-RAM machines, pinning >=24 GiB for an expert tier collapses the
   system (0.12 t/s).
4. PCIe 3.0 flatlines near 2.4 GB/s for this pattern regardless of thread count
   in the standalone reader.

The wall on this hardware is bytes/token: 4.13 GB/token pure streaming, 3.82
GB/token with 4 expert blocks in VRAM.

Happy to run Windows/CUDA tests for this RFC if useful.

[jerryjokesalot]

Benchmark data + calibrated pinning (complementary to this PoC)

We've been working on the same problem from the CPU side and have some measurements and a technique that might be useful here.

Your PoC is ahead of where we are — you have a working llama.cpp fork with Metal, LRU in shared memory, and pread from the GGUF, hitting 13 tok/s on M1 Pro 16GB. We built standalone benchmarks and tools, not an integrated engine. So what follows is meant to plug into what you've already built, not replace it.

What we measured (M1 MacBook Air, 16GB, CPU BLAS)

We wrote a C benchmark that runs MoE forward passes with Apple Accelerate (sgemv) and a dedicated I/O thread doing async pread from an expert-contiguous cache. The I/O thread loads the next layer's misses while the main thread computes the current layer's hits.

Results at controlled hit rates (f16 experts, Qwen3-30B-A3B):

Hit Rate	Sequential	Pipelined	Speedup
0%	1.53 tok/s	1.43 tok/s	--
50%	2.20 tok/s	2.40 tok/s	1.09x
75%	2.92 tok/s	3.81 tok/s	1.30x
88%	3.47 tok/s	4.10 tok/s	1.18x
100%	4.46 tok/s	4.45 tok/s	--

Per-layer compute: 4.7 ms. SSD: 3.0+ GB/s sequential. Pipelining hides up to 4 misses/layer, so you're compute-bound at 50% cache hit.

With Q4_K_M experts (~2.2 MB each), I/O per expert drops to ~0.27 ms vs 4.7 ms compute/layer. At that size, pipelining hides all misses — the cache becomes a bonus rather than a requirement. (Projected from measured SSD throughput, not directly benchmarked at Q4.)

Where this is useful for your PoC

These numbers are CPU-only, so they won't match your Metal throughput. But the hit-rate sweep gives a clean way to tune --moe-n-slots: if you know the compute-per-layer time on Metal and the I/O-per-expert time, the crossover point tells you the minimum slot count where pipelining makes you compute-bound.

Text-calibrated pinning (the part we think is most useful)

This is the technique we'd most like to contribute. Your PoC uses LRU for cache management, which starts cold — every expert is a miss on the first token.

We tested different pinning strategies:

Strategy	Hit Rate
Random activation profile	6% (random chance)
Text-calibrated (10 token pass)	42%
Oracle top-8/layer	45%
Calibrated + LRU	56%
Oracle top-32/layer	88%

Random profiles are useless — you have to calibrate from actual text. A short pass over ~10 real tokens, tracking which experts fire, then pinning the top-N per layer gives 42% hit rate. That matches oracle top-8 (45%) and stacks with LRU for 56% combined.

For your PoC, this would mean pre-warming the Metal shared memory slots before inference starts, so the first few tokens don't pay full I/O cost. On 16GB hardware where --moe-n-slots is small, cold-start misses are expensive and this would help the most.

A note on our contiguous re-layout

We also built a tool that dequantizes GGUF expert tensors and repacks them so each expert sits contiguously on disk. This was important for our path (mmap access to dequantized caches), but your pread from the raw GGUF with --no-mmap likely already reads contiguous quantized data per expert — the expert index is the outermost storage dimension. So the re-layout probably doesn't help your specific approach much. It would matter for mmap-based access paths or dequantized caches.

Code

Everything is at github.com/jerryjokesalot/tinygiant:

• nws_pipeline_bench.c — pipelined I/O benchmark (Accelerate + pthreads)
• nws_moe_bench.c — component-level microbenchmark
• nws_expert_relayout.py — contiguous cache builder
• nws_e2e_inference.py — Python/numpy inference (verified correct output)

We also posted a broader writeup in Discussion #27149.

If the calibrated pinning is useful, happy to help wire it into your branch.

[unbug]

Please make it work for all the model files, don't make another model file format.

[Atomic-Germ]

Every bit of progress I made with doing this same thing worked with standard gguf moe's