Expert-Aware SSD Streaming for MoE Models on Memory-Constrained Hardware

[jerryjokesalot]

Idea

Stream only active expert weights from NVMe SSD per token for MoE models, instead of loading the entire model into RAM. This is complementary to #20757 (GPU expert cache) — that proposal targets GPU+RAM setups; this one targets machines with no GPU and insufficient RAM (8-16GB consumer laptops).

The core insight: the assumption that "the entire model must be loaded into memory before you can use it" is a software architecture choice, not a law of nature. MoE models already tell us which parameters they need — the router selects 8 of 128 experts per token. We just need to make those parameters hierarchically addressable so only the active working set needs to be in RAM at any moment.

Think of it as virtual memory for neural networks, except the model predicts its own working set. Traditional virtual memory doesn't know what the program needs next. An MoE model does — the router completes before the expert computation begins, giving us a prefetch window.

For Qwen3-30B-A3B (128 experts, 8 active per token), per-layer I/O drops from 263 MB to 31 MB — an 11.4x reduction. The prototype achieves 4.7 tok/s on consumer hardware using ~2 GB RAM.

Why This Matters Now

Storage is getting stupidly cheap. RAM remains comparatively expensive.

Instead of asking "how do we make computers have more RAM?", the better question is: "how do we make inference engines hierarchy-aware?"

A $500 laptop with a 1 TB NVMe can already stream at 3-6 GB/s. That's enough bandwidth to feed active expert weights per token — if the inference engine knows how to ask for them selectively instead of loading all 128 experts into RAM and hoping the OS pages out the dormant ones.

Open weights don't equal accessible AI if the only practical way to run a powerful model is a $2,000+ workstation. Making inference hierarchy-aware changes the economics.

The Problem Today

Running MoE models on memory-constrained hardware:

Scenario (M1 MacBook Pro, 16GB)	Speed	RAM	System State
qwen3:32b (dense) via Ollama	<1 tok/s	20 GB+ swap	Machine freezes
qwen3:30b-a3b (MoE) via Ollama	1.7 tok/s	11.3 GB	Functional but slow

The MoE model is better because only 3B params are hot per token — the OS can page out dormant experts. But llama.cpp still loads all 128 experts into memory, consuming 15.75 GB for weights that are mostly idle.

The OS page cache is doing accidental expert streaming — just badly, because it has no idea which experts will be needed next. We can do it deliberately.

What I Built and Measured

I built a prototype (TinyGiant) that validates the I/O feasibility of expert-aware SSD streaming. All measurements are on a stock M1 MacBook Pro (16GB, built-in NVMe SSD).

SSD throughput is fast enough

Method	Throughput
Sequential layer read (286 MB chunks)	2.97 GB/s
Double-buffered pipeline	6.06 GB/s
Pipeline speedup	2.09x

GGUF format already supports selective expert reads

MoE expert weights are stored as stacked 3D tensors (blk.N.ffn_gate_exps.weight, shape [128, hidden_dim, expert_dim]). Each expert is a contiguous byte slice — a simple seek + read of ~3 MB per active expert per tensor. No format changes needed.

Parsed from the actual Qwen3-30B-A3B GGUF file:

Component	Size
Shared weights (attention, norms)	0.43 GB — keep in RAM
Router weights	0.05 GB — keep in RAM
Expert weights (all 128)	15.75 GB — stream from SSD
Per-layer, all experts	345 MB
Per-layer, 8 active experts	30 MB
I/O reduction	11.4x

Expert Cascade prototype results

The prototype uses real SSD reads of the actual 17.3 GB model file with simulated compute (confidence routing + layer streaming, but matrix multiplication is not wired up). The I/O bottleneck — which is what this proposal addresses — is fully validated.

Mode	Tokens	Speed	Routing
Full-layer reads (baseline)	50	0.9 tok/s	60% easy / 28% med / 12% hard
Expert-aware reads (31 MB/layer)	100	4.7 tok/s	55% easy / 29% med / 16% hard

With double-buffered I/O (measured 2.09x): ~6.3 tok/s projected.

Proposed Design for llama.cpp

The memory hierarchy

The key architectural change: treat parameters as living in a hierarchy, not a flat buffer.

         SSD (17 GB)
    ┌────────────────────┐
    │ Expert weights      │  ← 15.75 GB, dormant
    │ (128 per layer)     │     seek + read on demand
    └────────┬───────────┘
             │ selective read (31 MB/layer)
    ┌────────▼───────────┐
    │ RAM (~1-2 GB)       │  ← shared weights (persistent)
    │ + active expert buf │     + active experts (per-token)
    └────────┬───────────┘
             │
    ┌────────▼───────────┐
    │ CPU compute         │
    └────────────────────┘

This naturally extends to SSD → RAM → VRAM when a GPU is available — experts get promoted up the tiers based on access frequency.

Phase 1: Expert-aware model loading

Add a mode to llama_model_load() that keeps shared weights (attention, norms, router, embeddings) in RAM but does not load ffn_*_exps tensors. These stay on disk, accessed via file handle.

Memory budget for Qwen3-30B-A3B:

• Shared weights: ~0.5 GB (persistent)
• Active expert buffer: ~24 MB per layer (2 layers double-buffered)
• KV cache: variable
• Total: ~1-2 GB

Phase 2: On-demand expert streaming

During the forward pass for each MoE layer:

1. Run attention on shared weights (in RAM)
2. Run MoE router (small matmul, ~1 MB weights) to identify active experts
3. Seek + read active expert byte ranges from the GGUF file
4. Execute expert FFN with streamed weights
5. Release expert buffers

The router completes before attention finishes, creating a prefetch window for expert I/O. This is the "model predicts its own working set" property — the information about which parameters are needed arrives before they're needed.

Phase 3: Double-buffered pipeline

Overlap I/O for layer N+1 with compute on layer N. Measured 2.09x throughput improvement.

Integration points

File	What changes
llama_model_load()	Skip ffn_*_exps tensors in streaming mode
llama_decode_internal()	Before each MoE layer, trigger async expert reads
New	ggml_backend_buffer_ssd — SSD-backed buffer with selective reads
New	Expert prefetch scheduler tied to router output

Compatibility

• Falls back to standard loading when RAM is sufficient
• Works with existing GGUF files (no format changes)
• Compatible with all quantization types
• Complementary to Feature Request: Two-tier GPU+RAM expert cache for MoE offload (pluggable eviction policy) #20757 (GPU expert cache) — they can coexist

Relationship to #20757

	#20757 (GPU Expert Cache)	This proposal (SSD Streaming)
Target hardware	GPU + CPU RAM	CPU only, 8-16GB total
Expert storage	CPU RAM (mmap'd)	NVMe SSD
Cache strategy	SLRU eviction in VRAM	On-demand streaming per token
When to use	GPU with limited VRAM	No GPU, limited RAM

These are complementary tiers. A complete hierarchy-aware system could be: SSD → RAM → VRAM, with experts promoted up the tiers based on access frequency.

Reproducing the measurements

Everything is open source: https://github.com/jerryjokesalot/tinygiant

git clone https://github.com/jerryjokesalot/tinygiant
ollama pull qwen3:30b-a3b

# Find and symlink the model blob
ln -s ~/.ollama/models/blobs/sha256-HASH ~/models/qwen3-30b-a3b.gguf

# Map expert layout
python3 tools/gguf_expert_mapper.py ~/models/qwen3-30b-a3b.gguf

# Benchmark SSD streaming
python3 tools/layer_streaming_poc.py ~/models/qwen3-30b-a3b.gguf 48

# Run Expert Cascade prototype
python3 tools/cascade_prototype.py ~/models/qwen3-30b-a3b.gguf 100 --expert-ratio 0.088 --layers 48

AI Disclosure

I used Claude Code to help build the prototype tools, run benchmarks, and structure this write-up. The measurements are real (run on my M1 MacBook Pro), the design rationale is mine, and I can defend every technical claim here.

Update (2026-08-19): Compiled benchmarks validate the concept

Since the original post, we built compiled C benchmarks with real BLAS compute that validate the I/O feasibility claims above. The repo (tinygiant) now includes:

Expert-contiguous re-layout — GGUF stores all 128 experts interleaved in one tensor. Accessing 1 expert via mmap touches all 6,912 pages (111x amplification). Our relayout tool repacks experts contiguously: 1 expert = 192 pages (36x reduction).

Pipelined I/O benchmark — C benchmark using Apple Accelerate (BLAS sgemv) with async pread on M1 16GB:

Cache Hit Rate	Sequential	Pipelined
50%	2.20 tok/s	2.40 tok/s
75%	2.92 tok/s	3.81 tok/s
88%	3.47 tok/s	4.10 tok/s
100%	4.46 tok/s	4.45 tok/s

Compute ceiling: 4.4 tok/s (CPU only). SSD: 3.0+ GB/s from contiguous layout. At 88% hit, SSD reads are fully hidden behind compute.

Text-calibrated pinning — random activation profiles give 6% hit (= chance). Calibrating from 10 tokens of real text gives 42% hit, matching oracle top-8. Calibrated + LRU = 56%.

End-to-end inference — Python engine generating verified correct text from Qwen3-30B-A3B using the contiguous cache.

Related: Discussion #23324 has an independent Metal-based PoC achieving 13 tok/s on M1 Pro 16GB with LRU expert paging. We posted our benchmark data and a concrete integration proposal there.

[Atomic-Germ]

https://github.com/Atomic-Germ/Guanaco if you want to see how that idea performs.

The best test is running a 130B+ MoE model with llama.cpp forced to use ~4gb

It loads stupid fast because it doesn't really load at all, unless you provide an imatrix - then it'll pin initial experts based on that. It gets faster and faster as you use it but that's not really the goal, just a happy side effect of most conversations sticking to a small set of experts.

Think of it like live-pruning. Anyway yes the idea works and has been thrown around a bit. Guanaco is only one way to do it.

And for everyone's knowledge: Given the problem of RAM constraints, most models will converge on this solution as well as most people who are into inspecting their own mind and how it works.

My inspiration was actually from neuroscience and thinking about the way people coded when i was a kid and RAM prices were as bad as now.

I think the team Should take some kind of implementation of the idea and run with it. At this point there are like five examples other than mine or yours.

Also; have you solved how to pin with mxfp4? I can't get them to pin without loading the whole thing for some reason in mine.

Oh... and mmap is one way to go, but have you tried deliberate faulting?

[jerryjokesalot]

Update: I've since built a compiled C benchmark that validates the pipelining concept with real BLAS compute (Apple Accelerate) and measured pipelined I/O on an M1 MacBook Air, 16GB. Results and the expert-contiguous layout technique are posted in #23324 — MoE Expert Offload to Disk with On-Demand Paging, which is working on the same problem from a slightly different angle.

Key findings since this original post:

• Expert-contiguous re-layout reduces page faults by 36x (192 pages per expert vs 6,912 in interleaved GGUF)
• Pipelined async I/O hides SSD reads behind compute: 3.81 tok/s at 75% cache hit, 4.10 at 88% (CPU-only)
• With Q4 quantized experts (~2.2 MB), pipelining hides all I/O — compute-bound even at 0% cache hit
• Text-calibrated pinning achieves 42% hit rate vs 6% for random profiles