[RFC]: Incremental MoE Expert Offloading — GPU Cache + Async Pipeline

[e1n00r]

Summary

Dynamic MoE expert weight offloading for vLLM. Expert weights live in CPU pinned memory; a fixed-size GPU cache holds the hottest experts; LFRU eviction and cross-layer prediction minimize cache misses. Models that exceed GPU VRAM can run on smaller hardware.

PR 1 is open: #37190 (~980 LOC Python, passing CI).

This RFC covers the full 3-PR architecture and provides production data from tinyserve, an independent implementation of the same techniques (30 tok/s decode on 8 GB GPU, 550+ tests).

Motivation

Large MoE models (DeepSeek-V3 671B, Qwen3.5-122B) don't fit in a single GPU. Only a small subset of experts activates per token (e.g., 8 of 256), so most expert weights sit idle. Moving cold experts to CPU and caching hot ones on GPU lets these models run on hardware that would otherwise OOM.

Prior art in vLLM

PR	What it does	Limitation this RFC addresses
#34535 (merged)	Static CPU weight offload	No runtime migration — offloaded weights stay on CPU permanently
#29941 (merged)	Async H2D prefetch for non-MoE weights	Pattern reused for expert prefetch in PR 2
RFC #33869 / #31938	Monolithic MoE offload (cache + CPU kernels + DBO + prefetch)	Closed — too large to review/pass CI. This RFC takes the opposite approach

Key design principle

The cache is a weight provider, not a special forward path. The kernel does not know or care where weights came from. No bypass of the runner pipeline.

Production data (tinyserve)

RTX PRO 2000 8 GB, GPT-OSS-20B MXFP4, 238 cache slots, single-stream decode:

Metric	Result
Decode throughput	30 tok/s (stable across context lengths)
vs HF device_map="auto"	160x faster
Cache hit rate (temporal prediction)	97-100%
Expert loads per layer (batched prefill)	O(num_experts) vs O(seq_len x top_k)

Caveat: These numbers are single-stream on a laptop GPU. Multi-user batched inference on H100 will have different bottlenecks (higher H2D bandwidth, but batch diversity may reduce hit rates). In-tree benchmarks will accompany each PR.

Architecture

ExpertWeightProvider (ABC)
├── FullGPUProvider        -- zero-cost passthrough (default, no overhead)
└── CachedWeightProvider   -- GPU LFRU cache + CPU backing store
      ├── GPUSlotManager   -- fixed-address GPU buffers
      ├── LFRUEviction     -- score = freq / age (preserves hub experts)
      ├── PersistentMapping -- GPU int32 tensor updated in-place (no per-call alloc)
      └── CPUBackingStore  -- pinned DRAM, all local experts

Integration point

At FusedMoEModularMethod.apply() — replace direct layer.w13_weight access with provider.prepare(topk_ids). No bypass of the runner. All paths go through runner.forward() -> quant_method.apply(), preserving EP dispatch, DP chunking, and shared-expert overlap.

prepare() contract

class ExpertWeightProvider(ABC):
    @abstractmethod
    def prepare(self, topk_ids: torch.Tensor) -> ExpertWeightResult:
        """Ensure requested experts are GPU-resident. Returns GPU tensors."""
        ...

@dataclass
class ExpertWeightResult:
    w1: torch.Tensor       # [capacity, ...] GPU-resident, fixed address
    w2: torch.Tensor       # [capacity, ...] GPU-resident, fixed address
    topk_ids: torch.Tensor # always remapped to slot indices
    w1_scale: Optional[torch.Tensor] = None
    w2_scale: Optional[torch.Tensor] = None

Design choices driven by torch.compile compatibility (per @zou3519's review of #29941):

• topk_ids is always remapped — no boolean flag that changes tensor interpretation
• Scales are fixed attributes, not a dynamic dict — avoids graph breaks
• GPU buffers allocated once at init (fixed addresses for CUDA graph capture)
• prepare() is inherently dynamic Python (eviction, H2D copies) and must run outside any torch.compile boundary
• Persistent mapping tensor — [num_experts] -> slot_index lives on GPU, updated in-place at each miss. Eliminates per-call CPU alloc + Python loop + H2D transfer from hot path (optimization from @alvinttang review).

LFRU eviction policy

PR 1 uses LFRU (frequency-weighted LRU) instead of pure LRU:

# Eviction score = freq / (clock - last_access + 1)
# Lower score = evict first
for expert_id, (slot, freq, last_access) in cache.items():
    age = clock - last_access + 1
    score = freq / age

LFRU preserves high-frequency "hub" experts even during bursts of unusual tokens. In GPT-OSS-20B, experts E13@L11 and E26@L15 handle 50%+ of traffic — pure LRU would evict them during domain shifts, causing thrashing. LFRU's frequency term prevents this.

Quant-agnostic tensor registration

Each quant method declares what tensors to cache. The cache stores them as opaque blobs by name — adding a new quant format requires zero cache code changes.

EP compatibility

The provider composes expert_map (global->local) with cache mapping (local->slot) into a single GPU mapping tensor. The kernel's existing -1 skip logic works unchanged.

Batched prefill

prepare() accepts batch topk_ids, deduplicates to unique expert IDs, loads each once. At 3K context with top_k=4 and 32 experts: 32 loads vs 12K sequential — 375x reduction. Without this, prefill is the #1 bottleneck.

CUDA graph compatibility (PR 2)

GPU buffers are fixed-address. A persistent int32 mapping tensor [num_experts] -> slot_index is updated by prepare() on CPU before graph replay. Inside the graph, slot lookup is pure indexing (slot_ids = mapping[topk_ids]) — no Python, no control flow. Same pattern as KV cache block tables.

What prepare() does NOT do

• No CPU fallback computation. If more experts are needed than cache capacity, prepare() raises a clear error with guidance to increase --moe-expert-cache-size. Hand-rolled CPU MoE with Python loops is a correctness trap.
• No silent config downgrade. If the user requests offloading and it's incompatible, that's an error, not a silent no-op.

Phased PR plan

PR	Scope	LOC	Status
PR 1 [#37190]	ExpertWeightProvider ABC + CachedWeightProvider with LFRU eviction, integrated into apply(). Sync H2D. BF16 + FP8. --enforce-eager required.	~980 Python	Open, CI passing
PR 2	Async H2D via CUDA stream + cross-layer temporal prediction + GPU mapping tensor for torch.compile compat	~400 Python	After PR 1 merge
PR 3	Disk tier (mmap), additional quant formats, EPLB integration, telemetry, imatrix seeding	~400 Python	After PR 1 merge

PR 1 scope (what ships)

• ExpertWeightProvider ABC + FullGPUProvider (zero-cost passthrough)
• CachedWeightProvider with LFRU eviction (score = freq / age)
• Integration into FusedMoEModularMethod.apply() — the bypass path (_forward_with_expert_cache) is deleted
• Synchronous H2D copies (no streams, no prefetch)
• --enforce-eager required (validated at init, cache disabled if violated)
• BF16 + FP8 per-tensor scale support
• Tests: 26 unit/integration tests for cache logic + full runner path
• Zero overhead on default path (no cache): one if provider is not None check per layer

PR 1 limitations (stated explicitly)

• Not for latency-sensitive production serving. Synchronous H2D + no CUDA graphs = batch inference and evaluation only.
• Single-GPU only. TP stall behavior from cache misses is not characterized. Use on multi-GPU TP setups is unsupported until timing analysis is done.
• LFRU only. Other eviction policies (ARC, LIRS) ship in follow-ups if workload data justifies them.
• No bias tensor support. MoE layers with w13_bias/w2_bias are detected at init and the cache is disabled with a warning. Follow-up PR can wire bias tensors through (they're small).
• Thread-safety assumption. prepare() assumes single-threaded forward pass (vLLM's current model). If vLLM moves to concurrent forwards with shared weights, locking would be needed.

Default-path overhead

When moe_expert_cache_size == 0 (99%+ of users): one if check per layer per forward pass. No allocations, no imports, no code paths touched. FullGPUProvider is a passthrough that returns the existing weight tensors unchanged.

Open questions

1. Integration depth in PR 1: Should ExpertWeightProvider.prepare() be called inside FusedMoEModularMethod.apply(), or should it be lifted to the model runner level (outside compile boundary) from the start? The former is simpler for PR 1; the latter is required for PR 2. Preference?

2. Memory accounting: GPU slot buffers are capacity * expert_size (e.g., 32 slots x 67MB = ~2.1 GB for DeepSeek-V3 BF16). Should these be allocated during vLLM's memory profiling phase so they're visible to the memory profiler, or is a separate accounting path acceptable? Note: CPU-pinned allocations are invisible to GPU profiler, which may cause over-allocation of KV cache (a benefit, not a hazard, but affects gpu_memory_utilization accuracy).

3. Observability timeline: Production deployers want per-layer hit/miss metrics exposed to Prometheus from day one. Should basic counters ship in PR 1 (adds ~30 LOC) or is PR 3 acceptable?

Future directions

• Autofit: Automatic cache sizing based on available VRAM + estimated KV cache cost (requested by @TriDefender, implemented in tinyserve as VRAMBudget). Would calculate optimal expert/KV tradeoff at load time.
• imatrix seeding: Use llama.cpp imatrix activation counts to pre-seed cache at load, eliminating cold-start (48-56% hit rate for first 80-160 tokens). Prototyped in tinyserve.
• Routing path prediction: Per ExpertFlow paper, use router logits from layer N to predict layer N+1 experts and begin prefetch. Overlaps with PR 2's temporal prediction.

CC

@mgoin @zou3519 @pavanimajety

References

• RFC #33869 — prior MoE offload RFC
• PR #37190 — PR 1 implementation (open)
• PR #34535 — selective CPU offload (merged, static)
• PR #29941 — async prefetch handler (merged, pattern for PR 2)
• tinyserve — independent validation, 550+ tests
• FATE — cross-layer temporal prediction (83% cosine similarity, 97-99% prediction accuracy)
• ExpertFlow — routing path predictor + expert cache engine (27.65% hit rate improvement over LRU)

---

Architecture validated in tinyserve. AI-assisted drafting (Claude Code).

[RemizovDenis]

Instead of incremental offloading (which is often bottlenecked by PCIe latency), have you considered in-place compression? The Temporal Expert Fusion in the latest TurboQuant update monitors expert usage and merges inactive ones using SVD dynamically. It's much faster than moving weights back and forth to CPU and keeps the hot-path purely on-device.

[e1n00r]

@RemizovDenis — in-place compression and offloading solve different problems. Offloading enables models that don't fit in VRAM at all (63 GB model on 8 GB GPU). Compression reduces the footprint of models that do fit but are tight. They're complementary — you could compress the CPU-resident experts for faster transfer on cache miss.

We tested SVD-based expert merging (D2-MoE pattern) on GPT-OSS-20B and found expert cosine similarity of 0.0006 — fully independent weights, no low-rank structure to exploit. This is model-specific; models with shared expert initialization would compress better.

TurboQuant's temporal expert fusion (merging inactive experts dynamically) is interesting but architecturally distinct — it changes the model's expert count at runtime, which affects routing, load balancing, and quality in ways that are hard to validate without per-model evaluation. Worth exploring as a separate RFC rather than coupling it with the offloading mechanism. The ExpertWeightProvider interface in this RFC would work unchanged with fused experts — the cache doesn't care whether it's storing original or fused weights.

[e1n00r]

imatrix-based cache seeding for GGUF models (proposal)

@lee-b suggested on llama.cpp#20757 using imatrix activation counts (already embedded in most GGUF files) to pre-seed the expert cache at load time. This would eliminate the cold-start phase where hit rate is 48-56% for the first 80-160 tokens.

The mechanism fits into the ExpertWeightProvider architecture: imatrix counts rank experts by activation frequency per layer, the cache pre-loads the top-N into GPU slots before the first forward pass. The pluggable policy then handles runtime eviction as the conversation shifts domain.

We've prototyped this in tinyserve — parser for the llama.cpp .dat format + seed_cache_from_ranking(). For vLLM, the equivalent would be parsing imatrix data during process_weights_after_loading and seeding the CachedWeightProvider before serving begins.

This is obviously only useful if GGUF models see meaningful adoption in vLLM. Would appreciate community input on whether GGUF-based serving is a relevant use case here, or if this is better left to llama.cpp/ktransformers.

[TriDefender]

I sincerely hopes to have support like the autofit on llama.cpp, which:

Calculates how much the kv cache would cost in VRAM
Estimates expert distribution across RAM/VRAM for optimal speeds
This will be more practical for enthusathists because they often won't have enough GPUs to fit everything in VRAM

[lee-b]

I sincerely hopes to have support like the autofit on llama.cpp, which:

Calculates how much the kv cache would cost in VRAM
Estimates expert distribution across RAM/VRAM for optimal speeds
This will be more practical for enthusathists because they often won't have enough GPUs to fit everything in VRAM

Honestly, as performant as vLLM is, if you want to run high-end models and are at all resource constrained, without modern server-grade hardware available, vLLM is a difficult option. I'd recommend trying llama-server with -sm graph or ik-llama with -sm tensor, or sglang -- each get pretty close in terms of performance, but have much better options to use older hardware, tighter memory constraints, smaller quants, etc.

[biaks]

I sincerely hopes to have support like the autofit on llama.cpp, which:

Calculates how much the kv cache would cost in VRAM
Estimates expert distribution across RAM/VRAM for optimal speeds
This will be more practical for enthusathists because they often won't have enough GPUs to fit everything in VRAM

Honestly, as performant as vLLM is, if you want to run high-end models and are at all resource constrained, without modern server-grade hardware available, vLLM is a difficult option. I'd recommend trying llama-server with -sm graph or ik-llama with -sm tensor, or sglang -- each get pretty close in terms of performance, but have much better options to use older hardware, tighter memory constraints, smaller quants, etc.

I think this is a bit too much of a "consumer hardware vs server hardware" framing.

The problem is not only that enthusiasts have old GPUs. Even with modern server hardware you can hit this wall.

For example, an 8×H200 system has around 1.1 TB of HBM, which is an extremely expensive setup, but it still cannot fit some of the newest MoE models entirely in GPU memory. A model like Kimi-K3 is exactly this kind of case: the GPU cluster is powerful, but the model size grows faster than GPU memory capacity.

The question is not "how do we make vLLM run on cheap old hardware?" The question is "how do we efficiently use expensive but finite hardware when model sizes exceed a single node's VRAM?"

Techniques like expert offloading, intelligent expert placement, and automatic memory planning are useful not only for hobbyists, but also for people running large inference servers who want to avoid scaling from 8 GPUs to 16 or 32 GPUs just because of memory capacity.

llama.cpp is ahead here because it had to solve this problem early. It doesn't mean the problem belongs only to small users — MoE models are simply outgrowing even high-end single-node servers.