moe-l2 — Measured Results & Quick Start (2026-08-12)

[yalun753]

moe-l2 is a llama.cpp kernel-level patch + Python orchestration layer that runs large MoE models on small GPUs: experts stay in CPU RAM (mmap, zero VRAM), the scheduler copies only activated experts to the GPU each step, hot experts are cached in VRAM (A3 cache), and a router map pins the high-frequency experts ahead of time (selective pin). First-request prefill warms the cache at startup.

Repo: https://github.com/yalun753/moe-l2 · PyPI: pip install moe-l2 (binary bundle via moe-l2 download-bins)

---

Benchmark (measured, bins-v0.4.0, 2026-08-10/11)

GPU	Model (file size)	moe-l2 speed	VRAM	RSS
RTX 4090	DeepSeek-V2-Lite (16B MoE, 6 GB)	145.63 t/s	4.9 GB	2.1 GB
RTX 4090	Qwen3.6-A3B (32B MoE, 11.5 GB)	74.99 t/s	3.1 GB	2.3 GB
RTX 4090	DeepSeek-V4-Flash (157B MoE, 85 GB)	35.96 t/s (on-demand) / 34.67 (selective pin)	16.5-16.7 GB	17.5 / 26.8 GB
RTX 4090	Qwen3-235B-A22B (235B MoE, 85.7 GB)	~3.9 t/s steady	24 GB	54.7 GB
RTX 2080 Ti (11 GB)	DeepSeek-V2-Lite	87.25 t/s	~1.0-1.3 GB	—
RTX 2080 Ti (11 GB)	Qwen3.6-A3B	47.24 t/s	~2.1-2.4 GB	—
RTX 5090	DeepSeek-V2-Lite	135.57 t/s	—	—
RTX 5090	Qwen3.6-A3B	76.41 t/s	—	—

Headline numbers

• 85 GB DeepSeek-V4-Flash on a 24 GB card: 35.96 t/s — vs 10.1 t/s with vanilla llama.cpp binary (+255%). RSS cut from 84.4 GB (whole-pin) to 17.5 GB (on-demand) / 26.8 GB (selective pin, −68%).
• 32B MoE fits in ~3.1 GB VRAM (8 GB cards run it) — vanilla llama.cpp OOMs immediately on 8 GB.
• 2080 Ti (11 GB) full chain: DS 87.25 t/s, Qwen 47.24 t/s (multi-arch bins-v0.4.0, selective pin).
• First-request prefill (2026-08-12): 120 prefill ops complete before HTTP listening; first request 42.5 t/s on 2080 Ti Qwen (vs 6.5-7.4 t/s without prefill, ~6×).

How the memory numbers work

Experts live in CPU RAM via mmap — zero VRAM for expert weights. Per token, only the ~top-6 activated experts per layer are copied to GPU (A3 cache keeps hot ones resident). Router map (learned from real gating traces) pre-pins top-100 experts/layer → RSS scales with active set, not model size.

Quick start (Linux + NVIDIA)

pip install moe-l2
moe-l2 download-bins            # fetch prebuilt multi-arch binaries (sm_61→sm_120a)
moe-l2 start --gpu --model /path/to/model.gguf --router-top-k 100
# Connect any OpenAI-compatible client to http://127.0.0.1:11435

Key env vars are set automatically by the CLI: GGML_OP_OFFLOAD_MIN_BATCH=1, GGML_CUDA_EXPERT_CACHE=1, MOE_L2_ROUTER_FILE=<auto-generated router map>.

Why this exists

MoE models are huge (157B+ params) but activate only a few experts per token. Vanilla llama.cpp keeps all weights on the GPU → OOM on consumer cards. moe-l2 moves the weight storage to cheap RAM/SSD while keeping computation on the GPU (cuBLAS, PCIe DMA reads), so a 24 GB card runs an 85 GB model at interactive speed.

Full reports

• DeepSeek-V4-Flash verify
• Qwen3.6-A3B benchmark
• DeepSeek-V2-Lite benchmark
• Qwen3-235B-A22B benchmark
• Multi-arch three-GPU report
• Model matrix

Feedback / ideas

All measurements are from real runs (scripts + data in repo). Questions, hardware requests, and benchmark contributions welcome — open a discussion or issue. If you run it on a card we haven't tested, please share your numbers!