Running Pi's llama.cpp provider on low-VRAM hardware — a missing piece for local agents

[yalun753]

Running Pi's llama.cpp provider on low-VRAM hardware � a missing piece for local agents

Hi Pi team and community,

I've been using Pi's built-in llama.cpp provider (/login llama.cpp â�� router server) and it's great for multi-model local setups. But there's a gap I hit immediately: the standard llama.cpp router assumes you have enough VRAM to load the whole model. On 8-12 GB cards, most modern MoE models (DeepSeek-V4-Flash 157B, Qwen3-235B, etc.) simply OOM.

For anyone running Pi on a gaming laptop / small AI PC / home NAS, I wanted to share what I did to fix exactly this: moe-l2 â�� an open-source llama.cpp fork that keeps MoE expert weights in CPU RAM and streams only the active experts to the GPU on demand. It drops VRAM by ~95% with near-full speed, and it's a drop-in llama-server replacement, so Pi's provider works unchanged.

What it does (in one line)

Instead of loading all expert weights into VRAM (-ngl 999), expert tensors live in RAM via lazy mmap and only the ~top-6 activated experts per layer are copied to the GPU per token (A3 cache keeps hot ones resident).

Measured results (real hardware, 2026-08)

Setup	Model	Speed	VRAM
RTX 4090 + 64GB RAM	DeepSeek-V4-Flash 157B (85 GB file)	35.96 t/s	16.5-16.7 GB
RTX 2080 Ti (11 GB)	Qwen3.6-35B-A3B (11.5 GB)	47.24 t/s	~2.1-2.4 GB
RTX 2080 Ti (11 GB)	DeepSeek-V2-Lite (6 GB)	87.25 t/s	~1.0-1.3 GB

|