Empirical MoE Offload Benchmark on 4 GB VRAM: Static -ncmoe 41 (39.2 tok/s) vs Dynamic --moe-cache-mib (8.57 tok/s) + 94.8% 0.6B Speculative Acceptance

[AjayakumarV]

Sharing a complete 6-stage benchmark study and optimal llama-server configuration for running unsloth/Qwen3-30B-A3B-UD-IQ3_XXS.gguf (12.0 GB) on a 4 GB VRAM GPU (NVIDIA T1000) + 32 GB System RAM (Intel Core i7-12700K 8P+4E):

Full Open-Source Repo, Scripts & Paper: https://github.com/AjayakumarV/Qwen3Studio-LocalAI

Measured Performance Progression (4 GB VRAM + 32 GB RAM)

• Cold Baseline (-ngl 99 --cpu-moe, default mmap, 4 slots): 6.89 tok/s decode | 42,811 ms TTFT
• Dynamic GPU MoE LRU Cache (--moe-cache-mib 768): 8.57 tok/s decode (56% slowdown due to synchronous PCIe cache-miss thrashing across 48 layers)
• Static 7-Layer GPU Pinning (-ncmoe 41) + Aligned RAM (--load-mode none) + P-Core Affinity (-Cr 0-15) + Top-4 Routing (expert_used_count=int:4): 39.18 tok/s decode | 290.9 ms TTFT (82.49 prompt tok/s) | 3,762 / 4,096 MiB VRAM | 20.5 GB RAM free
• Full 8-Expert Mode (expert_used_count=int:8) + unsloth/Qwen3-0.6B-UD-Q4_K_XL.gguf Speculative Draft (--spec-type draft-simple --spec-draft-n-max 3 --spec-draft-p-min 0.80): 94.8% draft token acceptance rate (91/96 tokens) -> 27.11 tok/s (+45.1% speedup over 18.68 tok/s)

Optimal 4 GB VRAM Launch Command (llama-server b12121+)

llama-server.exe `
  -m Qwen3-30B-A3B-UD-IQ3_XXS.gguf `
  --load-mode none -ngl 99 -ncmoe 41 `
  -fa on -ctk q8_0 -ctv q8_0 `
  -np 1 --keep -1 --cache-reuse 256 `
  -c 8192 -ub 1024 -t 8 -tb 16 `
  -Cr 0-15 --prio 2 --prio-batch 2 --poll 100 `
  --override-kv qwen3moe.expert_used_count=int:4 `
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.06 `
  --repeat-penalty 1.1 --presence-penalty 0.4 --no-webui