16GB VRAM Setup

16 GB VRAM setup

Use this guide for one NVIDIA GPU with about 16 GB VRAM. The owner measured these runs directly on an RTX 5070 Ti with 16,303 MiB reported VRAM and 62.1 GiB system RAM. Build moe-cache with CUDA.

See Notable runs for the current Qwen3.8 Flash Next showcase and exact full-pin and partial-pin configurations.

Important

CUDA REQUIRED FOR MOE CACHING: Do not use the expert-cache flags on Vulkan, ROCm/HIP, Metal, SYCL, CPU-only, or another non-CUDA backend. Use the ordinary branch placement controls instead.

Important

WINDOWS USERS: Follow Windows WDDM partial-pinning setup before copying a MoE launch from this page.

Recommended single-request start

For Qwen3.6-35B-A3B Q4_K_M, begin at cache 96 rather than the measured cache-132 boundary:

./build/bin/llama-server \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl all -fit off \
  -c 65536 -b 8192 -ub 8192 -np 1 -t 12 \
  -fa on -ctk f16 -ctv f16 -kvo \
  --load-mode none \
  --moe-expert-cache-size 96 \
  --cache-ram 0 \
  --jinja

Cache 132 reached 14,369 MiB peak at NP1 and 14,675 MiB at NP4 on the owner card. It is a measured high-throughput setting, not a universal default. Use cache 96 or Q8 KV when the GPU drives a display or another process shares VRAM.

Verified by GenerelSchwerz: exact public GGUF runs

Model and exact artifact	Workload	Main geometry	Prefill	Decode	Loaded / peak VRAM	Result
Qwen3.6-35B-A3B Q4_K_M	NP1; 64,000 prompt + 512 output; 65,536 context	cache 132; ub=8192; F16 KV; backend sampling	4,326.76 tok/s server prompt eval	111.72 tok/s request	13,979 / 14,369 MiB	Exact accounting; zero server errors; valid Nsight profile
Same Qwen	NP4; 4 x 7,288 prompt + 1,024 output; 65,536 total context	cache 132; ub=8192; F16 KV	1,142.11-2,243.00 tok/s per request	247.79 tok/s aggregate; 62.00-67.78 per request	14,297 / 14,675 MiB	Exact accounting and four length stops; valid Nsight profile
Gemma-4-26B-A4B-it Q4_0	NP1; 63,247 prompt + 1,024 output	cache 48; ub=4096; F16 KV	3,999.88 tok/s effective	104.58 tok/s aggregate	9,809 / 10,291 MiB	Exact accounting and valid profile; repetitive continuation
Same Gemma	NP4; 4 x 15,123 prompt + 512 output	cache 48; ub=4096; F16 KV	4,772.28 tok/s effective	132.00 tok/s aggregate	12,937 / 13,429 MiB	Exact accounting; repetitive continuations
Same Gemma	NP8; 8 x 7,514 prompt + 512 output	cache 48; ub=2048; F16 KV	4,101.48 tok/s effective	122.54 tok/s aggregate	13,229 / 13,673 MiB	Exact accounting; mostly task-relevant with qualifications
Same Gemma	NP16; 16 x 3,448-3,451 prompt + 512 output	cache 48; ub=1024; F16 KV	3,221.31 tok/s effective	175.56 tok/s aggregate	14,605 / 14,997 MiB	Exact accounting; mostly task-relevant with qualifications

Gemma's physical ubatch was reduced as fan-out increased because NP8 at ub=4096 exceeded the startup memory envelope. This is a viability matrix, not a constant-geometry scaling curve. At NP8 and NP16, most work used the generic cached path with grouped tails; it is not full grouped-path performance.

Published Qwen fan-out evidence

The publication checkpoint also tested a locally converted mixed Q8-dense/NVFP4-expert GGUF derived from NVIDIA Qwen3.6-35B-A3B-NVFP4. There is no claim that a public file is byte-identical to that local GGUF.

NP and workload	Geometry	Effective prefill	Aggregate decode	Loaded / peak VRAM	Correctness
NP1; 64,000 + 1,024	cache 96; ub=8192; F16 KV	3,932.6 tok/s	105.86 tok/s	11,159 / 11,481 MiB	Exact; task-related
NP4; 4 x 15,500 + 512	cache 96; ub=8192; F16 KV	3,932.1 tok/s	105.53 tok/s	11,493 / 11,841 MiB	Exact; task-related
NP8; 8 x 7,288 + 512	cache 96; ub=8192; F16 KV	3,615.2 tok/s	147.08 tok/s	11,617 / 11,943 MiB	Exact; task-related
NP16; 16 x 3,500 + 512	cache 96; ub=8192; F16 KV	3,395.1 tok/s	132.99 tok/s	12,055 / 12,467 MiB	Exact; task-related; profile especially instrumentation-sensitive

Use the exact-public-GGUF table for a reproducible download. Use this second table as architecture and parallelism evidence only.

Parallel starting points

Intended load	Starting flags	Reason
NP1 long context	-c 65536 -np 1 -ub 8192 --moe-expert-cache-size 96	Leaves more reserve than the measured cache-132 Qwen run.
NP4 mixed requests	-c 65536 -np 4 -ub 2048 --moe-expert-cache-size 96	Reduces workspace pressure while keeping a substantial cache.
NP8	-c 65536 -np 8 -ub 1024 --moe-expert-cache-size 64	A cautious first fit; raise only after measuring the exact model.
NP16	-c 65536 -np 16 -ub 512 --moe-expert-cache-size 48	Prioritizes viability and headroom. Per-slot context is about 4K.

Treat these as setup recommendations. The tables above show the measured geometries.

System RAM

For the verified 12.1 to 20.4 GB files, 32 GB is a practical non-mmap minimum for one server. The 62.1 GiB test system provides much more room for operating-system cache, profiling artifacts, separate draft models, and parallel applications. Larger MoE GGUFs may require 64 GB or more even when the active GPU set stays under 16 GB.

Use --load-mode none whenever the complete cold expert source fits system RAM. It is the recommended performance setup and supports either full pinning or a bounded host budget. Use mmap only for an explicit file-backed deployment requirement and validate page-fault behavior separately. Windows users should follow the WDDM partial-pinning guide before choosing a budget. See the MoE cache flags page before changing either setting.

All hardware guides - 4 GiB - 8 GB - 12 GB - Evidence - Notable runs - MoE cache flags - Home