12GB VRAM Setup

12 GB VRAM setup

Use this guide for one NVIDIA GPU with about 12 GB VRAM, especially for long NP1 MoE workloads or conservative drafting.

Important

CUDA REQUIRED FOR MOE CACHING: Do not use the expert-cache flags on a non-CUDA backend. These are fits measured on a 16 GB RTX 5070 Ti, not physical 12 GB GPU tests. The GPT-OSS run peaked at 11,475 MiB, only 813 MiB below a nominal 12 GiB boundary. Use its safer starting values before replaying the measured geometry.

Important

WINDOWS USERS: Follow Windows WDDM partial-pinning setup before copying a MoE launch from this page.

Recommended target-only starts

GPT-OSS-20B MXFP4

The exact measured model is GPT-OSS-20B MXFP4, a 12.1 GB file. Begin below the measured cache and ubatch:

./build/bin/llama-server \
  --model /path/to/gpt-oss-20b-MXFP4.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl all -fit off \
  -c 65536 -b 8192 -ub 1024 -np 1 -t 12 \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none \
  --moe-expert-cache-size 20 \
  --cache-ram 0 \
  --jinja

The measured profile used cache 24, ub=2048, and F16 KV. Move toward those settings only if the load leaves enough headroom.

Gemma 4 Q4_0

For Gemma-4-26B-A4B-it Q4_0, cache 48 with ub=4096, F16 KV, NP1, and a 65,536 context pool peaked at 10,291 MiB in the owner run. That is a more comfortable 12 GB starting point, but output quality in the frozen matrix prompt was repetitive on both compared runtimes. Validate your real prompt before treating the throughput as representative.

Verified by GenerelSchwerz

Model and exact artifact	Workload	Main geometry	Prefill	Decode	TTFT	Loaded / peak VRAM	Result
GPT-OSS-20B MXFP4	NP1; 64,000 prompt + 1,024 output; 65,536 context	cache 24; ub=2048; F16 KV	2,735.58 tok/s server prompt eval	141.40 tok/s request	23.530 s	11,191 / 11,475 MiB	Exact accounting and length stop; zero server errors; valid Nsight profile
Gemma-4-26B-A4B-it Q4_0	NP1; 63,247 prompt + 1,024 output; 65,536 context	cache 48; ub=4096; F16 KV	3,999.88 tok/s effective prefill	104.58 tok/s aggregate	15.812 s	9,809 / 10,291 MiB	Exact accounting and valid profile; continuation was repetitive
Ornith-1.5-35B-A3B APEX MTP I-Compact	NP1; 64,904 prompt + 512 output; 65,536 context	cache 48; ub=8192; F16 KV; MTP depth 1	3,940.10 tok/s server prompt eval	99.64 tok/s request	16.527 s	8,339 / 9,246 MiB	Exact length accounting; all output tokens were in reasoning; valid Nsight profile
Same Ornith, target only	NP1; 64,904 prompt + 512 output	cache 48; ub=8192; F16 KV; speculation off	4,257.42 tok/s server prompt eval	86.33 tok/s request	15.297 s	6,905 / 7,296 MiB	Matched prompt and output count; valid Nsight profile

On this Ornith fixture, MTP depth one raised decode from 86.33 to 99.64 tok/s, about 15.4%, while peak VRAM rose by 1,950 MiB and server prefill fell about 7.5%. This is one model and prompt, not a universal MTP expectation.

System RAM choices

Model	Published file	Practical non-mmap RAM
GPT-OSS-20B MXFP4	12.1 GB	24 GB minimum; 32 GB preferred
Gemma 4 Q4_0	14.6 GB	24 GB minimum; 32 GB preferred
Ornith I-Compact	17.4 GB	32 GB
Qwen 3.6 Q4_K_M	20.4 GB	32 GB minimum

The benchmark host had 62.1 GiB RAM. The table is a planning recommendation from file sizes and runtime margin, not measured peak RSS.

Parallel requests

Twelve GB is enough for some NP4 configurations, but cache, KV, and workspace all scale differently. Do not infer NP4 fit from an NP1 peak. The exact Gemma NP4 matrix needed 13,429 MiB and therefore belongs in the 16 GB guide, while the older Qwen cache-48 NP4 profile fit below 8 GB because its pool and physical batch were much smaller.

For a first NP4 attempt:

-c 32768 -np 4 -b 8192 -ub 512
--moe-expert-cache-size 40
-ctk q8_0 -ctv q8_0

Treat this as a conservative starting geometry, not a benchmark. Confirm per-slot context, output accounting, aggregate decode, and peak VRAM.

All hardware guides - 4 GiB - 8 GB - 16 GB - Evidence - Home