4GB VRAM Setup

4 GiB VRAM setup

Use these measured NP1 configurations to target a 4 GiB process-VRAM ceiling. Every listed run peaked at or below 4,096 MiB.

Important

CUDA REQUIRED FOR MOE CACHING: These settings require moe-cache, an NVIDIA GPU, and a CUDA build. Do not use --moe-expert-cache-size with Vulkan, ROCm/HIP, Metal, SYCL, CPU-only, or another non-CUDA backend.

Important

WINDOWS USERS: Follow Windows WDDM partial-pinning setup before copying a MoE launch from this page.

Warning

The owner measured these ceilings on a 16 GB RTX 5070 Ti, not a physical 4 GB card. Display use, driver reservations, and other CUDA processes reduce usable VRAM. Gemma leaves only 150 MiB below the 4,096 MiB ceiling.

Measured contract

Item	Value
Branch and revision	moe-cache 71bc85d0e
GPU used to collect the measurements	NVIDIA GeForce RTX 5070 Ti, 16,303 MiB reported VRAM
Workload	NP1, 1,883-2,181 prompt tokens plus exactly 256 output tokens
Context and batches	-c 4096 -b 2048 -ub 512 -np 1
KV and attention	Q8_0 K/V, K-shift offload, CUDA flash attention
Loading	Non-mmap, no warmup, no prompt cache, no speculation
Sampling	Temperature 0, top-p 1, min-p 0, seed 12345
Profiling	Full-request Nsight Systems trace for every accepted cache0 and positive-cache row

The fixed deployment-review prompt had SHA-256 703b2e149f8ae542b90e3a6ae7f3726ff0240e9b4db71c197e87bd0dd3f02fa4. Token counts differ because each model uses its own tokenizer.

Exact downloads

Short name	Quant	Download
Qwen3.6	Q4_K_M	Qwen3.6-35B-A3B-GGUF
Ornith	I-Compact	Ornith-1.5-35B-A3B-APEX-MTP-GGUF
Gemma 4	Q4_0	gemma-4-26B-A4B-it-GGUF
GPT-OSS	MXFP4	gpt-oss-20b-GGUF
LFM2.5	Q3_K_M	LFM2.5-8B-A1B-GGUF
Nemotron	Q4_K_M	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

Configurations that fit

Model	Cache / CPU-MoE setting	Prefill	Decode	TTFT / wall	Loaded / peak VRAM	Execution tier
Qwen3.6	cache 12 / -ncmoe 20	1,408.92 tok/s	77.63 tok/s	1,460.65 ms / 4.746 s	3,480 / 3,724 MiB	Grouped
Ornith	cache 24 / -ncmoe 20	1,424.92 tok/s	92.00 tok/s	1,445.65 ms / 4.218 s	3,364 / 3,616 MiB	Grouped
Gemma 4	cache 12 / -ncmoe 40	1,606.10 tok/s	67.54 tok/s	1,366.15 ms / 5.142 s	3,598 / 3,946 MiB	Grouped
GPT-OSS	cache 4 / -ncmoe 24	2,027.56 tok/s	51.47 tok/s	938.59 ms / 5.893 s	3,100 / 3,302 MiB	Grouped
LFM2.5	cache 24 / -ncmoe 32	6,065.22 tok/s	320.18 tok/s	334.29 ms / 1.131 s	3,550 / 3,716 MiB	Grouped
Nemotron	cache 4 / -ncmoe 999	969.89 tok/s	42.90 tok/s	2,140.53 ms / 8.085 s	2,814 / 3,510 MiB	Established cached tier

All six runs completed the exact 256-token request with a length stop, nonempty output, no server/client errors, valid Nsight report and SQLite export, CUDA kernel rows, and clean process/port/GPU/profiler teardown.

Improvement over the 4 GiB cache0 control

Each control used the same branch, model, prompt, context, ubatch, Q8 KV, profiler, and sampling. All routed MoE layers were assigned to ordinary CPU-MoE execution in the cache0 control.

Model	Cache0 peak	Cache0 decode	Selected decode	Decode change	Prefill change	Wall change
Qwen3.6	2,688 MiB	27.46 tok/s	77.63 tok/s	+182.7%	+13.0%	-56.6%
Ornith	2,040 MiB	38.24 tok/s	92.00 tok/s	+140.6%	+15.6%	-49.4%
Gemma 4	2,626 MiB	25.44 tok/s	67.54 tok/s	+165.5%	+15.0%	-55.7%
GPT-OSS	1,924 MiB	21.38 tok/s	51.47 tok/s	+140.7%	+27.2%	-55.1%
LFM2.5	946 MiB	57.14 tok/s	320.18 tok/s	+460.3%	+49.1%	-77.2%
Nemotron	3,146 MiB	29.48 tok/s	42.90 tok/s	+45.5%	+6.5%	-26.0%

xychart-beta
    title "4 GiB NP1 decode: cache0 and selected cache"
    x-axis ["Q S", "Q C", "Orn S", "Orn C", "Gem S", "Gem C", "GPT S", "GPT C", "Nemo S", "Nemo C"]
    y-axis "tokens/s" 0 --> 110
    bar [27.46, 77.63, 38.24, 92.00, 25.44, 67.54, 21.38, 51.47, 29.48, 42.90]

Loading

LFM is separated because its 320.18 tok/s result would flatten the other bars.

xychart-beta
    title "LFM2.5 4 GiB NP1 decode"
    x-axis ["Cache0", "Cache24"]
    y-axis "tokens/s" 0 --> 350
    bar [57.14, 320.18]

Loading

Tested settings that exceed 4 GiB

These configurations completed on the 16 GB test GPU, but their measured application peak is above 4,096 MiB. They are non-fits for this guide, not CUDA failures.

Model	Setting	Loaded / peak VRAM	Decode	Ruling
Qwen3.6	cache 24 / -ncmoe 20	4,180 / 4,432 MiB	91.88 tok/s	Exceeds cap by 336 MiB
Ornith	cache 48 / -ncmoe 20	4,672 / 4,924 MiB	114.86 tok/s	Exceeds cap by 828 MiB
Gemma 4	cache 24 / -ncmoe 40	4,700 / 5,074 MiB	89.23 tok/s	Exceeds cap by 978 MiB
GPT-OSS	cache 12 / -ncmoe 24	5,522 / 5,758 MiB	96.90 tok/s	Exceeds cap by 1,662 MiB
Nemotron	cache 12 / -ncmoe 999	4,084 / 4,836 MiB	58.76 tok/s	Loaded memory is close, but request peak exceeds the cap by 740 MiB

Cache 24 was the largest LFM setting tested. It fits at 3,716 MiB; no above-cap LFM boundary is claimed.

Copyable server commands

The commands below keep the measured performance and memory settings. --jinja is included for normal chat use. The benchmark itself used the completion endpoint with the sampling contract shown above. Replace only the model path and port before the first run.

Qwen3.6-35B-A3B Q4_K_M

./build/bin/llama-server \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 4096 -b 2048 -ub 512 -np 1 -n 256 -t 12 \
  -ngl all -ncmoe 20 --fit off \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none --no-warmup --cache-ram 0 \
  --moe-expert-cache-size 12 --spec-type none \
  --temp 0 --top-p 1 --min-p 0 --seed 12345 --jinja

Ornith-1.5-35B-A3B I-Compact

./build/bin/llama-server \
  --model /path/to/Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 4096 -b 2048 -ub 512 -np 1 -n 256 -t 12 \
  -ngl all -ncmoe 20 --fit off \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none --no-warmup --cache-ram 0 \
  --moe-expert-cache-size 24 --spec-type none \
  --temp 0 --top-p 1 --min-p 0 --seed 12345 --jinja

Gemma-4-26B-A4B Q4_0

./build/bin/llama-server \
  --model /path/to/gemma-4-26B-A4B-it-Q4_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 4096 -b 2048 -ub 512 -np 1 -n 256 -t 12 \
  -ngl all -ncmoe 40 --fit off \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none --no-warmup --cache-ram 0 \
  --moe-expert-cache-size 12 --spec-type none \
  --temp 0 --top-p 1 --min-p 0 --seed 12345 --jinja

GPT-OSS-20B MXFP4

./build/bin/llama-server \
  --model /path/to/gpt-oss-20b-MXFP4.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 4096 -b 2048 -ub 512 -np 1 -n 256 -t 12 \
  -ngl all -ncmoe 24 --fit off \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none --no-warmup --cache-ram 0 \
  --moe-expert-cache-size 4 --spec-type none \
  --temp 0 --top-p 1 --min-p 0 --seed 12345 --jinja

LFM2.5-8B-A1B Q3_K_M

./build/bin/llama-server \
  --model /path/to/lfm25-8b-a1b-Q3_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 4096 -b 2048 -ub 512 -np 1 -n 256 -t 12 \
  -ngl all -ncmoe 32 --fit off \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none --no-warmup --cache-ram 0 \
  --moe-expert-cache-size 24 --spec-type none \
  --temp 0 --top-p 1 --min-p 0 --seed 12345 --jinja

NVIDIA Nemotron 3.5 Lightning 30B-A3B Q4_K_M

./build/bin/llama-server \
  --model /path/to/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 4096 -b 2048 -ub 512 -np 1 -n 256 -t 12 \
  -ngl all -ncmoe 999 --fit off \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none --no-warmup --cache-ram 0 \
  --moe-expert-cache-size 4 --spec-type none \
  --temp 0 --top-p 1 --min-p 0 --seed 12345 --jinja

Important interpretation notes

• Cache slots are per cached expert tensor. Cache 12 can require very different VRAM on Qwen, Gemma, or GPT-OSS.
• A nonzero cache overrides ordinary CPU-MoE placement for matching routed expert tensors. The -ncmoe values above are retained because they are part of the measured launch contract; they do not cap the active expert cache.
• Qwen, Ornith, Gemma, GPT-OSS, and LFM produced grouped planner kernels in the selected profiles. Nemotron cache4 did not: four slots are below the routed group width, so it used the established cached tier. Its cache was active and still improved decode, but this is not grouped-path evidence.
• Cache0 and positive-cache output hashes differ because the expert computation moves from CPU to GPU. This is the established placement-dependent numerical path. Each accepted run still had exact accounting, a deterministic repeatable output within its placement, and no malformed-output or runtime error.
• These are short-context NP1 measurements. They do not prove a 64K context or parallel request pool fits in 4 GiB. Increase -c or -np only after measuring the new peak.
• Non-mmap loading keeps the cold model source in system RAM. If the complete GGUF does not fit, prefer a smaller model. A mapped source can use the unified host budget, but page faults and staging can sharply change both target and speculative throughput.

System RAM planning

Model file	Approximate file size	Practical non-mmap RAM
LFM2.5 Q3_K_M	4.1 GB	8 GB minimum; 16 GB preferred
GPT-OSS MXFP4	12.1 GB	24 GB minimum; 32 GB preferred
Gemma 4 Q4_0	14.6 GB	24 GB minimum; 32 GB preferred
Ornith I-Compact	17.4 GB	32 GB
Qwen3.6 Q4_K_M	20.4 GB	32 GB minimum
Nemotron Q4_K_M	25.0 GB	32 GB minimum; 48 GB preferred

All hardware guides - 8 GB - 12 GB - 16 GB - Evidence - MoE cache flags - Home