Hardware Setup Guides

Hardware setup guides

Choose a conservative starting configuration by available NVIDIA VRAM. The MoE results use CUDA; ordinary llama.cpp placement still follows upstream backend support.

Important

The owner measured the 4, 8, and 12 GB ceilings on a 16 GB RTX 5070 Ti. They are allocation targets, not tests on physical lower-VRAM cards.

Important

WINDOWS USERS: Follow Windows WDDM setup. Use matched fork DLLs and verify the host-pinning budget that Windows grants.

Pick a guide

VRAM target	Start with	Measured examples	Guide
4 GiB	NP1, 4K context, Q8 KV, ub=512, small model-specific cache	Qwen cache12 at 3,724 MiB; Gemma cache12 at 3,946 MiB; GPT-OSS cache4 at 3,302 MiB	4 GiB
8 GB	NP1, Q8 KV, ub=512, modest cache	Qwen at 7,539 MiB; Ornith at 7,296 MiB	8 GB
12 GB	Long NP1 context, larger cache, or MTP depth one	GPT-OSS at 11,475 MiB; Ornith MTP at 9,246 MiB	12 GB
16 GB	Large cache or measured NP4/NP8/NP16 service	Qwen up to 14,675 MiB; Gemma up to 14,997 MiB	16 GB

Read Owner-verified benchmark evidence for the exact prompt depth, output length, source revision, prefill, decode, and validation status.

Read the labels correctly

• Owner-verified: preserved command and result, exact completion accounting, coherent or explicitly qualified output, and clean teardown.
• Measured under the tier ceiling: process VRAM stayed below the named target on the 16 GB test GPU.
• Recommended start: a conservative configuration derived from evidence, not a separately measured row.
• Community report: a user result from this fork, kept outside owner tables.

Owner tests used an RTX 5070 Ti with 16,303 MiB reported VRAM, 62.1 GiB system RAM, Linux, CUDA, and 12 CPU threads. Hardware, PCIe width, memory bandwidth, storage, quantization, and workload can change the result.

Metrics

Metric	Meaning
Server prefill	prompt eval rate from llama-server
Effective prefill	Total prompt tokens divided by the last first-token time in a simultaneous fan-out
Request decode	One request's generated tokens divided by its decode window
Aggregate decode	All generated tokens divided by the shared output window
Loaded / peak VRAM	Process VRAM after load / highest observed value during the request

Do not add per-request rates to estimate parallel throughput. Use aggregate decode.

System RAM

The GPU expert cache keeps a complete cold source in host memory. With --load-mode none, budget that source, other host tensors, runtime buffers, the operating system, and applications. With mmap, file pages remain reclaimable but page faults can dominate.

Use --load-mode none when the cold source fits. If full pinning fails, use a bounded host budget. Use mmap only when you need file-backed placement.

Model file	File size	Practical allocated-RAM target
LFM2.5-8B-A1B Q3_K_M	4.1 GB	8 GB minimum; 16 GB preferred
Qwen3.6-35B-A3B Q4_K_M	20.4 GB	32 GB minimum
Ornith I-Compact	17.4 GB	32 GB
Gemma-4-26B-A4B Q4_0	14.6 GB	24 GB minimum; 32 GB preferred
GPT-OSS-20B MXFP4	12.1 GB	24 GB minimum; 32 GB preferred
Nemotron 3.5 Lightning 30B-A3B Q4_K_M	25.0 GB	32 GB minimum; 48 GB preferred

These targets include planning margin, not measured peak RSS guarantees.

Tune MoE in this order

1. Build with CUDA and load the model with cache size 0.
2. Set --load-mode none and -fit off.
3. Choose a conservative cache and keep 1 to 2 GiB of VRAM free.
4. Measure prefill and decode separately at the intended context and parallelism.
5. Raise cache slots when decode misses dominate.
6. Raise -ub only when prefill needs it and the workspace fits.
7. Add a host-pinning budget if full pinning fails.
8. Add MTP, overlap, or prefetch only after the target-only baseline passes.

For Docker, use Docker Compose for large MoE models.

4 GiB - 8 GB - 12 GB - 16 GB - MoE flags - Home