8GB VRAM Setup

8 GB VRAM setup

Use this conservative starting point for one NVIDIA GPU with about 8 GB VRAM. Build moe-cache with CUDA.

Important

CUDA REQUIRED FOR MOE CACHING: Do not use --moe-expert-cache-size on Vulkan, ROCm/HIP, Metal, SYCL, CPU-only, or another non-CUDA backend. The ordinary placement guidance remains available on other supported backends.

Important

WINDOWS USERS: Follow Windows WDDM partial-pinning setup before copying a MoE launch from this page.

Warning

The owner measured these ceilings on a 16 GB RTX 5070 Ti, not a physical 8 GB card. The Qwen run peaked at 7,539 MiB; start lower on a display GPU.

Recommended start

For a 35B-class sparse Q4 model, use one request, Q8 KV, a 512-token physical batch, and a smaller cache than the measured boundary:

./build/bin/llama-server \
  --model /path/to/model.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl all -fit off \
  -c 16384 -b 8192 -ub 512 -np 1 -t 12 \
  -fa on -ctk q8_0 -ctv q8_0 -kvo \
  --load-mode none \
  --moe-expert-cache-size 40 \
  --cache-ram 0 \
  --jinja

Use Qwen3.6-35B-A3B Q4_K_M as the closest exact public artifact to the measured Qwen configuration. This file is 20.4 GB, so 32 GB system RAM is the practical minimum for non-mmap loading.

This start is intentionally lighter than the verified cache-48, F16-KV, 64K-pool run. Increase context first only if KV and workspace headroom remain. Increase cache only after observing misses and peak VRAM.

Verified by GenerelSchwerz

Model and exact artifact	Workload	Main geometry	Prefill	Decode	Loaded / peak VRAM	Result
Qwen3.6-35B-A3B Q4_K_M	NP4; 4 x 7,288 prompt + 1,024 output; 65,536 total context	cache 48; ub=512; F16 KV	358.63-764.10 tok/s per request	135.78 tok/s aggregate; 33.97-38.51 per request	7,277 / 7,539 MiB	Exact accounting, four length stops, zero server errors, valid Nsight profile
Ornith-1.5-35B-A3B APEX MTP I-Compact	NP1; 64,904 prompt + 512 output; 65,536 context	cache 48; ub=8192; F16 KV; speculation off	4,257.42 tok/s server prompt eval	86.33 tok/s request; 86.50 aggregate	6,905 / 7,296 MiB	Exact length accounting; output was carried in the reasoning field; valid Nsight profile

The Qwen prefill values are server phase timings for four overlapped requests, so they are a range rather than an aggregate. The Ornith result is a long single-stream phase measurement. See Owner-verified benchmark evidence for source revisions and methodology.

What to change when it does not fit

Apply changes in this order:

1. Lower --moe-expert-cache-size by 8 slots.
2. Keep -ub 512; do not raise it while diagnosing memory.
3. Lower -c, or keep Q8 KV if F16 was selected.
4. Close other GPU applications and check actual free VRAM before launch.
5. If system RAM cannot hold the allocated model, prefer a smaller model. An mmap deployment remains file-backed and can use the unified host budget, but page residency and storage behavior can dominate both target and speculative performance.

Do not reduce the cache below the routed experts required by a grouped row and then assume the grouped path remains active. Add --experimental-logs for one diagnostic request and check moe-grouped-decode as described in the MoE cache flag guide.

System RAM choices

RAM	Practical interpretation
16 GB	Too small for the verified Qwen and Ornith files with non-mmap loading. Use a smaller GGUF. A mapped source may fit, but page faults and staging can dominate and must be tested with the intended target or speculative workload.
24 GB	Borderline for the 17.4 GB Ornith file after OS and runtime allocations; not recommended for the 20.4 GB Qwen file.
32 GB	Practical minimum for the two verified 35B-class files in a dedicated process.
64 GB	Better for parallel services, larger GGUFs, filesystem cache, and a separate draft model.

All hardware guides - 4 GiB - 12 GB - 16 GB - MoE cache flags - Home