Benchmark Comparison Showcase

MoE cache benchmarks: stock llama.cpp vs the fork

Use this index to reproduce MoE cache measurements from one RTX 5070 Ti 16 GB system. Each model page starts with commands and raw results. It omits machine-local artifact paths.

For the current unprofiled three-runtime comparison, see FreeToken, MoE cache, and pristine llama.cpp. It matches FreeToken and the fork by 50% logical expert residency, then matches pristine llama.cpp to FreeToken peak VRAM.

Tested builds

Component	Revision
Stock comparator	ggml-org/llama.cpp 733905474
MoE cache fork	GenerelSchwerz/llama.cpp 0c332d714
Nemotron ungated-layout fix	GenerelSchwerz/llama.cpp 71bc85d0e
Current-stock check	ggml-org/llama.cpp 4cbe8b070 produced the same GPT-OSS output and nearly the same speed as the aligned stock comparator

Test system	Value
GPU	NVIDIA GeForce RTX 5070 Ti, 16 GB
Driver / CUDA toolkit	610.57.04 / CUDA 13.3
CPU threads	12
System RAM	About 62 GiB
Common settings	CUDA, flash attention, F16 KV, --load-mode none, --no-warmup, temperature 0, seed 12345
Safety ceiling	15,872 MiB process VRAM; the harness rejected runs at or above it

How to read the results

Important

Decode speed is only a fair headline comparison when peak VRAM is close. This wiki uses the following labels:

• Close: within 5% peak VRAM. Suitable for direct speed comparison.
• Near match: 5-10% apart. Useful for a failed-leg diagnosis, but not a headline speed ratio.
• Unmatched: more than 10% apart. Kept only as historical context, never as a direct speed claim.

The cache-enabled rows intentionally change expert placement. Output differences between CPU and GPU expert computation are expected numerical-path differences. Same-placement or fully GPU-resident stock/fork controls must match exactly.

Close-VRAM headline results

Model / workload	Stock decode	Cache decode	Change	Stock peak	Cache peak	VRAM delta	Fairness
Qwen3.6 35B NP1	42.793 tok/s	111.620 tok/s	+160.84%	15,240 MiB	14,758 MiB	-3.16%	Close
Qwen3.6 35B NP4	96.919 tok/s	220.725 tok/s	+127.74%	13,634 MiB	14,204 MiB	+4.18%	Close
Gemma 4 NP1	34.218 tok/s	102.164 tok/s	+198.57%	10,282 MiB	10,630 MiB	+3.38%	Close
Qwen3.6 28B REAP NP1	55.816 tok/s	139.511 tok/s	+149.95%	14,082 MiB	14,070 MiB	-0.09%	Close
Gemma 4 NP4	75.035 tok/s	124.105 tok/s	+65.39%	13,304 MiB	13,756 MiB	+3.40%	Close
GPT-OSS NP1	143.901 tok/s	134.614 tok/s	-6.45%	13,558 MiB	13,674 MiB	+0.86%	Close
GPT-OSS NP4	383.875 tok/s	365.838 tok/s	-4.70%	13,526 MiB	13,900 MiB	+2.77%	Close
DeepSeek NP1	6.320 tok/s	7.072 tok/s	+11.89%	9,560 MiB	9,206 MiB	-3.70%	Close
DeepSeek NP4	15.676 tok/s	16.054 tok/s	+2.41%	9,564 MiB	9,210 MiB	-3.70%	Close
LFM Q5 NP1	274.188 tok/s	247.200 tok/s	-9.84%	7,852 MiB	7,728 MiB	-1.58%	Close
LFM Q3 NP1	281.553 tok/s	257.264 tok/s	-8.63%	6,020 MiB	6,094 MiB	+1.23%	Close
Nemotron NP1	57.042 tok/s	114.873 tok/s	+101.40%	15,712 MiB	15,726 MiB	+0.09%	Close
Ornith NP1	35.244 tok/s	97.411 tok/s	+176.39%	9,836 MiB	9,610 MiB	-2.30%	Close
Ornith NP4	72.777 tok/s	164.402 tok/s	+125.90%	8,164 MiB	8,036 MiB	-1.57%	Close

xychart-beta
    title "NP1 decode: larger-model stock/cache pairs"
    x-axis ["Q35 S", "Q35 C", "Gemma S", "Gemma C", "Q28 S", "Q28 C", "GPT S", "GPT C", "Deep S", "Deep C", "Nemo S", "Nemo C", "Orn S", "Orn C"]
    y-axis "tokens/s" 0 --> 160
    bar [42.793, 111.620, 34.218, 102.164, 55.816, 139.511, 143.901, 134.614, 6.320, 7.072, 57.042, 114.873, 35.244, 97.411]

Loading

xychart-beta
    title "NP1 decode: LFM stock/cache pairs"
    x-axis ["Q5 S", "Q5 C", "Q3 S", "Q3 C"]
    y-axis "tokens/s" 0 --> 300
    bar [274.188, 247.200, 281.553, 257.264]

Loading

S means stock and C means cache-enabled.

xychart-beta
    title "NP4 aggregate decode: stock/cache pairs"
    x-axis ["Q35 S", "Q35 C", "Gem S", "Gem C", "GPT S", "GPT C", "Deep S", "Deep C", "Orn S", "Orn C"]
    y-axis "tokens/s" 0 --> 400
    bar [96.919, 220.725, 75.035, 124.105, 383.875, 365.838, 15.676, 16.054, 72.777, 164.402]

Loading

All speed values are tokens/second and all VRAM values are MiB. See the model pages for TTFT, prefill, wall time, exact launch settings, and output notes.

Outstanding matched-memory failure

Model / workload	Stock result	Positive-cache result	Stock peak	Cache peak	VRAM delta	Ruling
LFM Q5 NP4	598.174 tok/s	Failed grouped certificate	7,468 MiB	7,856 MiB	+5.20%	No cache ratio

The LFM run stopped without legacy fallback. Its near-close memory envelope rules out mismatched VRAM as the explanation.

Model reports

Report	Quant / layout	Coverage
Qwen3.6 Q4_K_M	Q4_K_M	35B NP1/NP4 and 28B REAP NP1
Gemma 4 Q4_0	Q4_0	NP1 and NP4
GPT-OSS 20B MXFP4	MXFP4	NP1 and NP4
DeepSeek Coder V2 Lite Q4_K_M	Q4_K_M	NP1 and NP4
LFM2.5 8B A1B Q5/Q3	Q5_K_M / Q3_K_M	NP1, plus Q5 NP4 controls
Nemotron 3.5 Lightning 30B A3B	Q4_K_M, ungated up/down	NP1 stock/cache0/cache72; grouped path validated
Ornith 1.5 35B A3B APEX	I-Compact	Target-only NP1 and NP4

Metric definitions

Metric	Meaning
TTFT	Wall time from request submission to first output token
NP1 prefill	Server-reported prompt-eval throughput
NP4 effective prefill	Sum of actual prompt tokens divided by the slowest request TTFT; includes scheduling and is not kernel-only throughput
NP1 decode	One request's completion throughput
NP4 decode	Aggregate completion throughput across four concurrent requests
Peak VRAM	Highest measured process VRAM during the run

The launch blocks include every model and performance flag from each run. They replace machine-local paths and ports and omit provenance variables and the nsys profile wrapper. Nsight Systems traced every measurement. Completion requests set temperature 0, top-p 1, min-p 0, seed 12345, and the listed output limit.

See Future model coverage for missing architecture and quant coverage.