Owner Verified Benchmark Evidence

Owner-verified benchmark evidence

Use this evidence ledger behind the 4 GiB, 8 GB, 12 GB, and 16 GB guides. Every rate includes its workload geometry. The owner recorded these runs in August and September 2026.

Acceptance rules

A verified row must show:

• successful health and request completion;
• the exact requested completion-token count and length stop;
• no measured server or client error;
• clean process, port, and GPU teardown; and
• a finalized Nsight Systems report and SQLite export with CUDA kernel rows.

Semantic qualifications remain visible. Exact token accounting does not prove a good answer, especially when the frozen prompt or parser produces repetition or exposes reasoning/control fields.

Test hardware and metric definitions

All rows ran on one NVIDIA GeForce RTX 5070 Ti with 16,303 MiB reported VRAM and 62.1 GiB system RAM. A 4, 8, or 12 GB label means observed process peak below that numerical ceiling, not physical-card validation.

• Server prefill is the phase timing printed by llama-server.
• Effective prefill is total prompt tokens divided by the latest first output in a simultaneous fan-out.
• Request decode is one request's post-first-output token rate.
• Aggregate decode is total completion tokens divided by the shared output window.

Effective prefill and aggregate decode include scheduler behavior. They are not isolated CUDA-kernel rates.

4 GiB target matrix at 71bc85d0e

These NP1 runs used -c 4096 -b 2048 -ub 512, Q8_0 K/V, non-mmap loading, no warmup, no speculation, deterministic sampling, and a 256-token completion. Every listed row has exact accounting, a length stop, no runtime error, clean teardown, and a valid full-request Nsight Systems profile. See the 4 GiB setup guide for exact model links, launch commands, non-fit boundaries, and interpretation.

Throughput

Model	Configuration	Prompt	Prefill	Decode
Qwen3.6 Q4_K_M	cache0, -ncmoe 40	2,048	1,246.62 tok/s	27.46 tok/s
Qwen3.6 Q4_K_M	cache12, -ncmoe 20	2,048	1,408.92 tok/s	77.63 tok/s
Ornith I-Compact	cache0, -ncmoe 40	2,048	1,232.83 tok/s	38.24 tok/s
Ornith I-Compact	cache24, -ncmoe 20	2,048	1,424.92 tok/s	92.00 tok/s
Gemma 4 Q4_0	cache0, -ncmoe 40	2,181	1,396.12 tok/s	25.44 tok/s
Gemma 4 Q4_0	cache12, -ncmoe 40	2,181	1,606.10 tok/s	67.54 tok/s
GPT-OSS MXFP4	cache0, -ncmoe 24	1,883	1,593.90 tok/s	21.38 tok/s
GPT-OSS MXFP4	cache4, -ncmoe 24	1,883	2,027.56 tok/s	51.47 tok/s
LFM2.5 Q3_K_M	cache0, -ncmoe 32	1,983	4,068.00 tok/s	57.14 tok/s
LFM2.5 Q3_K_M	cache24, -ncmoe 32	1,983	6,065.22 tok/s	320.18 tok/s
Nemotron Q4_K_M	cache0, -ncmoe 999	2,070	910.32 tok/s	29.48 tok/s
Nemotron Q4_K_M	cache4, -ncmoe 999	2,070	969.89 tok/s	42.90 tok/s

Latency, memory, and path

Model	Configuration	TTFT	Wall	Loaded / peak MiB	Path
Qwen3.6	cache0	1,651.19 ms	10.937 s	2,566 / 2,688	CPU MoE control
Qwen3.6	cache12	1,460.65 ms	4.746 s	3,480 / 3,724	Grouped
Ornith	cache0	1,669.51 ms	8.338 s	1,918 / 2,040	CPU MoE control
Ornith	cache24	1,445.65 ms	4.218 s	3,364 / 3,616	Grouped
Gemma 4	cache0	1,571.42 ms	11.594 s	2,510 / 2,626	CPU MoE control
Gemma 4	cache12	1,366.15 ms	5.142 s	3,598 / 3,946	Grouped
GPT-OSS	cache0	1,191.19 ms	13.117 s	1,816 / 1,924	CPU MoE control
GPT-OSS	cache4	938.59 ms	5.893 s	3,100 / 3,302	Grouped
LFM2.5	cache0	494.70 ms	4.958 s	838 / 946	CPU MoE control
LFM2.5	cache24	334.29 ms	1.131 s	3,550 / 3,716	Grouped
Nemotron	cache0	2,281.53 ms	10.932 s	3,014 / 3,146	CPU MoE control
Nemotron	cache4	2,140.53 ms	8.085 s	2,814 / 3,510	Established cached tier

The selected grouped profiles contain 10,200 Qwen planner launches, 10,200 Ornith launches, 7,650 Gemma launches, 6,120 GPT-OSS launches, and 5,610 LFM launches. Nemotron cache4 contains no grouped planner launch because the cache has fewer slots than its routed group width.

Exact public GGUF evidence

Source revision	Model	Workload and context	Cache / ubatch / KV	Prefill	Decode	Loaded / peak MiB	Profile and output status
de51ad1c6	Qwen3.6-35B-A3B Q4_K_M	NP4; 4 x 7,288 prompt + 1,024 output; 65,536 total context	48 / 512 / F16	358.63-764.10 tok/s server range	135.78 aggregate; 33.97-38.51 per request	7,277 / 7,539	2,449,001 kernel rows; exact; four length stops
3ba96046	Same Qwen	NP1; 64,000 + 512; 65,536 context	132 / 8192 / F16	4,326.76 tok/s server	111.72 request	13,979 / 14,369	905,032 kernel rows; exact; length stop
de51ad1c6	Same Qwen	NP4; 4 x 7,288 + 1,024; 65,536 total context	132 / 8192 / F16	1,142.11-2,243.00 tok/s server range	247.79 aggregate; 62.00-67.78 per request	14,297 / 14,675	2,107,112 kernel rows; exact; four length stops
d0c964a0f	GPT-OSS-20B MXFP4	NP1; 64,000 + 1,024; 65,536 context	24 / 2048 / F16	2,735.58 tok/s server	141.40 request	11,191 / 11,475	308,257 kernel rows; exact; length stop
d659bcefa plus worktree diff f80d8d51	Ornith-1.5-35B-A3B APEX MTP I-Compact	NP1; 64,904 + 512; 65,536 context; no speculation	48 / 8192 / F16	4,257.42 tok/s server	86.33 request	6,905 / 7,296	25,164 kernel rows; exact length; reasoning field only
Same	Same Ornith	NP1; 64,904 + 512; MTP depth 1, p-min 0	48 / 8192 / F16	3,940.10 tok/s server	99.64 request	8,339 / 9,246	53,924 kernel rows; exact length; reasoning field only

The Ornith pair is a matched target-only versus MTP-depth-one comparison. MTP improved request decode about 15.4%, reduced server prefill about 7.5%, and added 1,950 MiB peak VRAM in that run.

Published Gemma 4 fan-out matrix

The matrix used Gemma-4-26B-A4B-it Q4_0, source revision b6168db836, cache 48, F16 KV, total context 65,536, and no speculation. Every row has exact output accounting and a valid full-window Nsight profile.

Fan-out	Prompt + output per request	Ubatch	TTFT range	Effective prefill	Aggregate decode	Per-request decode	Loaded / peak MiB	Semantic status
1	63,247 + 1,024	4096	15.812 s	3,999.88 tok/s	104.578 tok/s	104.48 tok/s	9,809 / 10,291	Repetitive/degenerate continuation
4	15,123 + 512	4096	7.935-12.676 s	4,772.28 tok/s	132.002 tok/s	33.46-47.43 tok/s	12,937 / 13,429	Repetitive/degenerate continuations
8	7,514 + 512	2048	5.546-14.656 s	4,101.48 tok/s	122.541 tok/s	15.46-21.02 tok/s	13,229 / 13,673	Mostly task-relevant; isolated tags/repetition
16	3,448-3,451 + 512	1024	9.313-17.135 s	3,221.31 tok/s	175.563 tok/s	11.03-13.15 tok/s	14,605 / 14,997	Mostly task-relevant; isolated tags/repetition

The NP1 and NP4 prompts were degenerate on both tested runtimes, so their throughput remains useful as a systems measurement but not as a quality endorsement. NP8 and NP16 were qualified rather than declared uniformly sensible. Ubatch decreases with fan-out because larger chunks exceeded the startup memory envelope.

Published Qwen NVFP4-source fan-out matrix

These rows used source revision b6168db836 and a local mixed Q8-dense/NVFP4-expert GGUF converted from NVIDIA Qwen3.6-35B-A3B-NVFP4. The link is the exact source model, not a claim that NVIDIA publishes the same mixed GGUF. The run used cache 96, ub=8192, F16 KV, 65,536 total context, and no speculation.

Fan-out	Prompt + output per request	TTFT min / mean / max	Effective prefill	Aggregate decode	Per-request decode	Loaded / peak MiB	Status
1	64,000 + 1,024	16.274 / 16.274 / 16.274 s	3,932.6 tok/s	105.86 tok/s	105.76 tok/s	11,159 / 11,481	Exact; task-related; valid profile
4	15,500 + 512	6.300 / 11.557 / 15.766 s	3,932.1 tok/s	105.53 tok/s	27.02-51.41 tok/s	11,493 / 11,841	Exact; task-related; valid profile
8	7,288 + 512	6.073 / 12.618 / 16.127 s	3,615.2 tok/s	147.08 tok/s	18.92-28.71 tok/s	11,617 / 11,943	Exact; task-related; valid profile
16	3,500 + 512	5.499 / 12.222 / 16.492 s	3,395.1 tok/s	132.99 tok/s	8.43-10.10 tok/s	12,055 / 12,467	Exact; task-related; profile valid but trace-sensitive

Interpretation limits

• All rates include Nsight Systems tracing overhead. Use them for evidence and relative shape, not as an unprofiled maximum.
• The historical Qwen and Ornith rows predate the publication checkpoint. Their source revisions are recorded so they are not presented as fresh tip-of-branch results.
• The lower-tier labels are application-process peaks on a larger GPU. A physical display card can expose less usable memory than its nominal capacity.
• Cache slots are per cached tensor. The same number can consume very different VRAM on another architecture or quantization.
• Non-mmap source data must remain in host memory. Published GGUF size is not total peak process RSS.
• A strong single-stream rate does not prove parallel behavior. Use aggregate decode and tail TTFT for fan-out.
• Exact completion accounting does not prove token equality against another backend or semantic quality.

Community evidence policy

Keep community results from this fork separate from owner-verified tables. Require the exact model, quant, commit, command, hardware, token counts, parallelism, rates, VRAM, correctness notes, and a durable log.

All hardware guides - 4 GiB - 8 GB - 12 GB - 16 GB - Home