Notable Runs

Notable runs

This page lists only owner-verified runs from GenerelSchwerz/llama.cpp. Each model has its own section. Comparison tables do not merge the underlying run records.

Qwen3.8 Flash Next

Hardware for the owner runs: RTX 5070 Ti 16 GB, 62.1 GiB system RAM, Linux, CUDA, Qwen3.8 Flash Next Q3_K_XL, and moe-cache commit 925933801.

Owner-verified runs

64K F16 full pin with MTP

This run used a matched Q4_K_M MTP sidecar and generated 1,024 tokens after a 158-token prompt.

Measurement	Result
Prefill	136.83 tok/s
Decode	61.68 tok/s
TTFT	1.163 s
Prompt / generated	158 / 1,024 tokens
Context capacity	65,536 tokens
KV	F16 K / F16 V
Loaded / peak VRAM	13,795 / 14,467 MiB
Cache	56 slots per expert tensor
MTP accepted / drafted	579 / 887, 65.28%
Grouped coverage	49 / 49
Grouped fallback / rollback / errors	0 / 0 / 0
Cache hits / misses / evictions	24,135 / 16,202 / 8,138

The output remained coherent and valid UTF-8. It used the full token limit in reasoning_content before producing final-answer content, so treat this as a decode benchmark.

Launch:

env \
  CUDA_VISIBLE_DEVICES=0 \
  GGML_CUDA_MOE_EARLY_ROUTER=1 \
  GGML_CUDA_MOE_EARLY_ROUTER_LOOKAHEAD=1 \
  LD_LIBRARY_PATH="$PWD/build/bin" \
./build/bin/llama-server \
  --offline \
  --model ./models/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf \
  -md ./models/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -c 65536 -b 512 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 56 \
  -ctk f16 -ctv f16 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling \
  --decode-overlap \
  --decode-boundary-overlap \
  --ple-prefetch \
  --phase-aware-workspace \
  --live-context-workspace \
  --experimental-logs \
  -n 1024 \
  --host 127.0.0.1 --port 8080

Omitting --moe-expert-cache-host-pinned-mb requests full pinning. Windows users must follow Windows WDDM setup instead of copying that omission.

64K Q8 full pin

The short and long requests used separate cold launches with cache80, Q8 KV, no MTP, and the full feature composition shown below.

Short prompt

Measurement	Result
Prefill	115.86 tok/s
Decode	55.54 tok/s
TTFT / wall	1.390 / 19.814 s
Prompt / generated	158 / 1,024 tokens
Context capacity	65,536 tokens
Loaded / peak VRAM	14,681 / 15,145 MiB
Cache / grouped coverage	80 / 48 of 48
Decode overlap queued / discarded	1,023 / 0
Cache hits / misses / evictions	30,957 / 20,847 / 9,510
Grouped fallback / rollback / errors	0 / 0 / 0

The output remained coherent but exhausted the reasoning token limit before final-answer content.

Long prompt

Measurement	Result
Prefill	442.35 tok/s
Prefill time	16.688 s
Decode	48.52 tok/s over 64 tokens
Wall time	18.009 s
Prompt / generated	7,382 / 64 tokens
Context capacity	65,536 tokens
Loaded / peak VRAM	14,681 / 15,571 MiB
Cache / grouped coverage	80 / 48 of 48
Cache hits / misses / evictions	121,998 / 143,608 / 132,271
Grouped fallback / rollback / errors	0 / 0 / 0

The long fixture contains 27,609 bytes of llama.cpp documentation plus the deployment prompt. Its SHA-256 is 7b14ea48a1fbc13672693fb2606af5cfcca80de913c95f97580924c61dd5dcb4.

Launch either request with:

env \
  CUDA_VISIBLE_DEVICES=0 \
  GGML_CUDA_MOE_EARLY_ROUTER=1 \
  GGML_CUDA_MOE_EARLY_ROUTER_LOOKAHEAD=1 \
  LD_LIBRARY_PATH="$PWD/build/bin" \
./build/bin/llama-server \
  --offline \
  --model ./models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf \
  --spec-type none \
  -c 65536 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling \
  --decode-overlap \
  --decode-boundary-overlap \
  --ple-prefetch \
  --phase-aware-workspace \
  --live-context-workspace \
  --experimental-logs \
  -n 1024 \
  --alias moe-cache-all-flags-fullpin-64k \
  --host 127.0.0.1 --port 8080

Use -n 1024 for the short request. Use -n 64 for the long request. Do not add a finite reasoning budget because it conflicts with backend sampling and disables target decode overlap.

64K Q8 partial pin with a 24 GB host budget

These independent cold launches changed only the host source from full pinning to --moe-expert-cache-host-pinned-mb 22888. They exercised the Windows-shaped bounded path on Linux, not native WDDM.

Short prompt

Measurement	Result
Prefill	31.49 tok/s
Decode	36.18 tok/s server, 36.19 tok/s client
Decode / wall time	28.272 / 33.303 s
Prompt / generated	158 / 1,024 tokens
Context capacity	65,536 tokens
Loaded / peak VRAM	14,621 / 15,085 MiB
Cache / grouped coverage	80 / 48 of 48
Grouped fallback / rollback / errors	0 / 0 / 0

The output matched the Q8 full-pin short output. Cache traffic also matched.

Long prompt

Measurement	Result
Prefill	194.62 tok/s
Prefill time	37.930 s
Decode	33.69 tok/s over 64 tokens
Wall time	39.894 s
Prompt / generated	7,382 / 64 tokens
Context capacity	65,536 tokens
Loaded / peak VRAM	14,621 / 15,511 MiB
Direct source bytes	23,370,141,696
Peak pinned bytes	23,908,143,104
Grouped fallback / rollback / errors	0 / 0 / 0

Prefill packed 289,629.43 MiB on the CPU in 20.992 seconds. The run completed 2,498 helper jobs and 406,986 staging tiles. The output matched the full-pin long output.

Launch either request with:

env \
  CUDA_VISIBLE_DEVICES=0 \
  GGML_CUDA_MOE_EARLY_ROUTER=1 \
  GGML_CUDA_MOE_EARLY_ROUTER_LOOKAHEAD=1 \
  LD_LIBRARY_PATH="$PWD/build/bin" \
./build/bin/llama-server \
  --offline \
  --model ./models/Qwen3.8-Flash-Next-GGUF/UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf \
  --spec-type none \
  -c 65536 -b 4096 -ub 512 -np 1 -t 12 \
  -ngl all -fa on -fit off \
  --load-mode none --lazy-mode on \
  --moe-expert-cache-size 80 \
  --moe-expert-cache-host-pinned-mb 22888 \
  -ctk q8_0 -ctv q8_0 -kvo \
  --cache-ram 0 --jinja --no-warmup \
  --backend-sampling \
  --decode-overlap \
  --decode-boundary-overlap \
  --ple-prefetch \
  --phase-aware-workspace \
  --live-context-workspace \
  --experimental-logs \
  -n 1024 \
  --alias moe-cache-all-flags-partialpin-64k \
  --host 127.0.0.1 --port 8080

Use -n 1024 for the short request and -n 64 for the long request. Follow Windows WDDM setup before selecting a native Windows budget.

Full-pin versus partial-pin

Short prompt

Measurement	Full pin	24 GB partial pin	Partial / full
Prefill, tok/s	115.86	31.49	27.18%
Decode, tok/s	55.54	36.18	65.14%
Wall time, s	19.814	33.303	1.68x
Loaded / peak VRAM, MiB	14,681 / 15,145	14,621 / 15,085	-

Long prompt

Measurement	Full pin	24 GB partial pin	Partial / full
Prefill, tok/s	442.35	194.62	44.00%
Prefill time, s	16.688	37.930	2.27x
Decode, tok/s	48.52	33.69	69.44%
Wall time, s	18.009	39.894	2.22x
Loaded / peak VRAM, MiB	14,681 / 15,571	14,621 / 15,511	-

The partial-pin path spends extra CPU time packing pageable experts into pinned staging. These single runs describe this host and workload; they do not predict every Windows system.

Earlier public baseline

Commit b46f7f7a4 produced 47.00 tok/s after a 158-token prompt with cache80, 12,288-token Q8 context, no MTP, and no predictive prefetch. The run generated 1,024 tokens, reached final-answer content, and peaked at 14,384 MiB VRAM.

Read Flash Next 47 tok/s evidence for the measurement record, output hashes, and full response.

Shared short request

Send this request to /v1/chat/completions. Match model to the launch alias, or omit it when the server exposes only one model.

{
  "temperature": 0.0,
  "top_p": 1.0,
  "min_p": 0.0,
  "presence_penalty": 0.0,
  "frequency_penalty": 0.0,
  "seed": 12345,
  "cache_prompt": false,
  "stream": true,
  "model": "moe-cache-all-flags-fullpin-64k",
  "max_tokens": 1024,
  "stream_options": {"include_usage": true},
  "messages": [{
    "role": "user",
    "content": "Act as the release operator. First reason through the deployment hazards, dependencies, and rollback criteria. Then explicitly finish reasoning and produce a final executable rollout plan dominated by shell commands and configuration snippets. Deploy the payments API to the blue canary pool, hold traffic at ten percent, verify latency and error budgets, and publish a signed go-or-rollback decision. Keep the reasoning brief enough to leave most of the response budget for the final plan. Include concrete scripts with error handling, configuration examples, and verification commands rather than only prose.\n"
  }]
}

Hardware guides - Owner evidence - MoE flags - Home