Unprofiled FreeToken MoE Cache Stock Matrix

Unprofiled FreeToken, MoE cache, and stock matrix

This matrix compares FreeToken, the moe-cache fork, and pristine current llama.cpp on one RTX 5070 Ti. All six accepted measurements ran without Nsight Systems, Nsight Compute, or another profiler.

Result

Qwen3.6 35B A3B NVFP4, one request

Runtime	Residency / placement	Prompt tokens	TTFT	Effective prefill	Decode	Request wall	Peak VRAM
FreeToken	5,120 MiB expert cache, 50% experts	64,009	20.267 s	3,158.36 tok/s	118.766 tok/s	28.880 s	14,457 MiB
moe-cache fork	128 of 256 experts per tensor, 50% experts	64,009	15.699 s	4,077.33 tok/s	115.925 tok/s	24.527 s	14,069 MiB
Pristine llama.cpp	21 CPU MoE layers	64,009	14.655 s	4,367.59 tok/s	46.329 tok/s	36.738 s	14,541 MiB

The fork delivered 97.61% of FreeToken decode speed. It reduced TTFT by 22.54% and total request wall time by 15.07%. Pristine llama.cpp matched FreeToken within 84 MiB peak VRAM, but delivered 39.01% of FreeToken decode speed.

Gemma 4 26B A4B Q4_0, one request

Runtime	Residency / placement	Prompt tokens	TTFT	Effective prefill	Decode	Request wall	Peak VRAM
FreeToken	1,920 MiB expert cache, 50% experts	57,776	81.101 s	712.40 tok/s	105.630 tok/s	90.776 s	12,759 MiB
moe-cache fork	64 of 128 experts per tensor, 50% experts	57,782	13.155 s	4,392.55 tok/s	104.747 tok/s	22.923 s	11,993 MiB
Pristine llama.cpp	13 CPU MoE layers	57,782	13.548 s	4,265.06 tok/s	49.872 tok/s	34.061 s	12,947 MiB

The fork delivered 99.16% of FreeToken decode speed. It reduced TTFT by 83.78% and total request wall time by 74.75%. Pristine llama.cpp matched FreeToken within 188 MiB peak VRAM, but delivered 47.21% of FreeToken decode speed.

Fairness controls

This comparison uses two different controls because the runtimes expose different placement models.

1. FreeToken and the fork use the same logical expert residency. Qwen keeps 128 of 256 experts per layer resident. Gemma keeps 64 of 128 experts per layer resident. FreeToken expresses those capacities as 5,120 MiB and 1,920 MiB. The fork expresses them as 128 and 64 expert slots per tensor.
2. Pristine llama.cpp has no expert-slot cache. Its closest control is whole-layer CPU placement. --n-cpu-moe 21 matched Qwen FreeToken peak VRAM within 0.58%. --n-cpu-moe 13 matched Gemma FreeToken peak VRAM within 1.47%.
3. The fork uses less peak VRAM than FreeToken because residency, not allocator footprint, is the control for that pair. Do not reinterpret those two rows as a VRAM-matched comparison.
4. Every run used one request, a 65,536-token context, temperature 0, top-p 1, min-p 0, top-k disabled, seed 12345, and about 1,024 generated tokens. The llama-server rows used F16 KV. FreeToken used --dtype bfloat16 and its runtime-default KV representation.
5. Every row used a fresh server process and a new result directory. Both llama-server builds used --no-warmup. FreeToken performed its normal startup graph capture and short prefill warmup before becoming ready. The harness stopped each server afterward and confirmed that its process, port, and GPU allocation cleared.

Tested system and source

Component	Recorded value
Date	September 14, 2026
GPU	NVIDIA GeForce RTX 5070 Ti, 16,303 MiB reported total
Driver	615.71.09
CUDA toolkit	13.3
CPU	Intel Core Ultra 9 285K, 24 cores / 24 logical CPUs
System RAM	About 62 GiB usable
OS	CachyOS Linux, Linux 7.2.4-3-cachyos
Compiler	GCC 16.2.1
FreeToken	FlashML-org/FreeToken af71ba432
MoE cache fork	GenerelSchwerz/llama.cpp c5635e882
Pristine llama.cpp	ggml-org/llama.cpp 7cf1c54a9

Recorded executable hashes:

Build	File	SHA-256
Fork	llama-server	7c815c4e48383b935356e6d33fb7f784b464a6a508d1996fcbd58b0becdeefd0
Fork	libggml-cuda.so	04d343c676e9a38771208933285164edb8634f35aec9eadef6769fce4bcb8399
Stock	llama-server	69953c703bf8f8142a97f1bd823aa1e29ca16a7bec215ec0f8fd2435d023c830
Stock	libggml-cuda.so	b2f798287545117c7e2fcb65d8f72404eb713b1f3a5caad776c08a22a4e8489b
Stock	libllama.so	5f07179ab5c92e2222d58eea5830184556b30ce8affb8729177280128e6e5ca1

The unprofiled collection harness SHA-256 was 50e84202e0a13fa5a0198c582101924a9186573830c14f61ce176d96cd56d452.

Effective prefill is prompt tokens divided by client-observed TTFT. It includes request handling and the first streamed output event; it is not a kernel-only prompt-eval measurement.

Models

Qwen

FreeToken used the native NVIDIA Qwen3.6 35B A3B NVFP4 checkpoint. The llama runtimes used the corresponding Q8-dense/NVFP4-expert GGUF. This is the closest supported runtime-native pairing, but it is not a byte-identical model artifact. Treat the Qwen matrix as an implementation comparison that includes container and dense-weight representation differences.

Artifact	Bytes	SHA-256
Native shard 1	10,006,877,608	07141c2db92e47bc08777132cd1a0323faf300eab3a7d7c111bc2bf075fda050
Native shard 2	10,003,595,752	6dea9c759a0f941cf9e1cc1501216b0e107966a66c90425050e212efbd053f02
Native shard 3	3,413,864,960	9758875fc55e49561165f4a44342b654c123c3e25c6811b34abb83553fb1a164
Native index	-	d67403a4e9793c0ba8a136baf14b3b76ec7b32c822267978084895e07ebd8a3e
Native config	-	58aefa1c9eff7989f431d748f2ddec39446cb1fd2a69acc46e285c6a37b0cecc
Native tokenizer	-	5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42
Q8/NVFP4 GGUF	20,960,561,856	0d223b5b85f5970216ce7d9f622a8b916f0de415e190cb489acbb3035a36bf62

Gemma

All three Gemma runs used the exact same Google GGUF:

Artifact	Bytes	SHA-256
gemma-4-26B_q4_0-it.gguf	14,439,363,584	3eca3b8f6d7baf218a7dd6bba5fb59a56ee25fe2d567b6f5f589b4f697eca51d

Prompts and requests

Model	Prompt	Bytes	SHA-256	Tokenization
Qwen	llama.cpp deployment dossier and deployment-review task	222,033	7388a8cec3ab3fe15db20b361da530f463647ac89585ef60aced1583d9f6878e	64,009 tokens in all three runtimes
Gemma	Northstar records 0001-0424 and incident memo task	160,407	5bea4c4134fb7b477b9b7ff1743a0e34f40dff1ca3fc24d149d64fc60e2b4807	57,776 FreeToken; 57,782 llama-server

The Gemma prompt ends with this exact task:

FINAL USER TASK

Using only this archive, write a concise incident-response memo for a reliability review. Analyze Records 0073 and 0311. The request also names Record 0431; explicitly state whether that record exists inside the declared 0001 through 0424 scope instead of inventing it. For each available named record, report the date, region, service, owner, change, first-canary observation, decision, mitigation, and verification. Compare the supported incidents, separate direct evidence from inference, identify one shared control weakness, and propose a five-step prevention checklist. Use readable prose with short headings. Do not copy the archive wholesale. Aim for 700 to 900 words and end with a one-sentence conclusion.

The Qwen prompt ends with this exact task:

Deployment question: A team is preparing a reproducible llama.cpp service for long mixed natural-language and code workloads. Produce a technically detailed review of the build, server configuration, request behavior, observability, and validation plan. Explain the important tradeoffs and failure modes, cite relevant commands or JSON shapes from the dossier, and organize the response so an engineer can act on it.

The request body for each run had this shape. Replace MODEL_ALIAS and PROMPT_TEXT with the values for that row.

{
  "model": "MODEL_ALIAS",
  "messages": [{"role": "user", "content": "PROMPT_TEXT"}],
  "max_tokens": 1024,
  "stream": true,
  "stream_options": {"include_usage": true},
  "temperature": 0.0,
  "top_p": 1.0,
  "min_p": 0.0,
  "top_k": 0,
  "seed": 12345,
  "presence_penalty": 0.0,
  "frequency_penalty": 0.0,
  "cache_prompt": false
}

The Qwen FreeToken request set max_tokens to 1,025 and returned 1,024 tokens. The other Qwen requests set it to 1,024 and returned 1,024. The Gemma FreeToken request set it to 1,024 and returned 1,023; both Gemma llama-server requests returned 1,024. These are the recorded API values, not normalized replacements.

FreeToken does not provide llama-server's /tokenize endpoint. Its runs record token ID capture as unsupported rather than failed. Timing, streamed output, API usage accounting, and output hashes were retained.

Early-router environment check

The headline fork rows do not set an undocumented performance environment override. A follow-up unprofiled run tested the current opt-in pair:

GGML_CUDA_MOE_EARLY_ROUTER=1
GGML_CUDA_MOE_EARLY_ROUTER_LOOKAHEAD=1

Both Qwen and Gemma logged moe-prepack: disabled reason=no adjacent staged groups. Peak VRAM and output hashes were unchanged. Qwen measured 117.276 tok/s and Gemma measured 104.635 tok/s, versus 115.925 and 104.747 tok/s in the predetermined headline runs. Because prepack did not engage, the small Qwen difference is run variation, not an early-router speedup. The retired GGML_CUDA_MOE_EARLY_ROUTER_COPY_* variables must not be restored; current code disables prepack when those retired settings are nonzero.

Exact launch settings

Set these paths before using the launch blocks:

export FT_ROOT=/path/to/FreeToken
export FORK_ROOT=/path/to/GenerelSchwerz-llama.cpp
export STOCK_ROOT=/path/to/pristine-llama.cpp
export QWEN_NATIVE=/path/to/Qwen3.6-35B-A3B-NVFP4
export QWEN_GGUF=/path/to/Qwen3.6-35B-A3B-NVFP4-Q8-NVFP4.gguf
export GEMMA_GGUF=/path/to/gemma-4-26B_q4_0-it.gguf

Qwen FreeToken

CUDA_VISIBLE_DEVICES=0 "$FT_ROOT/.venv/bin/ft" serve \
  --model "$QWEN_NATIVE" \
  --served-model-name qwen36-freetoken-matrix \
  --dtype bfloat16 \
  --max-running-requests 1 \
  --max-seq-len-override 65536 \
  --max-output-tokens 1025 \
  --memory-ratio 0.96 \
  --host 127.0.0.1 --port 24101 \
  --cuda-graph-max-bs 1 \
  --max-prefill-length 8192 \
  --cache-type naive \
  --moe-backend offload \
  --nvfp4-backend auto \
  --moe-cache-size 5120 \
  --num-tokens 65537 \
  --moe-prefill-hit-d2d \
  --decode-log-interval 128 \
  --reasoning-parser off

Qwen MoE cache fork

CUDA_VISIBLE_DEVICES=0 \
LD_LIBRARY_PATH="$FORK_ROOT/build-notable-64k/bin" \
"$FORK_ROOT/build-notable-64k/bin/llama-server" \
  --offline --model "$QWEN_GGUF" \
  --alias qwen36-moecache-matrix \
  --host 127.0.0.1 --port 24102 \
  -c 65536 -b 8192 -ub 8192 -np 1 \
  -ctk f16 -ctv f16 -kvo \
  --load-mode none --lazy-mode on --no-warmup \
  -fa on -ngl all -fit off \
  --moe-expert-cache-size 128 \
  --moe-expert-cache-host-pinned-mb 0 \
  --cache-ram 0 \
  --phase-aware-workspace --live-context-workspace \
  --jinja --reasoning-format none \
  --spec-type none --backend-sampling \
  --decode-overlap --decode-boundary-overlap --ple-prefetch \
  --experimental-logs -n 1024 --log-colors off

Qwen pristine llama.cpp

CUDA_VISIBLE_DEVICES=0 \
LD_LIBRARY_PATH="$STOCK_ROOT/build-cuda/bin" \
"$STOCK_ROOT/build-cuda/bin/llama-server" \
  --offline --model "$QWEN_GGUF" \
  --alias qwen36-stock-matrix \
  --host 127.0.0.1 --port 24103 \
  -c 65536 -b 8192 -ub 8192 -np 1 \
  -ctk f16 -ctv f16 -kvo \
  --load-mode none --no-warmup \
  -fa on -ngl all -fit off \
  --n-cpu-moe 21 --cache-ram 0 \
  --jinja --reasoning-format none \
  --spec-type none -n 1024 --log-colors off

Gemma FreeToken

CUDA_VISIBLE_DEVICES=0 "$FT_ROOT/.venv/bin/ft" serve \
  --model "$GEMMA_GGUF" \
  --served-model-name gemma4-freetoken-matrix \
  --dtype bfloat16 \
  --max-running-requests 1 \
  --max-seq-len-override 65536 \
  --max-output-tokens 1024 \
  --memory-ratio 0.96 \
  --host 127.0.0.1 --port 24111 \
  --cuda-graph-max-bs 1 \
  --max-prefill-length 4096 \
  --cache-type naive \
  --moe-backend offload \
  --moe-cache-size 1920 \
  --num-tokens 65536 \
  --moe-prefill-hit-d2d \
  --decode-log-interval 128 \
  --reasoning-parser off

Gemma MoE cache fork

CUDA_VISIBLE_DEVICES=0 \
LD_LIBRARY_PATH="$FORK_ROOT/build-notable-64k/bin" \
"$FORK_ROOT/build-notable-64k/bin/llama-server" \
  --offline --model "$GEMMA_GGUF" \
  --alias gemma4-moecache-matrix \
  --host 127.0.0.1 --port 24112 \
  -c 65536 -b 8192 -ub 4096 -np 1 \
  -ctk f16 -ctv f16 -kvo \
  --load-mode none --lazy-mode on --no-warmup \
  -fa on -ngl all -fit off \
  --moe-expert-cache-size 64 \
  --moe-expert-cache-host-pinned-mb 0 \
  --cache-ram 0 \
  --phase-aware-workspace --live-context-workspace \
  --jinja --reasoning-format none \
  --spec-type none --backend-sampling \
  --decode-overlap --decode-boundary-overlap --ple-prefetch \
  --experimental-logs -n 1024 --log-colors off

Gemma pristine llama.cpp

CUDA_VISIBLE_DEVICES=0 \
LD_LIBRARY_PATH="$STOCK_ROOT/build-cuda/bin" \
"$STOCK_ROOT/build-cuda/bin/llama-server" \
  --offline --model "$GEMMA_GGUF" \
  --alias gemma4-stock-matrix \
  --host 127.0.0.1 --port 24113 \
  -c 65536 -b 8192 -ub 4096 -np 1 \
  -ctk f16 -ctv f16 -kvo \
  --load-mode none --no-warmup \
  -fa on -ngl all -fit off \
  --n-cpu-moe 13 --cache-ram 0 \
  --jinja --reasoning-format none \
  --spec-type none -n 1024 --log-colors off

Output validation

Model	Runtime	Completion	Finish	Output bytes	Output SHA-256
Qwen	FreeToken	1,024	length	3,996	3474e47ec6520924b5f7b19fe09b9084f172ed663f9d4e7ac6790cfe2f40ca59
Qwen	Fork	1,024	length	4,044	6420a0f5cc0aff721813a2246e83b421b0db68d7bbf9d05072ec81473e06fe48
Qwen	Stock	1,024	length	4,027	4f73cf25994d44d277b91925d6ce89fd5928026e013f63d4b66f3fd85f9651f6
Gemma	FreeToken	1,023	length	4,185	a4b7b41fb7a3ca45b4c926f61b7258042c13cc74c652614136fd48d9694fdda5
Gemma	Fork	1,024	length	3,594	7f4bc06e28b63c2fab8e3eb5ab3e60a8ababf667d22325033aad324d191838e6
Gemma	Stock	1,024	length	3,542	e8bf41438ac62a41ab89e63e94f8af6952f1467644716525333f3b78612fbba7

All outputs were valid UTF-8 and nonempty. Qwen produced coherent deployment analysis. Gemma identified Records 0073 and 0311 and explicitly stated that Record 0431 does not exist within the declared scope. The Gemma llama-server rows exposed their reasoning text in content because the launch used --reasoning-format none; all three outputs remained length-limited.

Temperature zero does not require byte-identical output across these rows. Qwen uses different runtime-native containers, and Gemma tokenized to a six-token difference between FreeToken and llama-server. Output hashes detect changes within a repeated configuration; they are not an expected cross-runtime equality condition.

Interpretation

At matched expert residency, the fork is within 2.4% of FreeToken Qwen decode and within 0.9% of FreeToken Gemma decode. The fork has much lower TTFT on both long prompts, especially Gemma. At matched FreeToken peak VRAM, pristine llama.cpp prefill remains fast but decode falls to 39.0% of FreeToken on Qwen and 47.2% on Gemma.

This is one fresh launch and one request per row. It establishes a reproducible point comparison, not a distribution. Repeat the full six-run matrix before attributing small decode differences to a code change.