FreeToken vs llama.cpp on RTX 5060 Ti

A reproducible companion for the NetworkCoder video comparing FreeToken's dynamic expert cache with llama.cpp on a 16GB consumer GPU.

---

📺 Watch the Video

FreeToken vs llama.cpp: Which Is Better for Local AI?

The video contains the full setup, terminal evidence, request outputs, VRAM monitoring, power observations, limitations and verdict.

This repository is a companion to the video. Results are measurements from one machine and should not be treated as universal performance claims.

---

What Was Tested?

• The same oversized 35GB-class Mixture-of-Experts model
• NVIDIA RTX 5060 Ti with 16GB VRAM
• Short-prompt and approximately 4,000-token coding workloads
• Uncached processing and repeated-context reuse
• FreeToken automatic expert-cache selection
• Runtime cache reduction without restarting the server
• VRAM allocation, latency and active GPU power

The model was larger than available VRAM, so the comparison evaluates two different CPU–GPU execution strategies rather than a fully GPU-resident workload.

---

Test Architecture

35GB MoE checkpoint in system memory
              │
      ┌───────┴────────┐
      │                │
  FreeToken        llama.cpp
  expert cache     CPU/GPU offload
      │                │
      └──── RTX 5060 Ti 16GB ────┘

---

Test Configuration

Component	Configuration
GPU	NVIDIA RTX 5060 Ti, 16GB VRAM
Model type	Sparse Mixture-of-Experts
Model size	Approximately 35GB
FreeToken API	http://127.0.0.1:1919/v1
Long workload	Approximately 4,000 input tokens
Repeated test	Same context reused
Manual cache test	3302 MoE slots, approximately 9.7 GiB

Record your exact model revision, quantization, runtime commit and driver version before comparing results.

---

1. Install FreeToken

The official project recommends uv:

uv pip install "freetoken[accel]"

Or build the current source:

git clone https://github.com/FlashML-org/FreeToken.git
cd FreeToken
uv venv
source .venv/bin/activate
uv pip install -e ".[accel]"

---

2. Start the FreeToken Server

ft serve --model /path/to/your/model

Wait until the terminal reports that the API server is available on 127.0.0.1:1919.

Confirm the exposed model ID:

curl http://127.0.0.1:1919/v1/models

---

3. Send a Basic Request

curl http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "PASTE_MODEL_ID",
    "messages": [{"role": "user", "content": "Explain expert caching in one paragraph."}],
    "max_tokens": 256,
    "stream": false
  }'

---

4. Run the Long Coding Workload

Save the complete OpenAI-compatible request as:

prompts/freetoken-code-4k-corrected-request.json

Then measure end-to-end wall time:

time curl -s http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @prompts/freetoken-code-4k-corrected-request.json \
  > results/freetoken-code-4k.json

Repeat the identical request to measure context reuse:

time curl -s http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @prompts/freetoken-code-4k-corrected-request.json \
  > results/freetoken-code-4k-repeat.json

---

5. Inspect and Resize the Expert Cache

Inspect the active cache:

ft ctl --base-url http://127.0.0.1:1919 cache

Apply the manual cache size used in the video:

ft ctl --base-url http://127.0.0.1:1919 cache --moe 3302 --wait 300

The observed status showed approximately:

10.3 GiB allocated
3302 MoE slots
9.7 GiB MoE cache VRAM

FreeToken applied this change while the server remained active. Reducing cache capacity can lower VRAM use, but it may also increase PCIe transfers and reduce performance.

---

6. Run the Same Request Through llama.cpp

Start an OpenAI-compatible llama.cpp server with the same model, context and comparable offload settings:

llama-server \
  --model /path/to/the/same-model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size YOUR_MATCHED_CONTEXT \
  --flash-attn on

Measure the identical request:

time curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data-binary @prompts/freetoken-code-4k-corrected-request.json \
  > results/llamacpp-code-4k.json

Match the model file, prompt, output-token limit, context size and sampling values. Engine-specific scheduling cannot be made identical and is part of what the test measures.

---

Results Shown in the Video

Workload	FreeToken	llama.cpp	Interpretation
Short prompt	Slower	Faster	FreeToken overhead had little opportunity to help
Long, uncached context	Approximately 2× faster	Baseline	Longer prefill benefited FreeToken's execution path
Repeated 4K context	10.669 s	31.419 s	Approximately 2.94× lower wall time in this run

These are end-to-end request times from this test, not universal throughput guarantees. Cache state, generated-token count, model format and runtime versions can materially affect the result.

---

Monitor VRAM and Power

nvidia-smi --query-gpu=timestamp,memory.used,utilization.gpu,power.draw \
  --format=csv -l 1

For a log file:

nvidia-smi --query-gpu=timestamp,memory.used,utilization.gpu,power.draw \
  --format=csv -l 1 > results/gpu-monitor.csv

Stop monitoring with Ctrl+C after the request completes.

---

What the Result Means

FreeToken is not automatically faster for every workload. In this test:

• llama.cpp performed better on short prompts.
• FreeToken performed better once the prompt became much longer.
• FreeToken reused repeated context effectively.
• The expert cache could be resized without restarting or reloading the model.
• A smaller cache reduced VRAM usage but also reduced performance because more expert traffic crossed PCIe.

FreeToken is most relevant when the complete sparse MoE model cannot fit in VRAM and the workload is long enough for caching, pipelining and overlapping data movement to matter.

---

Reproducibility Checklist

• Same checkpoint and quantization
• Same prompt bytes
• Same output-token limit
• Same sampling configuration
• Same context allocation
• Cold and warm runs labelled separately
• Generated-token counts recorded
• GPU driver and runtime commits recorded
• VRAM and power logs preserved
• At least three repetitions per configuration

---

Important Notes

• FreeToken targets sparse MoE inference; results should not be generalized to dense models.
• A 35GB model is not placed entirely inside 16GB VRAM. Full weights remain backed by system memory while useful experts are cached on the GPU.
• PCIe bandwidth, CPU performance and system-memory bandwidth can influence results.
• Do not expose either local API endpoint publicly without authentication and network controls.
• Commands and flags may change as both projects evolve; verify against their current documentation.

---

Official Resources

Resource	Link
FreeToken	GitHub
FreeToken quick start	Documentation
FreeToken paper	arXiv
llama.cpp	GitHub

---

Follow NetworkCoder

---

Same model. Same workload. Different execution strategy.
Measure on your own hardware before choosing a runtime.