Skip to content

Benchmarking

Jump to bottom
youngharold edited this page Feb 18, 2026 · 3 revisions

Benchmarking

Tightwad includes benchmark scripts to measure speculative decoding acceptance rates and wall-clock speedup across different model pairs, backends, and network configurations. All scripts are in scripts/ and output JSON results to benchmarks/.

Scripts

Script Purpose Draft Source Target Source
scripts/benchmark_proxy.py Multi-config benchmark (local Ollama + cloud APIs) Ollama or llamacpp (LAN) Ollama, llamacpp, or OpenRouter
scripts/benchmark_openrouter.py Llama 3.1 8B → 405B via OpenRouter Local llama-server OpenRouter API
scripts/benchmark.sh Quick proxy health + acceptance rate check Running proxy Running proxy

Prerequisites

# Install tightwad with dev dependencies
cd tightwad
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Both Python benchmark scripts use httpx (included in tightwad's dependencies).

benchmark_proxy.py — Multi-Configuration Benchmark

Tests multiple draft→target pairs in one run. Supports Ollama, llama-server, and OpenRouter backends. Uses multi-round speculative decoding with whitespace-normalized text-match verification.

Setup

Start the draft and target servers you want to test. The script has built-in configs — edit the configs list in main() to match your hardware.

Default configs:

  1. Qwen3-8B → Qwen3-32B (local Ollama, same-family baseline)
  2. Qwen3-8B → Qwen3.5-397B (OpenRouter, cloud target)
  3. Qwen3-8B → Llama 3.3 70B (OpenRouter, cross-family control)

Run

# Set OpenRouter key if testing cloud targets
export OPENROUTER_API_KEY="sk-or-..."

# Run all configs
.venv/bin/python scripts/benchmark_proxy.py

# Enable debug output (shows per-round draft/target text)
BENCH_DEBUG=1 .venv/bin/python scripts/benchmark_proxy.py

Output

Prints per-prompt results and a summary table. Saves detailed JSON to benchmark_results.json.

How It Works

For each prompt:

  1. Baseline: Target generates the full response alone (measures raw speed)
  2. Speculative: Multi-round loop:
    • Draft model generates N tokens from the current context
    • Target model generates N tokens from the same context (or continues via "Continue from exactly where you left off" for chat APIs)
    • Whitespace-normalized text-match finds the longest common prefix
    • Target output is accepted and appended to the growing response
    • Repeat until done or max rounds reached

The acceptance rate measures what fraction of draft characters the target agrees with — higher means more tokens come "free" from the fast draft model.

benchmark_openrouter.py — Llama 8B → 405B Cloud Benchmark

Tests same-family speculative decoding over a cloud API. Runs a local Llama 3.1 8B draft against Llama 3.1 405B on OpenRouter.

Setup

# Start local draft server (Llama 3.1 8B on any GPU or CPU)
llama-server -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
    --port 8081 --host 127.0.0.1 -ngl 999 --ctx-size 4096

# Verify it's running
curl http://127.0.0.1:8081/health

Run

export OPENROUTER_API_KEY="sk-or-..."

.venv/bin/python scripts/benchmark_openrouter.py

# Optional: override target model
TARGET_MODEL="meta-llama/llama-3.1-70b-instruct" .venv/bin/python scripts/benchmark_openrouter.py

# Debug mode
BENCH_DEBUG=1 .venv/bin/python scripts/benchmark_openrouter.py

Environment Variables

Variable Default Description
OPENROUTER_API_KEY (required) OpenRouter API key
DRAFT_URL http://127.0.0.1:8081 Local draft server URL
TARGET_MODEL meta-llama/llama-3.1-405b-instruct OpenRouter model ID
BENCH_DEBUG (unset) Set to any value for per-round debug output

Output

Saves JSON to benchmark_llama_openrouter.json with per-prompt and per-category breakdowns.

12 Prompts Across 5 Categories

Category Count Examples
Reasoning 3 Math, word problems
Code 2 Prime checker, linked list
Factual 3 Geography, science, networking
List 2 Countries by area, programming languages
Creative 2 Haiku, short story

Published Results

Same-Family, Local (Qwen3-8B → Qwen3-32B)

Metric Value
Acceptance Rate 62.8% (text-match) / 73.5% (logprobs)
Wall-Clock Speedup 1.27x
Best Category Reasoning (89%)
Worst Category Creative (39%)

Same-Family, Cloud (Llama 3.1 8B → Llama 3.1 405B, OpenRouter)

Category Acceptance Notes
Code 37.7% Deterministic output matches well
Reasoning 23.8% Step-by-step math overlaps
List 15.4% Formatting varies
Factual 12.6% Phrasing diverges at 405B scale
Creative 3.1% Nearly no overlap
Overall 18.9% Network latency negates speedup

Key finding: Over cloud APIs, the per-round network latency (~3-8s per OpenRouter call) makes speculative decoding slower than baseline, despite 18.9% acceptance. Spec decoding only speeds things up when both models are local or very low-latency.

Cross-Family (Qwen3-8B → Llama 3.3 70B)

Acceptance ~3%

Cross-family drafting is not viable — different training data produces completely different phrasings regardless of tokenizer similarity.

Combined Mode (Speculation Over RPC Pool)

Setup Speed Speedup
Qwen3-32B pool direct 3.0 tok/s —
Qwen3-32B pool + spec 5.4 tok/s 1.8x
Llama 3.3 70B pool direct 2.2 tok/s —
Llama 3.3 70B pool + spec 4.1 tok/s 1.86x

Adding New Benchmarks

To add a new model pair:

  1. Edit scripts/benchmark_proxy.py and add a config tuple to the configs list in main():
configs = [
    # ... existing configs ...
    (
        "MyDraft → MyTarget (description)",
        {"url": "http://...", "model": "draft-model", "backend": "ollama"},
        {"url": "http://...", "model": "target-model", "backend": "openai", "api_key": OPENROUTER_KEY},
    ),
]
  1. Run the benchmark and save results:
.venv/bin/python scripts/benchmark_proxy.py
cp benchmark_results.json benchmarks/benchmark_<description>.json

Methodology Notes

  • Temperature: All benchmarks use temperature: 0 (greedy decoding) for reproducibility
  • Whitespace normalization: re.sub(r'\s+', ' ', s).strip() — collapses all whitespace to single spaces before text comparison, preventing formatting differences from inflating rejection rates
  • Text-match verification: Finds the longest common prefix between normalized draft and target text. Character-level, not token-level
  • Multi-round: Each benchmark generates up to 256 tokens across up to 30 rounds (32 draft tokens per round)
  • Same-family requirement: Draft and target must share the same model architecture (e.g., Llama 3.1 8B → Llama 3.1 405B, not Llama 3.2 → Llama 3.3). Cross-family pairs get ~3% acceptance regardless of model size

Clone this wiki locally