Skip to content

Benchmarking

Jump to bottom
youngharold edited this page Feb 20, 2026 · 3 revisions

Benchmarking

Tightwad includes benchmark scripts to measure speculative decoding acceptance rates and wall-clock speedup across different model pairs, backends, and network configurations. All scripts are in scripts/ and output JSON results to benchmarks/.

Scripts

Script Purpose Draft Source Target Source
scripts/benchmark_proxy.py Multi-config benchmark (local Ollama + cloud APIs) Ollama or llamacpp (LAN) Ollama, llamacpp, or OpenRouter
scripts/benchmark_families.py Cross-family cloud benchmarks (Qwen3, Llama) Local llama-server OpenRouter API
scripts/benchmark_openrouter.py Llama 3.1 8B → 405B via OpenRouter Local llama-server OpenRouter API
scripts/benchmark.sh Quick proxy health + acceptance rate check Running proxy Running proxy

Prerequisites

# Install tightwad with dev dependencies
cd tightwad
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Both Python benchmark scripts use httpx (included in tightwad's dependencies).

benchmark_proxy.py — Multi-Configuration Benchmark

Tests multiple draft→target pairs in one run. Supports Ollama, llama-server, and OpenRouter backends. Uses multi-round speculative decoding with whitespace-normalized text-match verification.

Setup

Start the draft and target servers you want to test. The script has built-in configs — edit the configs list in main() to match your hardware.

Default configs:

  1. Qwen3-8B → Qwen3-32B (local Ollama, same-family baseline)
  2. Qwen3-8B → Qwen3.5-397B (OpenRouter, cloud target)
  3. Qwen3-8B → Llama 3.3 70B (OpenRouter, cross-family control)

Run

# Set OpenRouter key if testing cloud targets
export OPENROUTER_API_KEY="sk-or-..."

# Run all configs
.venv/bin/python scripts/benchmark_proxy.py

# Enable debug output (shows per-round draft/target text)
BENCH_DEBUG=1 .venv/bin/python scripts/benchmark_proxy.py

Output

Prints per-prompt results and a summary table. Saves detailed JSON to benchmark_results.json.

How It Works

For each prompt:

  1. Baseline: Target generates the full response alone (measures raw speed)
  2. Speculative: Multi-round loop:
    • Draft model generates N tokens from the current context
    • Target model generates N tokens from the same context (or continues via "Continue from exactly where you left off" for chat APIs)
    • Whitespace-normalized text-match finds the longest common prefix
    • Target output is accepted and appended to the growing response
    • Repeat until done or max rounds reached

The acceptance rate measures what fraction of draft characters the target agrees with — higher means more tokens come "free" from the fast draft model.

benchmark_openrouter.py — Llama 8B → 405B Cloud Benchmark

Tests same-family speculative decoding over a cloud API. Runs a local Llama 3.1 8B draft against Llama 3.1 405B on OpenRouter.

Setup

# Start local draft server (Llama 3.1 8B on any GPU or CPU)
llama-server -m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
    --port 8081 --host 127.0.0.1 -ngl 999 --ctx-size 4096

# Verify it's running
curl http://127.0.0.1:8081/health

Run

export OPENROUTER_API_KEY="sk-or-..."

.venv/bin/python scripts/benchmark_openrouter.py

# Optional: override target model
TARGET_MODEL="meta-llama/llama-3.1-70b-instruct" .venv/bin/python scripts/benchmark_openrouter.py

# Debug mode
BENCH_DEBUG=1 .venv/bin/python scripts/benchmark_openrouter.py

Environment Variables

Variable Default Description
OPENROUTER_API_KEY (required) OpenRouter API key
DRAFT_URL http://127.0.0.1:8081 Local draft server URL
TARGET_MODEL meta-llama/llama-3.1-405b-instruct OpenRouter model ID
BENCH_DEBUG (unset) Set to any value for per-round debug output

Output

Saves JSON to benchmark_llama_openrouter.json with per-prompt and per-category breakdowns.

12 Prompts Across 5 Categories

Category Count Examples
Reasoning 3 Math, word problems
Code 2 Prime checker, linked list
Factual 3 Geography, science, networking
List 2 Countries by area, programming languages
Creative 2 Haiku, short story

Published Results

Same-Family, Local (Qwen3-8B → Qwen3-32B)

Metric Value
Acceptance Rate 62.8% (text-match) / 73.5% (logprobs)
Wall-Clock Speedup 1.27x
Best Category Reasoning (89%)
Worst Category Creative (39%)

Same-Family, Cloud (Llama 3.1 8B → Llama 3.1 405B, OpenRouter)

Category Acceptance Notes
Code 37.7% Deterministic output matches well
Reasoning 23.8% Step-by-step math overlaps
List 15.4% Formatting varies
Factual 12.6% Phrasing diverges at 405B scale
Creative 3.1% Nearly no overlap
Overall 18.9% Network latency negates speedup

Key finding: Over cloud APIs, the per-round network latency (~3-8s per OpenRouter call) makes speculative decoding slower than baseline, despite 18.9% acceptance. Spec decoding only speeds things up when both models are local or very low-latency.

Same-Family, Cloud (Llama 3.1 8B → Llama 3.1 70B, OpenRouter)

Category Acceptance Notes
Creative 29.7% Haiku matched well
List 24.2% Formatting overlap
Code 9.3% Different implementations
Reasoning 8.7% Step divergence
Factual 7.4% Phrasing varies
Overall 9.9% Smaller target, more divergence than 405B

Same-Family, Cloud (Qwen3 1.7B → Qwen3-235B-A22B, OpenRouter)

Category Acceptance Notes
Factual 14.5% Best category
Reasoning 13.8% Math overlap
Code 6.5% Different styles
Creative 5.2% Low
List 2.0% Formatting diverges
Overall 6.6% 1.7B too small for 235B target

Same-Family, Cloud (Qwen3 1.7B → Qwen3.5-397B-A17B, OpenRouter)

Category Acceptance Notes
Code 16.9% Best category
Factual 10.5% Moderate
List 8.5% Some overlap
Reasoning 4.4% Low
Creative 0.0% No match
Overall 10.8% Slightly better than 235B

All Cloud Results Summary

Draft Target Size Gap Acceptance
Llama 3.1 8B Llama 3.1 405B 50x 18.9%
Qwen3 1.7B Qwen3.5 397B 233x 10.8%
Llama 3.1 8B Llama 3.1 70B 9x 9.9%
Qwen3 1.7B Qwen3 235B 138x 6.6%
Qwen3 8B Llama 3.3 70B cross ~3%

Key findings:

  • Cloud API latency (~3-8s per round) negates wall-clock speedup in all cases
  • Draft model size matters — 8B drafts outperform 1.7B drafts against large targets
  • Larger targets don't necessarily mean lower acceptance (405B beat 70B with the same 8B draft)
  • Cross-family is dead (~3%) regardless of model size

When Speculation Hurts: Target Speed Threshold

Target Speed Speculation Result Example
<3 tok/s 1.8x speedup Qwen3-32B on 4-GPU RPC pool
~3-5 tok/s ~1.5x speedup Dense models on slow pool
~8-10 tok/s Breakeven Estimated threshold
>15 tok/s Slowdown GPT-OSS 120B MoE (19 tok/s baseline)

Rule of thumb: If your target already generates >10 tok/s, speculation adds overhead without benefit. Use direct mode instead.

Cross-Family (Qwen3-8B → Llama 3.3 70B)

Acceptance ~3%

Cross-family drafting is not viable — different training data produces completely different phrasings regardless of tokenizer similarity.

Combined Mode (Speculation Over RPC Pool)

Setup Speed Speedup
Qwen3-32B pool direct 3.0 tok/s —
Qwen3-32B pool + spec 5.4 tok/s 1.8x
Llama 3.3 70B pool direct 2.2 tok/s —
Llama 3.3 70B pool + spec 4.1 tok/s 1.86x

MoE Model Test: GPT-OSS 20B → GPT-OSS 120B (LAN, CPU Offload)

Date: 2026-02-19 | Full report: benchmarks/GPT-OSS-120B-TEST-REPORT.md

Metric Value
Target GPT-OSS 120B (MoE, 5.1B active) on 2x RX 7900 XTX + CPU offload
Draft GPT-OSS 20B (MXFP4) on RTX 4070 Ti Super + RTX 3060
Proxy Tightwad on M4 Mac mini
Acceptance Rate 100.0% (1,659/1,659 tokens)
Baseline (direct) 19.1 tok/s (12.2s avg)
Speculative (proxy) 6.0 tok/s (31.1s avg)
Wall-clock Speedup 0.39x (2.5x slowdown)

Key finding: Despite perfect 100% acceptance, speculation is a net loss because the MoE target is already fast (~20 tok/s). With only 5.1B active parameters, the 120B model generates tokens faster than the draft→verify round-trip overhead. Speculation only helps when the target generates below ~8-10 tok/s.

MoE + RPC caveat: MoE models replicate routing/expert tables per device (~20GB overhead), making RPC distribution to consumer GPUs (8-16GB) infeasible. The original plan to pool 76GB across 4 GPUs failed — fell back to 48GB GPU + 16GB CPU offload.

benchmark_families.py — Multi-Family Cloud Benchmark

Tests multiple model families against OpenRouter targets. Handles per-family chat templates, thinking mode suppression (/no_think for Qwen3), and <think> block stripping.

Run

# List available configs
.venv/bin/python scripts/benchmark_families.py --list

# Run a specific config
OPENROUTER_API_KEY="sk-or-..." .venv/bin/python scripts/benchmark_families.py --config qwen3-235b

# Run all configs (switches draft model between runs)
OPENROUTER_API_KEY="sk-or-..." .venv/bin/python scripts/benchmark_families.py

Available Configs

Config Key Draft (local) Target (OpenRouter)
qwen3-235b Qwen3-1.7B qwen/qwen3-235b-a22b
qwen3.5-397b Qwen3-1.7B qwen/qwen3.5-397b-a17b
llama-70b Llama 3.1 8B meta-llama/llama-3.1-70b-instruct
llama-405b Llama 3.1 8B meta-llama/llama-3.1-405b-instruct

Note: You must start the correct local draft server for each config. Qwen configs need Qwen3-1.7B, Llama configs need Llama 3.1 8B.

Adding New Benchmarks

Using benchmark_families.py (recommended for OpenRouter)

Add a new entry to the CONFIGS dict in scripts/benchmark_families.py:

CONFIGS = {
    "my-config": {
        "name": "MyDraft → MyTarget (OpenRouter)",
        "draft_model": "MyDraft-8B",
        "draft_gguf": "~/models/MyDraft-8B.gguf",
        "draft_url": "http://127.0.0.1:8081",
        "target_model": "provider/model-id",
        "template": my_template_fn,      # chat template for local draft
        "stop_tokens": ["<|eot_id|>"],    # stop tokens for the draft family
        "system_message": "...",          # optional, e.g. "/no_think" for Qwen3
    },
}

Using benchmark_proxy.py (for local/mixed backends)

Add a config tuple to the configs list in main():

configs = [
    (
        "MyDraft → MyTarget (description)",
        {"url": "http://...", "model": "draft-model", "backend": "ollama"},
        {"url": "http://...", "model": "target-model", "backend": "openai", "api_key": OPENROUTER_KEY},
    ),
]

Methodology Notes

  • Temperature: All benchmarks use temperature: 0 (greedy decoding) for reproducibility
  • Whitespace normalization: re.sub(r'\s+', ' ', s).strip() — collapses all whitespace to single spaces before text comparison, preventing formatting differences from inflating rejection rates
  • Text-match verification: Finds the longest common prefix between normalized draft and target text. Character-level, not token-level
  • Multi-round: Each benchmark generates up to 256 tokens across up to 30 rounds (32 draft tokens per round)
  • Same-family requirement: Draft and target must share the same model architecture (e.g., Llama 3.1 8B → Llama 3.1 405B, not Llama 3.2 → Llama 3.3). Cross-family pairs get ~3% acceptance regardless of model size

Clone this wiki locally