Skip to content

Speculative Decoding

Jump to bottom
youngharold edited this page Feb 17, 2026 · 13 revisions

Speculative Decoding

Overview

The speculative decoding proxy accelerates inference by splitting work between two models on separate machines:

  • Draft model (small, fast) — proposes candidate tokens at high speed
  • Target model (large, accurate) — verifies the draft in batch

The output is identical to running the target model alone. The draft model only proposes — the target always has final say.

Why Not Just Use RPC?

The RPC cluster mode ships 100-300 MB of tensor data per inference step over the network. The speculative proxy ships token IDs — bytes. For models that fit on a single machine's VRAM, speculation is dramatically faster because the bottleneck shifts from network bandwidth to model agreement rate.

Approach Network per step Best when
RPC tensor-parallel 100-300 MB Model doesn't fit on one machine
Speculative proxy ~bytes Target fits locally, draft on a separate cheap GPU

How It Works

Each speculation round:

  1. Draft: The small model generates N tokens from the current prompt (default N=32)
  2. Verify: The draft text is appended to the prompt and sent to the target, which processes it as prompt (parallel evaluation) and generates 1 bonus token
  3. Accept: Tokenization is compared between draft and target — if they agree, all draft tokens are accepted plus the bonus token
  4. Fallback: If tokenization diverges, falls back to N+1 autoregressive generation with logprobs comparison for per-position verification
  5. Repeat until max_tokens reached

With same-family models (e.g., Qwen3-8B → Qwen3-32B), the draft agrees with the target 73%+ of the time. With max_draft_tokens: 32, each round produces ~31 verified tokens, achieving 1.27x wall-clock speedup over baseline.

Architecture

Client (OpenAI API)
        │
        ▼
┌─────────────────────────┐
│   Tightwad Proxy (:8088)│  Python async server (Starlette/uvicorn)
│   Speculation Loop:     │
│   1. Draft 8 tokens     │──► Draft Server (fast, small model)
│   2. Verify batch       │──► Target Server (slow, large model)
│   3. Accept/reject      │
│   4. Stream to client   │
└─────────────────────────┘

The proxy is a standard OpenAI-compatible API server. Clients don't need to know speculation is happening — they just see faster responses.

Verification Modes

Prompt-Append (default, requires llamacpp backend on both)

The draft generates N tokens. The draft text is appended to the prompt and sent to the target with max_tokens: 1. The target processes draft tokens as prompt (parallel, ~40 tok/s) instead of generating them autoregressively (~19 tok/s). The /tokenize endpoint on both servers compares tokenization — same-family models share tokenizers, so matching tokenization confirms the draft is valid.

Falls back to N+1 autoregressive generation with per-position logprobs comparison if tokenization diverges.

Text-Match (fallback, works with any backend including Ollama)

Both models generate from the same prompt independently. The proxy finds the longest common prefix and takes the target's output from the divergence point. Works with Ollama, llama-server, or any OpenAI-compatible API. Used automatically when either backend is set to ollama.

Logprobs Comparison (fallback for tokenization mismatch)

The draft generates N tokens with per-token IDs and logprobs. The target generates N+1 tokens with logprobs. At temperature=0, accept if token IDs match (greedy). At temperature>0, use rejection sampling: min(1, P_target/P_draft).

The speculation.py module implements both greedy and stochastic verification algorithms.

Server Backends

The proxy supports two backend types for draft and target:

Backend API Used Strengths
ollama /api/generate with raw: true Easy setup, any Ollama instance, auto model loading
llamacpp /v1/completions with logprobs Full logprobs support, best for logprobs-based verification

Set the backend per server in cluster.yaml:

proxy:
  max_draft_tokens: 32            # Sweet spot for cross-machine setups
  draft:
    url: http://192.168.86.250:8081
    model_name: qwen3-8b
    backend: llamacpp              # Best performance with prompt-append verification
  target:
    url: http://192.168.86.36:8080
    model_name: qwen3-32b
    backend: llamacpp

Use llamacpp backend for both when running llama-server — enables prompt-append verification (1.27x speedup). Use ollama backend as fallback (text-match only, no speedup).

Ollama model names use colons (qwen3:8b), not dashes. Using qwen3-8b will cause a "model not found" 404.

API Endpoints

The proxy serves these OpenAI-compatible endpoints on its configured port (default 8088):

Endpoint Method Description
/v1/completions POST Text completion with speculation
/v1/chat/completions POST Chat completion (applies Qwen3 chat template)
/v1/models GET List draft and target models
/v1/tightwad/status GET Health checks + acceptance rate stats

All POST endpoints support stream: true for SSE streaming.

Status Response

Single-drafter mode:

{
  "draft": {"url": "...", "model": "qwen3-8b", "health": {"alive": true}},
  "target": {"url": "...", "model": "qwen3-32b", "health": {"alive": true}},
  "stats": {
    "total_rounds": 39,
    "total_drafted": 1247,
    "total_accepted": 759,
    "acceptance_rate": 0.609,
    "effective_tokens_per_round": 33.5,
    "uptime_seconds": 120.0
  }
}

Multi-drafter mode:

{
  "target": {"url": "...", "model": "qwen3-32b", "health": {"alive": true}},
  "drafters": [
    {"url": "http://192.168.86.250:8081", "model": "qwen3-8b", "backend": "llamacpp", "health": {"alive": true}, "wins": 6},
    {"url": "http://192.168.86.28:8081", "model": "qwen3-1.7b", "backend": "llamacpp", "health": {"alive": true}, "wins": 4},
    {"url": "http://192.168.86.86:11434", "model": "qwen3:1.7b", "backend": "ollama", "health": {"alive": true}, "wins": 0}
  ],
  "stats": { ... }
}

Fallback Behavior

If fallback_on_draft_failure: true (default), the proxy forwards requests directly to the target when the draft server is unreachable. This means the proxy never goes down — it just gets slower (target-only speed) until the draft server recovers.

Use Cases

Local Multi-GPU

Draft on a consumer GPU ($200), verify on a larger GPU or multi-GPU rig. Example: Qwen3-8B on an RTX 2070 drafting for Qwen3-72B on 2x 7900 XTX.

Cloud API Cost Reduction

Draft locally, verify via a cloud API (OpenRouter, Together, any OpenAI-compatible endpoint). The draft model handles the cheap speculative work. The cloud API only confirms. With 70% acceptance, you make ~5-6x fewer API calls for the same output quality.

CPU Draft, GPU Verify

Run a tiny model (0.6B-1.7B) on CPU/RAM with no GPU needed, verify on GPU. Turns every idle CPU into usable inference compute. Tested: Qwen3-1.7B at 15-33 tok/s on consumer CPUs.

Multi-Drafter Parallelism

Multiple machines each run a draft model in parallel. The proxy picks the best candidate and the GPU target verifies it. All drafters race simultaneously via asyncio.gather — the fastest drafter with the best output wins.

XPS 2070 GPU  ──► draft 32 tokens (llamacpp) ──┐
M2 Mac CPU    ──► draft 32 tokens (llamacpp) ──┼──► Proxy picks best ──► GPU Target verifies
Unraid CPU    ──► draft 32 tokens (ollama)   ──┘    (single forward pass)

Winner selection priority:

  1. Has per-token logprobs (llamacpp with multiple tokens) — enables batch verification
  2. Most tokens generated
  3. Longest total text
  4. Highest mean logprob (tiebreaker)

Ollama drafters are deprioritized because they return one text blob without per-token IDs, forcing text-match fallback.

Tested results (3 drafters → Qwen3-32B on 4070):

  • 100% acceptance rate across 10 rounds (320/320 tokens)
  • 33 tokens/round (32 + bonus)
  • XPS qwen3-8b won 60% of rounds, M2 qwen3-1.7b won 40%
  • Ollama drafter: 0 wins (correctly deprioritized)

Config:

proxy:
  drafters:
    - url: http://192.168.86.250:8081
      model_name: qwen3-8b
      backend: llamacpp
    - url: http://192.168.86.28:8081
      model_name: qwen3-1.7b
      backend: llamacpp
    - url: http://192.168.86.86:11434
      model_name: qwen3:1.7b
      backend: ollama
  target:
    url: http://192.168.86.36:8080
    model_name: qwen3-32b
    backend: llamacpp

When drafters: is present, the single draft: field is ignored. When absent, single-drafter mode works as before.

Edge + Datacenter

Run the draft model on edge hardware for low latency. Route verification to a datacenter for accuracy. The user gets fast responses with datacenter-grade quality.

Benchmark Results

Wall-Clock Speedup (Qwen3-8B → Qwen3-32B, cross-machine llama-server)

Draft on RTX 2070 (8GB, WiFi), target on RTX 4070 Ti Super + RTX 3060 (28GB, LAN). Both via llama-server with prompt-append verification.

Prompt Baseline Speculative Speedup
Capital of France 1.17s 0.90s 1.30x
Thermodynamics 12.73s 9.09s 1.40x
Prime checker 12.76s 10.15s 1.28x
Average speed 13.24s 10.95s 1.21x
TCP vs UDP 5.58s 4.88s 1.14x
Total 45.43s 35.96s 1.27x

Tuning max_draft_tokens

Setting Rounds Tok/Round Speedup
8 96 8.8 0.63x (slower)
32 50 31.7 1.27x
64 16 56.5 1.21x

The sweet spot is 32 — fewer rounds reduce HTTP overhead, but too many draft tokens adds draft latency.

Tested Configurations

Draft Target Acceptance Speedup Notes
Qwen3-8B (2070, llamacpp) Qwen3-32B (4070, llamacpp) 73.5% 1.27x Best config, prompt-append
Qwen3-1.7B (M4 CPU, llamacpp) Qwen3-32B (4070, llamacpp) 68% 0.80x CPU draft, needs higher draft count
Qwen3-1.7B (Unraid CPU, Ollama) Qwen3-32B (4070, llamacpp) 68% 0.14x Text-match, no batch verification
Qwen3-8B (2070, Ollama) Qwen3-32B (4070, Ollama) 60.9% — Text-match only
Multi-drafter (3 machines) Qwen3-32B (4070, llamacpp) 100% — 3 drafters racing in parallel
Cross-family (Qwen→Llama) — ~3% — Different tokenizers, not viable

Key findings:

  • Same-family model pairs are critical (73% vs 3% cross-family)
  • max_draft_tokens: 32 is the sweet spot for cross-machine setups
  • The 1.27x speedup was achieved with the 2070 on WiFi — LAN drafters should be even faster
  • Multi-drafter parallelism achieves 100% acceptance by racing the best candidate from N drafters

PID Management

The proxy PID is stored at ~/.tightwad/proxy.pid (separate from the RPC coordinator's ~/.tightwad/coordinator.pid). Both can run simultaneously.

Module Structure

tightwad/
├── speculation.py   # Pure verification logic (no I/O)
│   ├── DraftToken, TargetLogprob, VerificationResult  (dataclasses)
│   ├── verify_greedy()        # Accept iff argmax matches
│   ├── verify_stochastic()    # Rejection sampling: min(1, P_target/P_draft)
│   └── verify_draft_tokens()  # Dispatcher (temperature=0 → greedy, else stochastic)
│
└── proxy.py         # Async server
    ├── SpeculativeProxy       # Draft/verify/accept loop
    │   ├── draft_tokens()           # Single-drafter drafting
    │   ├── draft_tokens_parallel()  # Multi-drafter racing (asyncio.gather)
    │   └── _draft_from_endpoint()   # Per-endpoint draft (llamacpp or ollama)
    ├── ProxyStats             # Acceptance rate + drafter win tracking
    ├── apply_chat_template()  # Qwen3 chat format
    └── create_app()           # Starlette app factory

Clone this wiki locally