Skip to content

Speculative Decoding

Jump to bottom
youngharold edited this page Mar 28, 2026 · 13 revisions

Speculative Decoding

Overview

The speculative decoding proxy accelerates inference by splitting work between two models on separate machines:

  • Draft model (small, fast) — proposes candidate tokens at high speed
  • Target model (large, accurate) — verifies the draft in batch

The output is identical to running the target model alone. The draft model only proposes — the target always has final say.

Why Not Just Use RPC?

The RPC cluster mode ships 100-300 MB of tensor data per inference step over the network. The speculative proxy ships token IDs — bytes. For models that fit on a single machine's VRAM, speculation is dramatically faster because the bottleneck shifts from network bandwidth to model agreement rate.

Approach Network per step Best when
RPC tensor-parallel 100-300 MB Model doesn't fit on one machine
Speculative proxy ~bytes Target fits locally, draft on a separate cheap GPU

How It Works

Each speculation round:

  1. Draft: The small model generates N tokens from the current prompt (default N=32)
  2. Verify: The draft text is appended to the prompt and sent to the target, which processes it as prompt (parallel evaluation) and generates 1 bonus token
  3. Accept: Tokenization is compared between draft and target — if they agree, all draft tokens are accepted plus the bonus token
  4. Fallback: If tokenization diverges, falls back to N+1 autoregressive generation with logprobs comparison for per-position verification
  5. Repeat until max_tokens reached

With same-family models (e.g., Qwen3-8B → Qwen3-32B), the draft agrees with the target 73%+ of the time. With max_draft_tokens: 32, each round produces ~31 verified tokens, achieving 1.27x wall-clock speedup over baseline.

Architecture

Client (OpenAI API)
        │
        ▼
┌─────────────────────────┐
│   Tightwad Proxy (:8088)│  Python async server (Starlette/uvicorn)
│   Speculation Loop:     │
│   1. Draft 8 tokens     │──► Draft Server (fast, small model)
│   2. Verify batch       │──► Target Server (slow, large model)
│   3. Accept/reject      │
│   4. Stream to client   │
└─────────────────────────┘

The proxy is a standard OpenAI-compatible API server. Clients don't need to know speculation is happening — they just see faster responses.

Verification Modes

Prompt-Append (default, requires llamacpp backend on both)

The draft generates N tokens. The draft text is appended to the prompt and sent to the target with max_tokens: 1. The target processes draft tokens as prompt (parallel, ~40 tok/s) instead of generating them autoregressively (~19 tok/s). The /tokenize endpoint on both servers compares tokenization — same-family models share tokenizers, so matching tokenization confirms the draft is valid.

Falls back to N+1 autoregressive generation with per-position logprobs comparison if tokenization diverges.

Text-Match (fallback, works with any backend including Ollama)

Both models generate from the same prompt independently. The proxy finds the longest common prefix and takes the target's output from the divergence point. Works with Ollama, llama-server, or any OpenAI-compatible API. Used automatically when either backend is set to ollama.

Logprobs Comparison (fallback for tokenization mismatch)

The draft generates N tokens with per-token IDs and logprobs. The target generates N+1 tokens with logprobs. At temperature=0, accept if token IDs match (greedy). At temperature>0, use rejection sampling: min(1, P_target/P_draft).

The speculation.py module implements both greedy and stochastic verification algorithms.

Consensus Verification (multi-drafter)

When multiple drafters are configured, consensus mode compares all drafter outputs before contacting the target. Positions where drafters agree can be accepted directly, skipping the expensive target call entirely.

Three modes (set via proxy.consensus_mode in cluster.yaml):

  • strict — require unanimous agreement from all drafters
  • majority — >50% of drafters must agree (majority token accepted)
  • any_disagree — accept unanimous positions, verify with target at first disagreement
proxy:
  consensus_mode: strict    # "off" (default), "strict", "majority", "any_disagree"
  drafters:
    - url: http://machine1:8081
      model_name: qwen3-8b
      backend: llamacpp
    - url: http://machine2:8081
      model_name: qwen3-8b
      backend: llamacpp

Best for scenarios where the target is already fast (e.g., MoE models) and the goal is reducing target calls rather than speeding up generation.

Server Backends

The proxy supports two backend types for draft and target:

Backend API Used Strengths
ollama /api/generate with raw: true Easy setup, any Ollama instance, auto model loading
llamacpp /v1/completions with logprobs Full logprobs support, best for logprobs-based verification

Set the backend per server in cluster.yaml:

proxy:
  max_draft_tokens: 32            # Sweet spot for cross-machine setups
  draft:
    url: http://192.168.86.250:8081
    model_name: qwen3-8b
    backend: llamacpp              # Best performance with prompt-append verification
  target:
    url: http://192.168.86.36:8080
    model_name: qwen3-32b
    backend: llamacpp

Use llamacpp backend for both when running llama-server — enables prompt-append verification (1.27x speedup). Use ollama backend as fallback (text-match only, no speedup).

Ollama model names use colons (qwen3:8b), not dashes. Using qwen3-8b will cause a "model not found" 404.

API Endpoints

The proxy serves these OpenAI-compatible endpoints on its configured port (default 8088):

Endpoint Method Description
/v1/completions POST Text completion with speculation
/v1/chat/completions POST Chat completion (applies Qwen3 chat template)
/v1/models GET List draft and target models
/v1/tightwad/status GET Health checks + acceptance rate stats

All POST endpoints support stream: true for SSE streaming.

Status Response

Single-drafter mode:

{
  "draft": {"url": "...", "model": "qwen3-8b", "health": {"alive": true}},
  "target": {"url": "...", "model": "qwen3-32b", "health": {"alive": true}},
  "stats": {
    "total_rounds": 39,
    "total_drafted": 1247,
    "total_accepted": 759,
    "acceptance_rate": 0.609,
    "effective_tokens_per_round": 33.5,
    "uptime_seconds": 120.0
  }
}

Multi-drafter mode:

{
  "target": {"url": "...", "model": "qwen3-32b", "health": {"alive": true}},
  "drafters": [
    {"url": "http://192.168.86.250:8081", "model": "qwen3-8b", "backend": "llamacpp", "health": {"alive": true}, "wins": 6},
    {"url": "http://192.168.86.28:8081", "model": "qwen3-1.7b", "backend": "llamacpp", "health": {"alive": true}, "wins": 4},
    {"url": "http://192.168.86.86:11434", "model": "qwen3:1.7b", "backend": "ollama", "health": {"alive": true}, "wins": 0}
  ],
  "stats": { ... }
}

Fallback Behavior

If fallback_on_draft_failure: true (default), the proxy forwards requests directly to the target when the draft server is unreachable. This means the proxy never goes down — it just gets slower (target-only speed) until the draft server recovers.

Use Cases

Local Multi-GPU

Draft on a consumer GPU ($200), verify on a larger GPU or multi-GPU rig. Example: Qwen3-8B on an RTX 2070 drafting for Qwen3-32B on 2x 7900 XTX.

Cloud API Cost Reduction

Draft locally, verify via a cloud API (OpenRouter, Together, any OpenAI-compatible endpoint). The draft model handles the cheap speculative work. The cloud API only confirms. With 70% acceptance, you make ~5-6x fewer API calls for the same output quality.

CPU Draft, GPU Verify

Run a tiny model (0.6B-1.7B) on CPU/RAM with no GPU needed, verify on GPU. Turns every idle CPU into usable inference compute. Tested: Qwen3-1.7B at 15-33 tok/s on consumer CPUs.

Multi-Drafter Parallelism

Multiple machines each run a draft model in parallel. The proxy picks the best candidate and the GPU target verifies it. All drafters race simultaneously via asyncio.gather — the fastest drafter with the best output wins.

XPS 2070 GPU  ──► draft 32 tokens (llamacpp) ──┐
M2 Mac CPU    ──► draft 32 tokens (llamacpp) ──┼──► Proxy picks best ──► GPU Target verifies
Unraid CPU    ──► draft 32 tokens (ollama)   ──┘    (single forward pass)

Winner selection priority:

  1. Has per-token logprobs (llamacpp with multiple tokens) — enables batch verification
  2. Most tokens generated
  3. Longest total text
  4. Highest mean logprob (tiebreaker)

Ollama drafters are deprioritized because they return one text blob without per-token IDs, forcing text-match fallback.

Tested results (3 drafters → Qwen3-32B on 4070):

  • 100% acceptance rate across 10 rounds (320/320 tokens)
  • 33 tokens/round (32 + bonus)
  • XPS qwen3-8b won 60% of rounds, M2 qwen3-1.7b won 40%
  • Ollama drafter: 0 wins (correctly deprioritized)

Config:

proxy:
  drafters:
    - url: http://192.168.86.250:8081
      model_name: qwen3-8b
      backend: llamacpp
    - url: http://192.168.86.28:8081
      model_name: qwen3-1.7b
      backend: llamacpp
    - url: http://192.168.86.86:11434
      model_name: qwen3:1.7b
      backend: ollama
  target:
    url: http://192.168.86.36:8080
    model_name: qwen3-32b
    backend: llamacpp

When drafters: is present, the single draft: field is ignored. When absent, single-drafter mode works as before.

Combined Mode — Speculation Over RPC Pool

The killer feature. When a model is too large for any single machine, pool GPUs via RPC and use speculative decoding to overcome RPC's per-token latency. The draft model runs on any junk hardware (CPU, 2GB GPU) and the pooled target verifies 32 tokens per batch instead of generating one at a time.

ANY junk hardware (P400 2GB, GTX 770, laptop CPU)
    │ runs Qwen3-1.7B, drafts 32 tokens (~30 tok/s)
    │ sends token IDs (bytes, not megabytes)
    ▼
Tightwad Proxy (:8088)
    │ sends draft to pool for BATCH verification
    ▼
RPC GPU Pool (4070+3060+2070+M2, running 32B)
    │ verifies 32 tokens in ONE forward pass
    │ 1 RPC round-trip for 32 tokens instead of 32 round-trips
    ▼
5+ tok/s instead of 3 tok/s

Why this works: RPC tensor-parallelism ships 100-300 MB per inference step. Autoregressive generation means paying that cost per token — 3 tok/s. With speculation, the draft model generates 32 tokens locally (no network), then the pool verifies all 32 in one batch forward pass — one RPC round-trip for 32 tokens instead of 32 round-trips for 32 tokens.

Requirements:

  • Draft model must be same family as target (e.g., Qwen3-1.7B → Qwen3-32B). Tightwad auto-detects families at proxy startup and warns on mismatch
  • Draft backend must be llamacpp (not Ollama) for prompt-append verification. Ollama falls back to text-match which provides NO batch speedup
  • Draft model can run on literally anything — 2GB GPU, laptop CPU, Raspberry Pi
  • Target is the RPC pool coordinator URL

Config:

proxy:
  max_draft_tokens: 32
  draft:
    url: http://127.0.0.1:8081     # Any junk hardware running llama-server
    model_name: qwen3-1.7b
    backend: llamacpp               # CRITICAL: must be llamacpp, not ollama
  target:
    url: http://192.168.86.36:8090  # RPC pool coordinator
    model_name: qwen3-32b
    backend: llamacpp

Tested results (M4 CPU draft → 4-GPU RPC pool over WiFi):

Mode Speed Notes
RPC pool direct (autoregressive) 3.0 tok/s Each token = full RPC round-trip
RPC pool + speculation 5.4 tok/s 32 tokens verified per batch, 100% acceptance
Speedup 1.8x
Desktop local only (no RPC) 17.0 tok/s For comparison

Edge + Datacenter

Run the draft model on edge hardware for low latency. Route verification to a datacenter for accuracy. The user gets fast responses with datacenter-grade quality.

Benchmark Results

Wall-Clock Speedup (Qwen3-8B → Qwen3-32B, cross-machine llama-server)

Draft on RTX 2070 (8GB, WiFi), target on RTX 4070 Ti Super + RTX 3060 (28GB, LAN). Both via llama-server with prompt-append verification.

Prompt Baseline Speculative Speedup
Capital of France 1.17s 0.90s 1.30x
Thermodynamics 12.73s 9.09s 1.40x
Prime checker 12.76s 10.15s 1.28x
Average speed 13.24s 10.95s 1.21x
TCP vs UDP 5.58s 4.88s 1.14x
Total 45.43s 35.96s 1.27x

Tuning max_draft_tokens

Setting Rounds Tok/Round Speedup
8 96 8.8 0.63x (slower)
32 50 31.7 1.27x
64 16 56.5 1.21x

The sweet spot is 32 — fewer rounds reduce HTTP overhead, but too many draft tokens adds draft latency.

Tested Configurations

Draft Target Acceptance Speedup Notes
Qwen3-8B (2070, llamacpp) Qwen3-32B (4070, llamacpp) 73.5% 1.27x Best config, prompt-append
Qwen3-1.7B (M4 CPU, llamacpp) Qwen3-32B (4070, llamacpp) 68% 0.80x CPU draft, needs higher draft count
Qwen3-1.7B (Unraid CPU, Ollama) Qwen3-32B (4070, llamacpp) 68% 0.14x Text-match, no batch verification
Qwen3-8B (2070, Ollama) Qwen3-32B (4070, Ollama) 60.9% — Text-match only
Multi-drafter (3 machines) Qwen3-32B (4070, llamacpp) 100% — 3 drafters racing in parallel
Qwen3-1.7B (M4 CPU) Qwen3-32B (4-GPU RPC pool) 100% 1.8x Combined mode: speculation over pool
Llama 3.1 8B (local llamacpp) Llama 3.1 405B (OpenRouter) 18.9% — Same-family over cloud API
Qwen3 1.7B (local llamacpp) Qwen3.5 397B (OpenRouter) 10.8% — 1.7B too small for 397B target
Llama 3.1 8B (local llamacpp) Llama 3.1 70B (OpenRouter) 9.9% — Surprisingly worse than 405B
Qwen3 1.7B (local llamacpp) Qwen3 235B (OpenRouter) 6.6% — 1.7B too small for 235B target
Cross-family (Qwen→Llama) — ~3% — Different tokenizers, not viable

Key findings:

  • Same-family model pairs are critical (73% vs 3% cross-family)
  • max_draft_tokens: 32 is the sweet spot for cross-machine setups
  • The 1.27x speedup was achieved with the 2070 on WiFi — LAN drafters should be even faster
  • Multi-drafter parallelism achieves 100% acceptance by racing the best candidate from N drafters
  • Combined mode (speculation over RPC pool) achieves 1.8x speedup over pool-only — making models that don't fit on one machine actually usable
  • llamacpp backend is CRITICAL for prompt-append verification. Ollama falls back to text-match which provides NO batch speedup (1.4 tok/s vs 5.4 tok/s)
  • Draft model size matters for cloud targets — 1.7B drafts get 6-11% against 200B+ targets, while 8B drafts get 10-19% against similar-sized targets
  • Cloud API latency negates wall-clock speedup — even with 19% acceptance, per-round network latency makes spec slower than baseline over the internet

Chat Inline Stats

tightwad chat displays per-response speculation stats inline after each AI response:

You: What is the capital of France?
AI: The capital of France is Paris.
  ↳ 3 rounds, 96 drafted, 71 accepted (74.0%), 24.3 tok/round, 18.2 tok/s

Stats are delta-based (per-response, not cumulative) and fetched from the /v1/tightwad/status endpoint after each response. In --direct mode, stats are not shown since there's no proxy to query.

PID Management

The proxy PID is stored at ~/.tightwad/proxy.pid (separate from the RPC coordinator's ~/.tightwad/coordinator.pid). Both can run simultaneously.

Module Structure

tightwad/
├── speculation.py   # Pure verification logic (no I/O)
│   ├── DraftToken, TargetLogprob, VerificationResult  (dataclasses)
│   ├── verify_greedy()        # Accept iff argmax matches
│   ├── verify_stochastic()    # Rejection sampling: min(1, P_target/P_draft)
│   └── verify_draft_tokens()  # Dispatcher (temperature=0 → greedy, else stochastic)
│
└── proxy.py         # Async server
    ├── SpeculativeProxy       # Draft/verify/accept loop
    │   ├── draft_tokens()           # Single-drafter drafting
    │   ├── draft_tokens_parallel()  # Multi-drafter racing (asyncio.gather)
    │   └── _draft_from_endpoint()   # Per-endpoint draft (llamacpp or ollama)
    ├── ProxyStats             # Acceptance rate + drafter win tracking
    ├── apply_chat_template()  # Qwen3 chat format
    └── create_app()           # Starlette app factory

Clone this wiki locally