Skip to content

Speculative Decoding

Jump to bottom
youngharold edited this page Feb 17, 2026 · 13 revisions

Speculative Decoding

Overview

The speculative decoding proxy accelerates inference by splitting work between two models on separate machines:

  • Draft model (small, fast) — proposes candidate tokens at high speed
  • Target model (large, accurate) — verifies the draft in batch

The output is identical to running the target model alone. The draft model only proposes — the target always has final say.

Why Not Just Use RPC?

The RPC cluster mode ships 100-300 MB of tensor data per inference step over the network. The speculative proxy ships token IDs — bytes. For models that fit on a single machine's VRAM, speculation is dramatically faster because the bottleneck shifts from network bandwidth to model agreement rate.

Approach Network per step Best when
RPC tensor-parallel 100-300 MB Model doesn't fit on one machine
Speculative proxy ~bytes Target fits locally, draft on a separate cheap GPU

How It Works

Each speculation round:

  1. Draft: The small model generates N tokens from the current prompt (default N=8)
  2. Verify: The target model generates N tokens from the same prompt independently
  3. Accept: Find the longest matching prefix between draft and target output
  4. Output: Accept the common prefix + target's continuation from the divergence point
  5. Repeat until max_tokens reached

With same-family models (e.g., Qwen3-8B → Qwen3-72B), the draft agrees with the target 60-80% of the time. This means each round produces ~5-7 verified tokens instead of waiting for the target to generate them one at a time.

Architecture

Client (OpenAI API)
        │
        ▼
┌─────────────────────────┐
│   Hydra Proxy (:8088)   │  Python async server (Starlette/uvicorn)
│   Speculation Loop:     │
│   1. Draft 8 tokens     │──► Draft Server (fast, small model)
│   2. Verify batch       │──► Target Server (slow, large model)
│   3. Accept/reject      │
│   4. Stream to client   │
└─────────────────────────┘

The proxy is a standard OpenAI-compatible API server. Clients don't need to know speculation is happening — they just see faster responses.

Verification Modes

Text-Match Greedy (current, works with any backend)

Both models generate from the same prompt. The proxy finds the longest common prefix and takes the target's output from the divergence point. Works with Ollama, llama-server, or any OpenAI-compatible API.

Logprobs-Based (planned, requires llama-server)

The draft generates N tokens. The target scores all N in a single forward pass using logprobs. At temperature=0, accept if argmax matches. At temperature>0, use rejection sampling: accept with probability min(1, P_target/P_draft). This is more efficient because the target does one forward pass instead of generating N tokens.

The speculation.py module implements both greedy and stochastic verification algorithms. The proxy currently uses text-match; logprobs mode will be used when both servers support it.

Server Backends

The proxy supports two backend types for draft and target:

Backend API Used Strengths
ollama /api/generate with raw: true Easy setup, any Ollama instance, auto model loading
llamacpp /v1/completions with logprobs Full logprobs support, best for logprobs-based verification

Set the backend per server in cluster.yaml:

proxy:
  draft:
    url: http://192.168.86.250:11434
    model_name: qwen3:8b
    backend: ollama
  target:
    url: http://192.168.86.36:11434
    model_name: qwen3:32b
    backend: ollama

Ollama model names use colons (qwen3:8b), not dashes. Using qwen3-8b will cause a "model not found" 404.

API Endpoints

The proxy serves these OpenAI-compatible endpoints on its configured port (default 8088):

Endpoint Method Description
/v1/completions POST Text completion with speculation
/v1/chat/completions POST Chat completion (applies Qwen3 chat template)
/v1/models GET List draft and target models
/v1/hydra/status GET Health checks + acceptance rate stats

All POST endpoints support stream: true for SSE streaming.

Status Response

{
  "draft": {"url": "...", "model": "qwen3:8b", "health": {"alive": true}},
  "target": {"url": "...", "model": "qwen3:32b", "health": {"alive": true}},
  "stats": {
    "total_rounds": 39,
    "total_drafted": 1247,
    "total_accepted": 759,
    "acceptance_rate": 0.609,
    "effective_tokens_per_round": 33.5,
    "uptime_seconds": 120.0
  }
}

Fallback Behavior

If fallback_on_draft_failure: true (default), the proxy forwards requests directly to the target when the draft server is unreachable. This means the proxy never goes down — it just gets slower (target-only speed) until the draft server recovers.

Use Cases

Local Multi-GPU

Draft on a consumer GPU ($200), verify on a larger GPU or multi-GPU rig. Example: Qwen3-8B on an RTX 2070 drafting for Qwen3-72B on 2x 7900 XTX.

Cloud API Cost Reduction

Draft locally, verify via a cloud API (OpenRouter, Together, any OpenAI-compatible endpoint). The draft model handles the cheap speculative work. The cloud API only confirms. With 70% acceptance, you make ~5-6x fewer API calls for the same output quality.

Edge + Datacenter

Run the draft model on edge hardware for low latency. Route verification to a datacenter for accuracy. The user gets fast responses with datacenter-grade quality.

Tested Configurations

Draft Target Acceptance Rate Notes
Qwen3-8B (2070, Ollama) Qwen3-32B (4070, Ollama) 60.9% Same family, best results
Qwen3-8B (2070, Ollama) GLM-4.7-Flash (4070, llama-server) 69.6% Cross-family, still works

PID Management

The proxy PID is stored at ~/.hydra/proxy.pid (separate from the RPC coordinator's ~/.hydra/coordinator.pid). Both can run simultaneously.

Module Structure

hydra/
├── speculation.py   # Pure verification logic (no I/O)
│   ├── DraftToken, TargetLogprob, VerificationResult  (dataclasses)
│   ├── verify_greedy()        # Accept iff argmax matches
│   ├── verify_stochastic()    # Rejection sampling: min(1, P_target/P_draft)
│   └── verify_draft_tokens()  # Dispatcher (temperature=0 → greedy, else stochastic)
│
└── proxy.py         # Async server
    ├── SpeculativeProxy       # Draft/verify/accept loop
    ├── ProxyStats             # Acceptance rate tracking
    ├── apply_chat_template()  # Qwen3 chat format
    └── create_app()           # Starlette app factory

Clone this wiki locally