Distributed speculative decoding: GPU draft model over network

[7avgerenko]

Problem

Speculative decoding with --spec-draft-model requires the draft model file to be local. This prevents using a GPU-accelerated draft model running on a separate machine (e.g., a laptop GPU drafting for a server CPU target).

Proposed solution

A lightweight HTTP proxy (Layer 2 approach) that orchestrates distributed spec decoding between two existing llama.cpp instances:

Client → proxy:8080 → laptop:8081 (draft, GPU, generates K tokens)
                     → server:8080  (target, CPU, validates in 1 pass)

The proxy uses only the existing /completion API from both instances. No internal llama.cpp changes required.

Architecture

• Draft server: llama-server on GPU machine (e.g., -m 4B.gguf -ngl 99)
• Target server: llama-server on CPU machine (e.g., -m 122B.gguf -ngl 0)
• Proxy: thin Python/Go layer that:
  1. Gets K draft tokens from draft server
  2. Feeds prompt + K tokens to target server as extended prompt
  3. Measures acceptance by comparing where target continues from
  4. Loops until generation complete

Key design constraints

• Works with unmodified llama.cpp instances
• Draft and target can run different models, different hardware, different locations
• Network overhead is <1% of total latency for targets slower than ~50 t/s
• Tokenizer compatibility (same vocab) required between draft and target

Tasks

• Implement proxy server with OpenAI-compatible /completion endpoint
• Add acceptance rate measurement
• Benchmark: solo vs distributed spec speedup
• Document setup for common configurations

[SleepinDevil]

I have two different GPU in my PC and using Vulkan to play with them both at the moment. Having the lightweight speculative draft model on one GPU and having the main large model on the main GPU already introduces so much overhead that the performance I get is a lot lower than using a model with MTP as my main large model 🤔.

I know what I am mentioning is sort of a side topic, but if there's already that much overhead through just the PCIe bus, I am not sure if the extra overhead adding the networking stuff will be worth it.

In your use case, what I am suggesting is that if that large model you have running on CPU supports an MTP varient, using that will likely yield better performance than offloading the speculative model onto a networked GPU in another machine (even if that was a feature).

[ggerganov]

@7avgerenko This is already supported via the RPC backend (-DGGML_RPC=ON). Here is a working example you can adapt to your use case:

# main machine
llama serve \
  -hf ggml-org/Qwen3.6-27B-GGUF:Q8_0 \
  -dev MTL0 \
  --rpc 127.0.0.1:50052 \
  --spec-type draft-mtp \
  --spec-draft-hf ggml-org/Qwen3.6-27B-GGUF \
  --spec-draft-model mtp-Qwen3.6-27B-Q4_0.gguf \
  --spec-draft-n-max 3 \
  --spec-draft-device RPC0 \
  --host 0.0.0.0 --port 8014 \
  --chat-template-kwargs "{\"preserve_thinking\": true}" -lv 4

# drafting machine with IP=127.0.0.1
./bin/rpc-server --host 127.0.0.1

[7avgerenko]

Wow, I wasn't expecting a response from a real HF member. You helped me twice: solved my problem AND made me feel lucky today ;-).