Systematic benchmark for draft model selection in speculative decoding. Given a fixed target model, measures tokenizer compatibility, cost ratio, acceptance rate, and predicted speedup for every candidate draft.
speculative-decoding-real showed that cost_ratio is the dominant factor in speculative decoding performance. But cost_ratio is not fixed — it depends entirely on which draft model you choose.
This benchmark answers the practical question:
Given Qwen2-1.5B as target, which draft model produces the best speedup?
It closes a five-project arc:
speculative-decoding-impl -> how to implement
speculative-decoding-real -> why cost_ratio dominates
tree-speculative-decoding -> tree-based candidate expansion
batched-speculative-decoding -> gamma tuning at batch_size > 1
draft-model-selection-bench -> which draft to use
| Draft | Quantization | Compatible | Reason |
|---|---|---|---|
| Qwen2-0.5B | fp16 | Yes | — |
| Qwen2-0.5B | int4 | Yes | — |
| Qwen2-1.5B | int4 | Yes | — |
| gpt2 | fp16 | No | tokenizer class, vocab, specials, encoding |
| TinyLlama-1.1B | fp16 | No | tokenizer class, vocab, specials, encoding |
Cross-family drafts fail all four compatibility checks.
| Draft | Quantization | cost_ratio (prompt=256) | cost_ratio (prompt=512) |
|---|---|---|---|
| Qwen2-0.5B | fp16 | 1.31x | 1.22x |
| Qwen2-0.5B | int4 | 0.63x | 0.60x |
| Qwen2-1.5B | int4 | 0.54x | 0.51x |
| gpt2 | fp16 | 4.85x (incompatible) | 4.62x (incompatible) |
| TinyLlama | fp16 | 1.41x (incompatible) | 1.32x (incompatible) |
int4 quantization made the draft slower, not faster. At batch_size=1 on RTX 2070, BitsAndBytes int4 increases token latency relative to fp16 due to dequantization overhead.
| Draft | prompt_len | gamma | alpha | predicted_speedup |
|---|---|---|---|---|
| Qwen2-0.5B fp16 | 512 | 2 | 0.651 | 0.824x |
| Qwen2-0.5B fp16 | 256 | 2 | 0.604 | 0.797x |
| Qwen2-0.5B fp16 | 512 | 4 | 0.651 | 0.621x |
All predicted speedups are below 1.0. Speculative decoding is a slowdown on this hardware for all tested pairs.
Qwen2-1.5B int4 (same model as target) achieves alpha=0.875 but:
- cost_ratio = 0.51–0.54x
- predicted_speedup = 0.35–0.57x
High acceptance alone does not make speculative decoding viable. cost_ratio is the dominant term.
GPT-2 fp16 would achieve:
- cost_ratio = 4.62–4.85x
- max_theoretical_speedup = 2.25–3.44x
But it is incompatible. The bottleneck is not a lack of cheap models in the world — it is a lack of models that are both cheap and tokenizer-compatible.
round_cost = gamma * draft_decode_ms + target_verify_ms
expected_progress = mean_accepted_prefix_len + 1
spec_ms_per_token = round_cost / expected_progress
predicted_speedup = target_decode_ms / spec_ms_per_token
breakeven_accept_len = (round_cost / target_decode_ms) - 1
git clone https://github.com/JohnScheuer/draft-model-selection-bench
cd draft-model-selection-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python run.py
Runtime: approximately 25-40 minutes on an RTX 2070.
results/
model_profiles.csv — decode latency per model, prompt_len, gamma
pairs.csv — full pairwise analysis with all metrics
ranking.csv — best exact-compatible draft per prompt_len × gamma
plots/
01_speedup_vs_gamma.png — predicted speedup vs gamma
02_cost_ratio_vs_alpha.png — cost ratio vs acceptance
03_breakeven_margin.png — margin to breakeven vs gamma
draft-model-selection-bench/
├── src/
│ ├── config.py — model specs, prompts, gammas
│ ├── data.py — sliding window examples from corpus
│ ├── compatibility.py — tokenizer compatibility checks
│ ├── profiler.py — per-model timing measurements
│ ├── pairwise.py — acceptance simulation and speedup
│ ├── predictor.py — analytical speedup formulas
│ ├── bench.py — sweep orchestration
│ └── analysis.py — plots and ranked tables
├── results/
├── plots/
├── run.py
├── SUMMARY.txt
├── DESIGN.md
├── LICENSE
└── requirements.txt
- Python 3.10+
- PyTorch >= 2.1.0
- Transformers >= 5.0.0
- BitsAndBytes >= 0.43.0
- Accelerate >= 0.30.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
- NumPy >= 1.26.0
- NVIDIA GPU with >= 8GB VRAM
- End-to-end generation speedup with full speculative decoding implementation
- Temperature-based acceptance (argmax agreement used as proxy)
- Batch size > 1
- Target models larger than 1.5B
- Frameworks other than BitsAndBytes for quantization
- DESIGN.md — full design rationale and module descriptions
- SUMMARY.txt — plain-text findings with all numbers
- LICENSE — MIT License
- speculative-decoding-impl
- speculative-decoding-real
- tree-speculative-decoding
- batched-speculative-decoding
- prompt-compression-bench
- attention-sparsity-bench
MIT License — Copyright (c) 2026 João Felipe De Souza
See LICENSE for details.
João Felipe De Souza
