v0.5.0: Speculative Decoding with Draft Model

What's New

Speculative decoding — use a small draft model to accelerate inference by 2-3x.

How It Works

1. Draft model (e.g. 1B, fits in RAM) generates K candidate tokens
2. Target model (e.g. 70B, streams from SSD) verifies them in parallel
3. Accepted tokens skip expensive SSD-streaming forward passes
4. Rejection resampling ensures mathematically lossless output distribution

Why This Is Huge for SSD-LLM

The draft model lives entirely in RAM while the target streams from SSD. With 60-80% acceptance rates, the target model does ~40% fewer forward passes = ~40% less SSD I/O.

Usage

# Run with speculative decoding
ssd-llm run model-70b.gguf --draft-model model-1b.gguf --prompt "Hello" --draft-ahead 5

# Serve with speculative decoding  
ssd-llm serve model-70b.gguf --draft-model model-1b.gguf --memory-budget 8G

New Features

• --draft-model <path> — Path to draft GGUF (same tokenizer family)
• --draft-ahead <K> — Tokens to draft per round (default: 5)
• KV cache rollback for efficient rejection handling
• Streaming speculative generation
• Stats: acceptance rate, target forward passes saved

Stats

• ~5200 lines of Rust + Metal shaders
• 23 tests, all passing
• 1020 new lines in this release

Full Changelog

https://github.com/redbasecap-buiss/ssd-llm/blob/main/CHANGELOG.md