Skip to content

v0.10.0 — Speculative decoding

Choose a tag to compare

@cattolatte cattolatte released this 07 Jul 01:02
90d88f3

Greedy-exact speculative decoding, from scratch: a small draft model proposes tokens, the large target verifies them in one pass, and the output is byte-for-byte identical to greedy decoding on the target. SpeculativeStats reports acceptance rate and the target-forward speedup; KVCache.truncate rolls back rejected drafts.

Real benchmark (0.6M draft + 10.7M target, tiny-shakespeare): ~3× fewer target forward passes at identical output. Honest caveat in BENCHMARKS — forwards ≠ wall-clock at this tiny scale (~1.05–1.08× on MPS); the win grows with the target/draft size gap. Includes phase-10 design doc + ADR-0010.