Skip to content

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 01 Sep 22:38
· 18 commits to main since this release

Speculative decode with the model's own draft head

The checkpoint ships a multi-token-prediction head that predicts the token
after next. The community conversion dropped those tensors; slotstream now
restores them with a one-time conversion from the official release (exactly
the 31 needed tensors via ranged downloads — 4.9 GB, ~2 minutes, not the full
checkpoint) and uses the head to draft a few tokens per step, verifying them
in one batched main-model pass.

  • 85.8% measured first-draft accept rate; 3.5 accepted tokens per verify
    round at depth 4 (measured across prose, code, lists, arithmetic).
  • --mtp auto|on|off on run/serve/doctor. Auto enables the head
    only where the multiplier beats spending the same 1.6 GB on expert cache
    (~120 experts/layer after the charge). Below that it measured ×0.96 — so
    auto keeps it off there, and that A/B is in MEASUREMENTS.md.
  • The Swift port of the head is bit-exact against the MLX Python
    reference (slotstream mtp-parity, a standing battery gate).
  • Sampling semantics are exact by construction: draws happen sequentially off
    verified logits, only for tokens the plain loop would also sample — same
    rng stream, same presence-penalty evolution, any temperature.
  • New gates: mtp-check (determinism, cross-request state integrity through
    the prefix cache, accept sanity) and mtp-accept (measure the accept curve
    yourself).

Convert the head once (needs the pinned model already pulled):

.venv/bin/python Tools/mtp_convert.py

Without mtp.safetensors everything runs exactly as before, head off.