Releases · carloslfu/slotstream

Release list

v0.2.2 Latest

Latest

github-actions released this 02 Sep 15:55

v0.2.2

4a8dcbe

0.2.2 — speculative decode that pays, and ships

Speculative decode with the model's own draft head now gives a measured +24% decode on a quiet 48 GB Mac (10.3 → 12.8 tok/s at the 122 experts/layer auto plans with the head on; ×1.33 on a code prompt, ×1.19 on a list, ×1.18 with the server's default sampling), and pull fetches the head, so --mtp auto works out of the box. Everything below is measured on the released binaries, five interleaved pairs per number; MEASUREMENTS.md M9 has the method.

The draft head ships with the weights. mtp.safetensors (1.47 GB, sha256-pinned) is hosted on the weights mirror and pulled with everything else, as the manifest's one optional file: a source without it leaves the pull green with a notice, pull --verify skips it when absent, and the startup check never asks to repair it. The weights are 105.3 GB in 25 files.
A rejected draft no longer re-runs the kept tokens. The verify pass records the recurrent state after every position (the GDN recurrence stepped one token at a time, bit-identical to the fused kernel), so a rejection rolls back for free. This is what turned the head from break-even into a gain: at 57 experts/layer one draft went ×1.12 → ×1.20; at 122, ×1.17 → ×1.24 (two drafts ×1.27).
Default draft depth is 1 (was 4). Four drafts lose at every size measured, including ×0.88 where auto turns the head on; one draft is the best or tied-best everywhere and wastes the least on a rejection. SLOTSTREAM_DRAFT_DEPTH still overrides.
The numbers are honest now. The ×1.5–1.9 the 0.2.0 docs projected assumed a five-token verify pass costs one token's pass; new hidden mtp-passcost measured 1.65 (a sixth of a pass per extra token), and the plateau A/B that was "pending" has run. Auto's threshold reads 28 GB, not ~26.
Gates. mtp-check now bounds the reused speculative state's logits by the plain re-chunking band (the prefix-check method) instead of comparing liveness, proves the recording pass exact against the batched one, and checks a rollback state-by-state against the plain path. mtp-bench --sample measures the sampled case.

Below a 28 GB target nothing changes: auto keeps the head off there, where it would lose, and every install without the head runs exactly as before.

Built by CI from commit 4a8dcbe (build log: https://github.com/carloslfu/slotstream/actions/runs/33650634699).

sha256: 2f459ece57e3ccfe3bf34abf9ae268c29161295b2f2a3f6a5a58e50a8b6b6c47

Verify provenance: download slotstream-arm64.tar.gz, then: gh attestation verify slotstream-arm64.tar.gz --repo carloslfu/slotstream

Install or upgrade: curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

Assets 4

v0.2.1

github-actions released this 02 Sep 03:43

v0.2.1

2009889

Built by CI from commit 2009889 (build log: https://github.com/carloslfu/slotstream/actions/runs/33587525974).

sha256: 2326d00ff7611049b3ebe73c5a505f183fc9a950a92e9b95e8ac3e624d5118a8

Verify provenance: download slotstream-arm64.tar.gz, then: gh attestation verify slotstream-arm64.tar.gz --repo carloslfu/slotstream

Install or upgrade: curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

Assets 4

v0.2.0

github-actions released this 01 Sep 22:38

v0.2.0

3bb094f

Speculative decode with the model's own draft head

The checkpoint ships a multi-token-prediction head that predicts the token
after next. The community conversion dropped those tensors; slotstream now
restores them with a one-time conversion from the official release (exactly
the 31 needed tensors via ranged downloads — 4.9 GB, ~2 minutes, not the full
checkpoint) and uses the head to draft a few tokens per step, verifying them
in one batched main-model pass.

85.8% measured first-draft accept rate; 3.5 accepted tokens per verify
round at depth 4 (measured across prose, code, lists, arithmetic).
--mtp auto|on|off on run/serve/doctor. Auto enables the head
only where the multiplier beats spending the same 1.6 GB on expert cache
(~120 experts/layer after the charge). Below that it measured ×0.96 — so
auto keeps it off there, and that A/B is in MEASUREMENTS.md.
The Swift port of the head is bit-exact against the MLX Python
reference (slotstream mtp-parity, a standing battery gate).
Sampling semantics are exact by construction: draws happen sequentially off
verified logits, only for tokens the plain loop would also sample — same
rng stream, same presence-penalty evolution, any temperature.
New gates: mtp-check (determinism, cross-request state integrity through
the prefix cache, accept sanity) and mtp-accept (measure the accept curve
yourself).