v0.2.0

v0.2.0 — continuous batching, auto-calibration, prompt-lookup, penalties & logprobs

mlx-dspark now scales past single-user: serve concurrent requests in one batched pass, speculate on any model with a new drafter-free prompt-lookup mode (for the models without a matching DSpark/DFlash drafter), and let it tune the draft length to your Mac — plus OpenAI penalties, logprobs, and a fix for serving Gemma-4.

Highlights

• Continuous batching (serve --max-batch N) — run up to N concurrently-queued requests through one batched target forward so they share a single weight-read per step (ideal for a local agent swarm). Both the target verify and the DSpark drafter are batched; a lone request — or one using penalties / logprobs / temperature > 0 dspark — takes the serial path, so B=1 latency never regresses. Measured Qwen3-4B-8bit on an M4 Pro, 4 concurrent: baseline B=4 2.48× aggregate, batched dspark B=4 2.51× vs serialized baseline (130 tok/s; 1.73× over serialized dspark). Dense mlx-lm targets (Qwen3 / Llama / Mistral-class); Gemma-4 (mlx-vlm) falls back to serialized.
• --mode auto — picks the best available speculation for any target (a known DSpark drafter → else DFlash → else drafter-free n-gram lookup), so any repo serves with some speedup and no extra flags.
• --mode lookup + hybrid drafting — prompt-lookup (n-gram) speculation with no drafter at all, for any model; great when output copies its own context (RAG quotes, code edits, repeat/refine turns). Hybrid drafting (dspark, on by default) verifies a free n-gram continuation instead of running the drafter on rounds where the suffix already occurred, reaching ~2.4× on copy-heavy content while general chat stays ≈neutral (--no-lookup-drafts to disable).
• --max-draft auto — measures this machine + model's verify/drafter cost curves once (a few seconds, cached on disk) and picks the cap per round from the curves + a live acceptance estimate, so the cap tracks the hardware (M1→M5) instead of a hard-coded default. Lossless — the cap only sets how many drafts get verified.
• presence_penalty / frequency_penalty — OpenAI penalties applied to the target logits so speculative/greedy output equals sequential decoding of the penalized target (works at temperature > 0 too).
• logprobs / top_logprobs — chosen + top-k target log-probabilities per token, for both chat and completions, computed only when requested.
• mlx-dspark benchmark — a warm, device-stamped, reproducible sweep (--json) for the community M1→M5 matrix.

Fixes & performance

• import mlx_dspark crashed on transformers ≥5.13 — which fresh installs now resolve to — with AttributeError: 'str' object has no attribute '__module__' (mlx_lm's string-keyed tokenizer registration hitting transformers 5.13's stricter register, at import time). Fixed with a scoped, version-agnostic import-time compat shim (no transformers pin). Thanks to @zboyles for the report (#1).
• Serving Gemma-4 (mlx-vlm targets) was broken since 0.1.0 — every request failed with There is no Stream(gpu, 1) in current thread (mlx-vlm's load switches the loading thread's default stream). The engine now loads and generates on one thread; Gemma multi-turn also gets prefix caching now (reused while under the sliding window).
• Decode-path perf pass — incremental streaming detokenization (was O(n²) on long / thinking outputs), one device sync per spec round, and a pipelined baseline: Qwen3-4B baseline now matches mlx_lm.generate (~52 tok/s), dspark ~1.46×.
• Server polish — sampling defaults from the model's generation_config.json, --default-max-tokens 2048 / --max-tokens-cap 32768 (was 512/8192, which truncated thinking traces), a client disconnect keeps the prefix cache, LRU prefix-cache slots, and chunked prefill + wired-memory limit for small Macs.

Upgrade

pip install -U mlx-dspark          # or:  uv pip install -U mlx-dspark

No breaking changes — every new behavior is opt-in or a transparent default; existing --model / --mode dspark|dflash|baseline usage is unchanged.

Speedup summary (all lossless)

scenario	speedup vs baseline
single request, general content (dspark)	~1.4× (Qwen3-4B) … ~1.75× (Gemma-4 12B)
single request, copy-heavy content (hybrid / lookup)	up to ~2.4×
concurrent, 4 requests (--max-batch 4)	~2.5× aggregate
multi-turn follow-up (prefix caching, long shared context)	~13× faster follow-up turns

Full diff: v0.1.0...v0.2.0