v0.1.0

v0.1.0 — serve it: OpenAI-compatible API, tool calls, prefix caching

mlx-dspark grew up from a library + demo CLI into a usable local tool. You can now serve a DSpark / DFlash model to LM Studio, the openai SDK, or any OpenAI-compatible client — with the speculative speedup transparent behind the API. Everything stays lossless (greedy is byte-identical to plain decoding up to fp ties; --temperature > 0 is an exact sample from the target at T). No heavy dependency added — the server is stdlib-only.

Highlights

• OpenAI-compatible API server — mlx-dspark serve --model <repo> → http://127.0.0.1:8080/v1. Point any OpenAI client (LM Studio, the openai SDK, curl, LangChain) at it:
  • POST /v1/chat/completions (streaming SSE and non-stream, multi-turn), POST /v1/completions, GET /v1/models, GET /health, GET /metrics.
  • Serves dspark / dflash / baseline on one target. temperature, top_p, top_k, max_tokens, stop, seed, optional --api-key, CORS. Each response carries an x_mlx_dspark block (accept length + tok/s) so the spec-decode gain is visible.
• Prefix caching (in-memory + optional SSD spill) — reuse the shared conversation prefix's KV across turns instead of re-prefilling it. ~13× faster follow-up turns on a ~750-token shared context (measured: 87 ms vs 1132 ms). Lossless to the same fp-tie standard; invalidated on any error so it can't desync. On by default for dspark/baseline on dense targets (Qwen3); falls back for Gemma-4's rotating caches and DFlash.
• Tool calling — OpenAI tools / tool_calls, parsed from both native formats (Qwen3 Hermes-JSON and Gemma-4's <|tool_call>call:…), streamed as delta.tool_calls, with a full request → tool-call → result → answer round-trip.
• Lossless top-p / top-k sampling — nucleus / top-k truncation applied to both the draft and the target, so temperature sampling stays an exact sample from the (truncated) target. Validated model-free.
• Model-centric interface — name the target with --model <hf-repo | local-path> (like mlx-lm); the matched drafter auto-resolves (quantization-agnostic), or pass --drafter. This makes Qwen3-8B a first-class target and replaces the old 2-value --family. mlx-dspark models lists targets with a known drafter.
• Thinking toggle — per-request enable_thinking / chat_template_kwargs and a server --no-thinking default (silences Qwen3 <think> blocks for a served endpoint).
• Subcommand CLI + tests — serve / generate / models / doctor, a mlx-dspark console-script entry point, and a 45-test model-free suite (server protocol, streaming, stop sequences, tool-call parsing, top-p losslessness, prefix-cache manager, drafter resolver).

Quickstart

pip install -U mlx-dspark
mlx-dspark serve --model mlx-community/Qwen3-8B-8bit        # → http://127.0.0.1:8080/v1

from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed")
print(c.chat.completions.create(model="Qwen3-8B-8bit",
      messages=[{"role": "user", "content": "Explain rainbows briefly."}]).choices[0].message.content)

API

• New: mlx_dspark.server (Engine, run_server), encode_messages (multi-turn), resolve + REGISTRY (target→drafter), mlx_dspark.tools, mlx_dspark.sampling, mlx_dspark.prefix_cache.
• speculative_generate / dflash_generate / greedy_generate gained prompt_ids=, stop=, top_p= / top_k=, cache= / ctx_caches= / reuse_len= (prefix reuse), and a finish_reason on GenResult.
• No breakage: --family / --target / load_pair("qwen3") are kept as deprecated aliases, and the old flat python -m mlx_dspark --prompt … form still works.

Notes

• Prefix caching is exact for dense trimmable KV caches (Qwen3). Gemma-4's sliding-window caches and the DFlash drafter cache can't be safely rolled back to an arbitrary prefix, so they fall back to a fresh prefill (correct, just no reuse).
• MoE / linear-attention targets (Qwen3.5-*, gpt-oss-*) still need the gated-delta KV rollback that isn't wired yet — PRs welcome.
• The benchmark numbers from v0.0.3 (DSpark vs DFlash head-to-head) are unchanged and still reproducible.

Full diff: v0.0.3...v0.1.0