MTPLX 2.6.0

MTPLX 2.6.0 — concurrency

Until now, speculative decoding was a single-user feature: the moment two
requests hit the daemon at once, everyone fell back to plain autoregressive
batching and lost the MTP speedup. 2.6.0 removes that trade-off. This release
also brings embedding and reranking endpoints, LiquidAI LFM2 support, and a
real correctness fix to temperature-0 decoding.

Concurrent speculative decoding (--scheduler-mode mtp_batch)

The new scheduler serves independent requests through fixed-width MTP
cohorts. Each row owns its own state and sampling decisions, drafts are
verified in one batched target forward, and rows join and leave mid-flight
without disturbing their neighbours. Two cohort widths (three-wide and
eight-wide) install side by side and the scheduler seals each cohort at the
narrowest width that fits, so two concurrent agents don't pay for eight
lanes of padding.

Measured on Qwen3.6-35B-A3B on an M5 Max: the three-wide lane holds
1.7-1.8x the per-request decode of the padded eight-lane shape, and against
the previous production ar_batch route the same concurrent agent workloads
decode at 1.6-2.25x per lane, sampled at the model's shipped settings.

Honesty controls ship with it: --mtp-batch-numerics picks between
throughput, balanced, and b1-exact profiles with documented
trade-offs and an install-time self-check, and per-request stats report
each row's own truth (its own accepted-depth histogram, cohort width, and
restore provenance) instead of cohort averages.

The session bank composes with the cohorts: a request whose prefix is
banked restores it at cohort admission, prefills only its uncovered suffix,
and commits its own prompt boundary before the merge — agent fleets with a
shared system prompt keep warm time-to-first-token under concurrency. The
plain ar_batch lane learned the same trick.

This work is by David Tai (@davidtai): the scheduler contract, the
row-owned decode, the cohort serving, the numerics profiles, and the
docs. The width-3 bucket, session-bank composite, and live QA came out of
the joint hardening passes on top.

Embeddings and reranking (/v1/embeddings, /v1/rerank)

Contributed by @Cyb3rb1ade (PR #212). The daemon can now serve embedding
and reranker models beside chat, so a retrieval-backed setup doesn't need a
second inference server. OpenAI-shape embeddings (including the
dimensions Matryoshka truncation), Cohere/Jina-shape rerank, opt-in
per-model flags, lazy loading, an LRU resident cap, and idle release under
memory pressure. /v1/models stays chat-only by default so model pickers
never offer an embedder as a chat target; retrieval-only ids answer chat
requests with a clear 400. Checkpoints that ship their own Python code are
refused unless you explicitly pass --retrieval-trust-remote-code.

LiquidAI LFM2 / LFM2.5

By David Tai (@davidtai). The LFM2 family serves natively with a bit-exact
ShortConv decode fast-path and a verified think/tool grammar (parser stamp,
native tool prompt, pythonic streaming dialect). IQuest-Coder checkpoints
serve target-only AR through the same registry honesty: recognized, served
without MTP claims, refused cleanly when the quantization can't execute.

Temperature-0 output is token-identical again

The speculative lane's cold prefill fed the whole prompt through the model
in one window while plain decoding splits it into body plus a final
single-token step. The two shapes round differently in the last bit, so the
speculative lane started from a cache one ulp apart from the plain lane's —
enough to flip greedy argmax at a near-tie and break the "temperature 0
matches plain decoding" contract. Every cold-prefill path now partitions
the prompt identically. The Optimized Speed V2 artifact, which surfaced the
flip, passes its greedy exactness gate at every depth again.

Fixes

• Prefix restores no longer corrupt the session bank (#247): restores
  install fresh zero-copy views, so an interleaved near-prefix request can
  never rewrite a neighbour's banked span. From the reporter's reproducer,
  which ships as a regression test.
• Streaming tool calls no longer duplicate argument values into
  delta.content (#249).
• Solo requests carrying presence/frequency penalties answer on the
  composite scheduler instead of returning HTTP 500.
• mtplx serve --no-auth actually parses (#235 follow-through) — 2.5.4
  promised it, the public CLI rejected it.
• qwen3_5_mtp checkpoints validate and serve again — the MTP surface now
  attaches at the TextModel level. Cherry-picked from PR #242 by
  @davidtai with its regression test.
• Artifacts launch at their measured depth again: the typed runtime
  contract silently dropped measured-depth maps, so 35B launched at its D3
  ceiling — a measured ~22% decode loss against its fastest depth.
• --reasoning-parser is authoritative; the backend codec no longer
  silently overrides an operator-typed parser.
• Metal buffer-object leak in long decodes: mlx-lm's ArraysCache regrew
  buffer objects on every advance; the fix is vendored in-tree so pip
  installs against stock mlx-lm 0.31.x get it too.
• Missing MTP heads degrade to target-only AR serving with the reason
  surfaced, instead of refusing the checkpoint. auto_map checkpoints are
  refused with the policy stated plainly; unrunnable quantizations are
  refused with the offending bit-width named.
• OpenAI dimensions honored on /v1/embeddings; out-of-range values are a
  clear 400.
• AR batch hardening: cache-removal errors fail closed, completed streams
  no longer starve behind running neighbours, and the vendored Metal shader
  cache is keyed by MLX ABI so an MLX upgrade can't serve stale kernels.

QA (this release)

• Full pytest battery green at the release tip; 567 Swift app tests green.
• Greedy exactness gate for Optimized Speed V2: 3/3 depths token-exact
  (was 0/3 on 2.5.4).
• Four-arm performance sweep against the shipped 2.5.2, 2.5.3, and 2.5.4
  wheels (same harness bytes, fans verified at max, die-temp gated,
  candidate interleaved with baseline): single-request decode
  flat-to-faster than 2.5.4 (interleaved means 74.8 vs 69.2 tok/s on the
  27B artifact), cold TTFT flat (6.25s vs 6.26s on a 4.8k-token prompt),
  warm TTFT flat at the 2.5.4 session-bank floor (0.14s vs 0.15s). A
  serial-lane sampling regression introduced mid-cycle was caught by this
  sweep and fixed before release.
• Live QA on both product surfaces at the release tip: mtplx serve from
  a clean wheel install (streamed think and content, 64-70 tok/s) and the
  macOS app driven end-to-end (chat round trip, 52.5 tok/s reported by
  the app, clean stop and quit).

Credits

• David Tai (@davidtai) — concurrent MTP serving stack, LFM2 support,
  qwen3_5_mtp fix (PR #242). 26 commits in this release.
• @Cyb3rb1ade — embeddings + rerank endpoints (PR #212, 17 commits).