mlx-bun v0.0.11

v0.0.11 release notes

Everything since v0.0.10. The big themes: mlx-bun grows a second modality
(audio), a constrained-decoding stack (structured output / grammar),
speculative decoding, a novel KV-cache quantization scheme (TurboQuant),
completed dynamic batching, crash-isolated serving, and a ground-up
rewrite of the web chat UI. The decode-default story also resets: after
benchmarking every custom performance kernel in a paired A/B against the
bit-exact baseline and finding none of them won, they were deleted —
naked mlx-bun serve now runs the faithful, mlx-lm-parity kernel set by
default.

Audio input (Gemma e4b)

• OpenAI-style input_audio parts are served end-to-end: WAV/AAC
  transcode, a USM mel-spectrogram extractor, and a Conformer tower
  ported bit-exact (rel-RMSE 2.4e-8) against the oracle.
• Build-out ran phase-by-phase (A0–A5): a bun:ffi conv2d binding for
  the front-end, the multimodal prompt builder wired to the LM, and
  capability discovery for audio surfaced consistently across
  /v1/models, /library, and the pi websocket handshake — mirroring
  how vision capability is advertised.

Structured output / grammar-constrained generation

• response_format (json_object, json_schema, text) and
  guided_grammar/guided_regex/guided_choice/structured_outputs
  are wired into both /v1/chat/completions and /v1/completions, built
  on xgrammar: a GrammarController masks logits inside the decode
  loop, with a per-TokenizerInfo compiler cache and a MLX_BUN_GRAMMAR=0
  kill switch.
• The batched lane got its own per-row matcher path so grammar-constrained
  requests don't force a request onto the serial lane, and grammar
  composes with --draft-model (the verify walk respects the mask).
• Two defects found and fixed during the build: the oMLX-parity
  JSON-system-prompt degrade path (used when a client can't do function
  calling) had gone unreachable behind a dropped hint, and the WASM
  grammar controllers were leaking on every early-400 reject path
  (SSRF-blocked media, vision/audio validation failures, bad
  logit_bias, etc.) — both fixed, with disposal now covering all paths.

Speculative decoding

• --draft-model lands in serve: a two-model, serial-lane speculative
  decoding path verified token-for-token against mlx-lm's own
  --draft-model oracle (DraftSource seam, rewind rule, batched-verify
  lm-head).
• An ngram / jump-forward draft kind ships alongside the model-based
  drafter for cases with no small model available.
• A DeepSpec (DSpark) drafter-quant research track is underway
  (Phase 1 landed) targeting the larger models; per
  benchmarks/RESULTS.md, the first live pair (12B target +
  bf16 DeepSpec drafter, 26–33% acceptance) measured net slower
  than serial on a loaded box — spec decoding stays off by default
  pending a quantized-drafter pair that clears the 1.3× bar.

TurboQuant KV-cache quantization

• --kv-quant turbo[:kXvY] adds a rotation-based per-group KV
  quantization scheme (sign-FWHT + Lloyd-Max for values), proven
  bit-exact against the vendored vllm-metal reference
  (goldens/turboquant.json).
• Per benchmarks/RESULTS.md, the k8v3 default holds a lower KL-vs-bf16
  than affine uniform kv4 while compressing KV 2.56× (measured on
  MiniCPM5-1B, paired quality/KL harness). It's a memory/context lever,
  not a speed lever — v1 dequant-on-fetch is slower per decode step at
  long context, so it stays opt-in, not default, per the project's
  "levers must earn defaults" rule.

Batching: Phase D completed

• Aggregate --kv-budget <GB> admission with FIFO queuing, vectorized
  homogeneous sampling (penalties/logit_bias/min_p/XTC all run
  per-row in the batch), and extend-join so a request can join an
  in-flight batch mid-flight instead of waiting for a full drain.
• Per-row RoPE lands for Tier-0 universal architectures, lifting the
  batching gate for those models (e.g. Llama).
• Batched mixed-precision quantized KV — the first stack to compose
  per-row KV quantization with batching.
• --batch default changes from 1 to 8 (see Breaking changes below).

Runtime isolation

• --isolate runs the engine as a crash-isolated child process behind a
  reactor parent, restarting with backoff on a crash — the product rule
  is "the AI may crash, the UI never may" (docs/design/runtime-isolation.md).
• --model-pool extends this to a child-per-model pool: LRU residency,
  spawn-overlap switching (no dead time swapping models), and
  POST /admin/drain for graceful eviction.

Optional paged KV cache

• --paged-kv adds a vLLM-style block-table paged KV cache
  (PagedKVCache + BlockPool) feeding the unchanged stock SDPA via
  block gather. Default off; v1 scope is serial --batch 1, the
  Gemma4 family, bf16 — it refuses to silently combine with
  --batch N>1, --kv-quant, or --draft-model.

Prompt cache overhaul

• Prefix-sharing v1: take() now returns non-consuming, ref-counted
  clones instead of consuming/trimming the donor entry, so multiple
  agents or sessions can reuse one prefill without cannibalizing each
  other's cache.
• The SSD cold tier is promoted from an add-on to a first-class property
  of the cache store, with --ssd-demote-idle <seconds> (default 300,
  requires --ssd-cache) freeing GPU memory by demoting idle entries;
  /stats gains ssd_cache.demotions.
• A stable-boundary snapshot fix restores prompt-cache hits for
  multi-turn agent traffic beyond the sliding window — this was
  previously a silent cache miss on turn 2+ of long conversations.
• The SSD write-behind flush is now idle-gated, so background disk
  writes stop contending with decode at long context; the earlier A7
  RSS regression on the ssd-cache path closed with zero-copy write views
  and a streamed copy-restore.

Web chat UI: ground-up redesign (Phases 0–3)

• Streaming render rewritten to be block-memoized (no more O(n²)
  re-parse of the growing message), with a typed module split.
• New surfaces: a memory panel, adapter routing, a system-prompt editor,
  an approval gate, client-side RAG over chat history (BM25, no server
  round-trip), a Model Hub, Canvas, per-message sampling controls,
  self-healing tool calls, a command palette, full-text search, and
  chat export.
• An app-aware assistant (v1) that can see and navigate its own UI —
  spotlight/selector resolution now treats a model-invented or malformed
  selector as a miss instead of throwing inside the websocket handler.
• PWA support, plus a round of visual-QA fixes: the phone nav band,
  light-theme code legibility, a composer placeholder regression, memory
  panel markdown chrome, the theme-toggle Auto button, a clipped status
  pill, and tab flex-shrink pinned to zero on phones.

Security: media-fetch SSRF hardening

• image_url/audio_url fetches now refuse private/loopback/link-local/
  CGNAT destinations and non-http(s) schemes by default, re-validate
  through up to 5 redirect hops, enforce a 10s wall-clock timeout, and
  cap bodies at 64MB.
• LAN use needs the new escape hatch: --allow-private-media /
  MLX_BUN_ALLOW_PRIVATE_MEDIA=1.

Fixed

• 12B /v1/completions probe parity: adopted the oracle's step-0
  prefill convention (drain to len−1, step 0 from an L=1 forward of the
  last prompt token). Per benchmarks/RESULTS.md, live HTTP probes are
  now byte-identical to mlx_lm.server/optiq serve for MiniCPM5-1B,
  gemma-4-e4b, and gemma-4-12B, on completion and chat probes, in both
  the --batch 8 default lane and --batch 1.
• BPE detokenization now mirrors mlx-lm's bare-space hold-back and
  never-finalize semantics exactly.
• Cache invariant enforced on every tier; an FFI destructor deadlock
  fixed; write-behind made non-blocking; per-row batch cache extraction
  fixed.
• generate() now yields a macrotask between prefill chunks, keeping the
  isolation path (and the server generally) responsive during long
  prefills.
• Benchmark harness: the --batch 1 serial control arm now runs by
  default (it had been skippable, which undermined validating the new
  batch-8 default); --engine is additive instead of mutually exclusive
  with the serve pass; the batch-path test now asserts cumulative
  /stats submitted_rows instead of a stale cachedTokens=0 premise.
• any_whitespace grammar knob landed; batch-grammar gates re-anchored
  to compact separators.

Infra / benchmark harness redesign

• benchmark.sh + scripts/bench-modes.ts now spawn real CLI paths
  instead of bench-local server wrappers, folding bit-parity and
  peak-memory checks into the same serve cells (one pass instead of
  several). New columns: cold-start and restart-survival; new workload:
  xl (~4k prefix); a feature-matrix benchmark mode.
• Repo cleanup: lab/ consolidation, a binary-tracking hygiene gate, and
  a git history rewrite that took the repo from 182MB to 20MB, removing
  the last multi-MB binaries from the index.
• New design docs: unified-engine-frontier-plan, dspark-serving-program
  (the DeepSpec/DSpark spec-decode research program), structured-output,
  web-chat-redesign (+ app-aware-assistant, superset-doctrine, and
  visual-QA-bar notes), paged-kv-cache, batching-perf-path /
  batching-v2-plan, runtime-isolation.
• Doc drift swept: env-lever table entries (MLX_BUN_DSPARK_MINCONF,
  MLX_BUN_BATCH_NO_PIPELINE, MLX_BUN_EXPERT_OFFLOAD), ngram
  draft-kind coverage in README/server-api, doc-map sync, an openwiki
  investigation write-up. STATUS.md was rewritten to current-state-only
  several times through the release to keep handoff notes accurate.
• A new ground rule was recorded in CLAUDE.md: changes to the served
  surface must update the user-facing reference docs in the same commit.
• A direct optiq mixed-KV bit-parity golden replaced an indirect
  ours-vs-ours comparison; a bit-exact L1 parity harness was added for
  the Qwen3-30B-A3B MoE po...

mlx-bun v0.0.10

mlx-bun v0.0.10 — batching parity, SSD KV cache, sharper serving

Highlights

Concurrent serving now matches the fastest MLX batching servers. With
--batch 4, aggregate throughput at 4 parallel requests is at parity with
(or ahead of) oMLX on every shared model we benchmarked on an M1 Max —
MiniCPM5-1B 349 vs 339 tok/s, gemma-4-e4b within 3%, Qwen3.5-4B within 1% —
with 2–3× better time-to-first-token under load, and per-token SSE
streaming (no burst batching of your tokens).

• Qwen3.5 joins the batch lane. Hybrid gated-DeltaNet (SSM) caches grew
  the dynamic-batching ops; the batched decode path is verified
  token-for-token identical against mlx-lm's B=2 batched oracle.
• Sampler features batch too. Repetition/presence/frequency penalties,
  logit_bias, min_p, and XTC now run per-row inside the batch instead
  of forcing requests onto the serial path. (Qwen3.5 ships a default
  repetition penalty — before this fix it silently serialized every
  request under --batch N.)

SSD cold tier for the prompt cache (--ssd-cache <dir>). Long-context
prefix KV now spills to disk on eviction, is snapshotted after requests
settle, and survives server restarts: re-attaching a 13.7k-token
coding-agent conversation went from a 12.1 s full re-prefill to a 0.24 s
zero-copy mmap restore in our measurements — with 0% steady-state decode
overhead (nothing runs on the token loop). Entries are keyed by model
fingerprint + KV scheme + tokenizer hash + adapter namespace; corrupt or
incompatible files self-quarantine; the tier is always safe to delete.
Companion flags: --ssd-cache-max <GB> (default 32), --ssd-cache-verify.

Fixes

• serve/benchmark now honor --model <path-or-query> as an explicit
  override (it was previously accepted but ignored — auto-pick could
  silently load a different model). A directory containing config.json
  loads directly from that path.
• The server stays responsive during long serial generations: /stats,
  /health, and new connections previously stalled until the generation
  finished (measured 2.5 s on a 512-token run); they now answer in tens of
  milliseconds, with no measurable decode cost.

Internals

• KV persistence format v2: all cache kinds round-trip (bf16, per-layer
  quantized, sliding-window, quantized sliding-window, SSM), with
  integrity hashes, invalidation metadata, and atomic writes.
• Golden .bin parity fixtures are no longer tracked in git (they are
  machine-specific and regenerable); JSON manifests remain.
• New design docs: docs/design/omlx-adoption-map.md (feature adoption
  scoreboard), docs/design/batching-perf-path.md,
  docs/design/ssd-kv-cold-tier.md.

Full details and measured numbers: STATUS.md and the design docs above.
Credit where due: the SSD-tier and batching comparisons were driven by
studying oMLX (Apache 2.0) — a strong
project whose ideas we benchmarked and adapted where they made mlx-bun
faster.

mlx-bun v0.0.9

Everything since v0.0.8. The big themes: the mlx-lm drop-in surface is now materially complete, ORPO is paper-faithful, mlx-bun runs its first generically-supported architectures at verified bit-exact parity, and an adversarial review swept the whole engine — fixing 12 confirmed bugs and overturning a load-bearing performance assumption.

Drop-in compatibility with mlx_lm.server (the headline)

• Endpoints: POST /v1/completions (raw text completion, stream +
  non-stream), GET /health (byte-exact body), GET /v1/models now lists all
  registry models (+ /v1/models/<id>).
• Request fields: min_p, xtc_probability/xtc_threshold, logit_bias,
  presence_penalty/frequency_penalty (+ *_context_size windows),
  logprobs/top_logprobs with mlx-lm's EXACT semantics (same distribution,
  same response shape, same [0,11] validation). All L1-faithful ports of
  sample_utils.py.
• Flags/defaults: port 8080 + loopback host (matching mlx_lm.server;
  --host 0.0.0.0 = LAN opt-in), --temp alias, --max-tokens,
  serve --adapter <dir> mounts at startup.
• New CLI verbs: fuse (fold LoRA into base — mlx-lm math, untouched
  modules bit-identical), convert (wraps the mixed-precision quantizer,
  --target-bpw exposed), perplexity (mlx_lm.perplexity methodology),
  upload (native HF push).
• Deliberately not ported: role_mapping (unreachable — every supported model
  ships a chat template). Remaining: cache_prompt, evaluate (planned as an
  lm-eval shim), --draft-model, awq/dwq/gptq (plan:
  docs/design/mlx-lm-tool-parity-plan.md).

ORPO: paper-faithful by default

• sft_scope: full | response (default full): the chosen-NLL is now the
  token-mean CE over the full prompt+response — matching the ORPO paper, TRL,
  and xfactlab/orpo — implemented across every path (naive/chunked/fused/
  flash-CCE/prefix-shared/segmented). response reproduces old runs
  BIT-EXACTLY (regression-pinned). The odds-ratio terms stay response-only in
  both modes (matches TRL). CLI: mlx-bun train --sft-scope.
• The rest of the ORPO stack was adversarially verified correct against
  primary sources (odds-ratio math, gradients vs finite differences,
  prefix-sharing gradient flow, tokenization/label-shift).

Generic model support (Tier-0)

• New: src/model/universal/ — a config-driven UniversalDense module with
  explicit descriptors for llama, qwen2, qwen3, gemma, gemma2, phi3, olmo2,
  glm4, granite, starcoder2, smollm3 (+ mlx-lm's remapping: mistral etc.).
  bf16 (unquantized) checkpoints now load. Rope factory (llama3/yarn/longrope)
  verified bit-exact vs the oracle.
• First three archs verified at L1 parity (bit-exact per-step logits):
  Llama-3.2-1B-Instruct-4bit, Qwen2.5-0.5B-Instruct-4bit, gemma-2-2b-it-4bit.
• Dispatch ladder: dedicated → generated → generic → reject-with-helpful-error.
  Targeted models keep their optimized paths; a model can graduate.
  Design: docs/design/generic-model-support.md.

Fixed (the adversarial-review wave)

• --l2 tier contract restored: the L2 preset was enabling an
  envelope-gated perf kernel (not bit-exact vs optiq). Bare --l2 is now
  genuinely optiq-bit-exact; the perf kernel is --l3/explicit opt-in.
  generate --l2/--l3 no longer silently degrades to L1.
• Batched same-millisecond seed collision (identical concurrent completions);
  streaming per-layer KV-quant conversion (the missing half of optiq's
  tight-RAM fix); segmented compiled-decode mid-step failure KV corruption;
  MLX_BUN_TRAIN_ATTN=flash now refused on Gemma (unrevalidated crash path);
  eval-loss error swallowing; wrong tokens_per_sec metric; silent steel-kernel
  fallbacks now warn; flash-attention tests extended (D=256, sliding-window,
  non-tile T).
• Memory (the Dreaming): batching default flipped to serial (measured 1.7–1.9×
  faster for the real workload); memory status truth; third-person voice fix
  merged.
• CLI audit (14 findings): fit --ctx help matched code, embed auto-pick
  fixed, pi flag leakage into first message fixed, --l1/2/3 + generate +
  train-watch documented, setup is a real alias, doc lies corrected.

Performance truth (research, groundwork for the next release)

• The "decode is at the memory floor" claim was overturned with measured
  rooflines: only the 12B is at the bandwidth wall (~92%); CPM5/e4b/26B sit at
  58–70% with the host-side graph build as the top recoverable term. Ranked
  fix plan in-repo; wins land next release.
• The batching engine was audited (numerics solid, engine naive) — hotfixes +
  containment landing; plan: docs/design/batching-v2-plan.md.
• Registry duplicates root-caused (snapshot-per-commit accumulation);
  dedupe + gc landing.

Infra

• CI gate added (typecheck + model-free tests on push/PR) — previously nothing
  ran on push.
• Repo hygiene: scratch artifacts untracked, megakernel research relocated,
  STATUS.md rewritten as a true current-state front door, doc map regenerated.

Also in this release

• Batching engine v2 (steps 1–3): Qwen3.5/SSM models no longer fail under
  --batch (capability gate + serial drain); per-row failure containment; no
  serial-lane starvation; pipelined batched decode + sane cache cadence; the
  huge admit transient fixed; all GPU work under one lock.
• Model management: ls deduplicates to one row per repo (snapshot
  history via ls --all-revisions); new mlx-bun gc reclaims superseded
  snapshots (24.7 GB found on the dev machine) with a safety guard for
  unique files; get accepts substring queries; download lockfile, streamed
  resume verification, gated-repo auth hints; vision capability detected from
  config (the unified-vision 12B now shows vision). New doc:
  docs/reference/models.md.
• Pi integration: web-chat memory tools actually callable; harness pi
  registers exactly the served model; tool-calling fixed for Tier-0 generic
  models (Gemma sentinels no longer applied to non-Gemma tokenizers); vision
  advertised to pi; mlx-bun pi no longer leaks unknown flags into the first
  message.
• Website + README: six-goals restructure, drop-in guide, memory guide,
  the lab page, full CLI reference from a tracked source; deploy now triggers
  on reference-doc edits. NOTE the release sequence: push and publish in the
  same sitting — the site deploys from the push.
• Test infra: batched-serving goldens are machine-keyed (the M1-Max
  "failures" were M4-Pro fixtures; all code exonerated — mlx-lm reproduces
  mlx-bun token-for-token per machine); CI gate (typecheck + model-free tests).
• Research (the lab): decode-roofline re-measurement overturned the "at
  the floor" assumption (docs/investigations/decode-roofline-lookagain.md);
  curve-sampler distinctness theorem + witness numbers + preregistered
  protocol (docs/planning/curve-sampler-research-plan.md).

mlx-bun v0.0.8

mlx-bun v0.0.8

mlx-bun v0.0.7

mlx-bun v0.0.7

mlx-bun v0.0.6

mlx-bun v0.0.6

Highlights

• mlx-bun train — ORPO/SFT/DPO LoRA fine-tuning from the CLI. Runs the full
  ORPO stack (flash-CCE head + prefix-sharing + segmented backward) on by default,
  auto-detects e4b/Gemma, streams loss, and saves a mountable adapter.
• mlx-bun generate — a raw one-shot entry point with explicit sampling params.
• --l1 / --l2 / --l3 tier aliases — one intent switch over the
  parity/performance route. --l1 = mlx-lm bf16 bit-for-bit, --l2 = mlx-optiq
  bit-for-bit (perf kernel on), --l3 = mlx-bun's fast originals.

Added

• Web UI: sampling controls as sliders + floating popover; mixed-precision
  quantize with HF-cache output; ORPO web defaults.
• Live train-watch dashboard + metrics stream with ORPO stability guards.
• Control-flow DAG route map (flags, entry points, provenance) wired into the UI.

Fixed

• --prompt-cache 0 now actually disables the prompt cache.
• Tier default selects the fast kernel that holds the parity guarantee.
• Training: fixed flash-CCE/attention host-buffer pin leak; dispose nested-op
  intermediates in the flash-CCE forward + loss head; de-dupe duplicate BOS when
  tokenizing templated DPO prompts.

Install: brew upgrade joshuarossi/tap/mlx-bun · npm i -g mlx-bun · bunx mlx-bun

mlx-bun v0.0.5

mlx-bun v0.0.5

Headline: ORPO LoRA training on Apple Silicon, with an [M,vocab]-free flash-CCE head that makes large-vocab preference fine-tuning feasible on a single Mac. Plus the new mlx-bun.dev docs site and one-command install.

✨ ORPO training stack (new)

Train LoRA adapters with ORPO (Odds Ratio Preference Optimization) — reference-free preference tuning that reuses the DPO {prompt, chosen, rejected} data format at half the forwards/step.

• Flash-CCE head (forward + backward). The response-head logits are computed inside one Metal kernel built on MLX's verbatim steel quantized GEMM, so neither the [M,vocab] logits nor a dequantized head ever touch HBM. Backward 3687 → 754 ms on e4b (exact), peak 0.93 GB flat at M=8192 — large vocab (e4b 262k, MiniCPM5 130k) no longer dominates memory.
• Prefix-sharing (orpo_prefix_shared) — one forward over [prompt; chosen; rejected] (block-sparse mask + block-wise RoPE), so a shared prompt is encoded once. Big win on prompt-dominant data (e.g. long-document preference pairs). Per-row two-forward fallback when chosen/rejected prompts differ.
• Segmented backward (segment_size) — gradient-checkpointed layer activations for long context, composed with both the flash head and prefix-sharing.
• Warm-start — continue a run from a checkpoint's LoRA weights (RESUME=<adapter-dir> on the launcher; warm_start_adapter in the job API).
• Launcher — scripts/train-orpo.ts: full stack on by default, auto-detects e4b and sets its env, checkpoints every N steps, RESUME. See docs/reference/orpo-quickstart.md.

Supported scope (important): the full stack targets OptiQ-quantized (affine 4/8-bit) MiniCPM5 (Llama-arch) and Gemma-4 (e4b / 12B / 26B) models. Other architectures aren't wired for the segmented/prefix paths yet.

Validation: parity vs autograd — dh 0.40% (e4b) / 0.28% (MiniCPM5), bf16 class. Integration suite (tests/train-orpo-fused-ce.test.ts): flash / segmented+flash / prefix+flash / segmented+prefix+flash all train end-to-end. e4b @ 8192 full stack ≈ 13 GB / ~70 s/step on an M1 Max — the prior "e4b OOMs ≥ 2048" wall is gone.

📖 Docs site + reference

• mlx-bun.dev — an Astro Starlight documentation site, deployed via Pages, with reference docs generated from source at build (no drift).
• README corrected against source; experimental features labeled; KV-flag / library-API / benchmark sections fixed.
• New training reference (docs/reference/training.md) + ORPO quickstart.

📦 Install / distribution

• One-command install — mlx-bun.dev/install.sh (the recommended path), plus bunx and a direct-download option.
• Homebrew tap — one-command publish pipeline that auto-syncs the tap.
• Fix: disable-library-validation so brew-relocated dylibs load.

⚠️ Notes

• This release ships the ORPO trainer (the recipe), not a pre-trained adapter — you bring your own preference data and a supported base model.
• Apple-CCE backward skips (MLX_BUN_CCE_BWD_FILTER_EPS / _BLOCK_EPS) are off by default — exact gradients; opt in only on genuinely peaked data.
• Gemma e4b training requires MLX_BUN_PERF_KERNEL=0 + MLX_BUN_FUSED_GELU=0 (the launcher sets these automatically).

Full changelog: v0.0.4...v0.0.5

mlx-bun v0.0.4

mlx-bun v0.0.4

mlx-bun v0.0.3

First Homebrew release

MLX native runtime pack v0.1.0

libmlx + libmlxc + libjaccl + mlx.metallib (arm64, from brew mlx/mlx-c on 2026-06-12), load-commands rewritten for a self-contained directory. Downloaded automatically on first run into ~/Library/Caches/mlx-bun/. sha256: 90fa6a85bae648910bb957df3757270d581caf3294a06fb5195b44c5937d99da