rMLX 0.3.0

Metrics run-identity is now trustworthy. observations.backend_version was
wrong on 11 of 12 rMLX emitters — hard-coded '0.0.1' literals, absent values
that silently became NULL, and raw git SHAs stuffed into a semver field. The
root cause was structural: the §8.5 record had 12 construction sites and no
single integration point, so identity was merely the first field group to rot.
This release replaces all of them with one builder that cannot be bypassed, one
validator on every ingest path, and a rule the binary now follows without
exception: it stamps only what it can honestly know, and refuses to invent the
rest.

The serving surface — HTTP API, serve, chat — is unchanged. The breaking
changes are confined to the metrics/bench subsystem.

Changed

• BREAKING — §8.5 ingest now validates run identity. A record with
  backend: "rmlx" must carry a semver-shaped backend_version; a missing or
  malformed value is rejected on every ingest path (metrics record --file,
  --replay-pending, and the in-process recorder) instead of failing open to a
  NULL row. Other backends keep the field free-form and optional — llama.cpp has
  no semver and legitimately emits build_commit. See docs/METRICS_DB.md
  §8.5.1.

• BREAKING — the binary performs no git operations, at all. Not at runtime,
  not in build.rs. It previously resolved git_sha by shelling out to git in
  the process working directory, so an installed rmlx serve launched from a
  user's project stamped that project's HEAD — plus its -dirty state — into
  every metrics row it produced. Baking the SHA in at compile time was tried and
  rejected: Cargo does not re-run build.rs on source edits, so a work-in-progress
  binary filed rows as if they came from the pristine commit.

  git_sha is therefore caller-supplied provenance, exactly like
  hardware_tag: bench scripts stamp it (they run git -C <repo> rev-parse in
  their own checkout, where the question is cheap and honest), or a caller passes
  the new rmlx baseline --git-sha / rmlx eval ppl --git-sha. Absent → NULL,
  never guessed. Live-telemetry rows from the server carry NULL, which is
  correct — nothing bisects them.

• BREAKING — run_id is now YYYYMMDD-HHMMSS-<version>, not
  -<short-git-sha>. Affects logs/<run-id>.jsonl filenames and events.run_id.

• build_profile now reliably distinguishes release / release-perf /
  release-debug. cfg!(debug_assertions) reported all three as "release",
  so cross-profile perf comparisons were silently comparing unlike builds.

• RunRecord and RunIdentity can no longer be constructed or mutated outside
  rmlx-metrics. A hand-rolled record, a forged identity, or a post-hoc field
  write is now a compile error. Adding a new metric requires zero identity code.

Added

• --metrics {off|events|full} (global, default full), mirroring the
  existing --log flag. off is a producer-side no-op — no database opened, no
  drainer thread spawned, no runs.db created.
• rmlx metrics identity --json — the measured binary reports its own
  identity block, so shell emitters never guess or hard-code it.
• --git-sha <SHA> on rmlx baseline and rmlx eval ppl, for callers that
  want commit attribution on a recorded run.
• Migration 003 adds backend_version and build_profile to the events
  table, stamped from the same identity source as observations.

Fixed

• One-time pending-buffer quarantine. rmlx metrics record --replay-pending
  now rejects pre-contract rmlx buffer files (written before the
  backend_version requirement existed) rather than ingesting them as another
  NULL-version row. On the first run after upgrading, any such files move to
  metrics/buffer/failed/ and the command exits 2. This is expected,
  one-time behavior — not a regression. No file is deleted. See
  docs/METRICS_DB.md §8.5.1.
• RUSTSEC-2026-0204 — crossbeam-epoch bumped 0.9.18 → 0.9.20 (transitive,
  via criterion → rayon). make deny and make audit are green again (#198,
  #202).
• Clippy lints introduced by Rust 1.97.0, which had turned main latently red:
  every PR failed build + clippy regardless of content (#200).

Removed

• The compile-time git SHA, the RMLX_SOURCE_ROOT stamp, and the runtime
  working-tree -dirty probe — together roughly 300 lines, including the whole
  of build.rs's git handling (201 → 50 lines, it now only resolves the Cargo
  profile). They were the source of a recurring wrong-but-plausible identity bug
  that reappeared one layer down after each fix. Do not reintroduce them: the
  binary cannot honestly answer "what commit am I?", so it no longer tries.
• events.git_sha — a column no caller could ever fill. events is written only
  by the binary, which has no SHA to give, and nothing read the column.

Dependencies

• rustc-hash 2.1.2 → 2.1.3, uuid 1.23.4 → 1.23.5, time 0.3.51 → 0.3.53
  (#201).

rMLX 0.2.8

Qwen3.5-family model-loading correctness and a CI-gateable smoke probe. The
weight-quant loaders no longer corrupt mxfp8/mxfp4 scales, dense Qwen3.5 mxfp8
checkpoints now load via fact-driven dispatch (no longer hardwired to the PARO
path), and rmlx info --probe-smoke returns distinct exit codes so a load
failure can no longer masquerade as success. No breaking changes.

Added

• Dense Qwen3.5 mxfp8 loader + fact-driven dispatch. Both
  Qwen3_5ForConditionalGeneration and Qwen3_5MoeForConditionalGeneration
  now route by checkpoint facts, not the arch string: is_paroquant() selects
  the PARO vs the standard loader (the two share an arch string and differ only
  by quantization_config.quant_method), a shared resolve_prefix probes shard
  headers for the tensor prefix, and the MLP block is chosen per layer by tensor
  presence (dense SwiGLU vs sparse MoE). A defensive guard hard-errors if a PARO
  checkpoint ships MoE expert tensors. Dense Qwen3.5 mxfp8 snapshots now serve
  end-to-end. (#191, closes #189)

Fixed

• mxfp8/mxfp4 uint8 E8M0 scales corrupted at load → MoE prefill crash. The
  Qwen3.5-MoE and Qwen3 loaders blanket-cast every quantized .scales tensor to
  bf16, which is correct for affine (float) scales but corrupts mxfp's uint8 E8M0
  scales, crashing the first prefill with dequantize: Scale type must be uint8.
  A new per-tensor bf16_scales gate casts only float scales and passes uint8
  scales through verbatim. (#190, closes #188)

Changed

• rmlx info --probe-smoke now returns distinct exit codes for CI gating.
  Previously every non-Broken* outcome — including a supported-arch load
  failure and an inconclusive zero-token run — exited 0, so a loader regression
  read as a pass. Exit codes are now 0 ok, 1 broken, 3 load-fail, 4
  inconclusive, 5 unsupported (2 is reserved for clap arg-parse errors).
  healthcheck marks load-fail / inconclusive / broken as Red and unsupported
  as a non-fatal skip. (#193, closes #192)
• Bumped anyhow 1.0.102 → 1.0.103 (fixes a Stacked-Borrows UB in
  Error::downcast_mut) and uuid 1.23.3 → 1.23.4. (#187)

rMLX 0.2.7

Constrained-decode hot-path and Gemma4-unified vision tuning. The json_schema
and json_object per-token allow-mask probes no longer deep-clone their grammar
across the ~152K-token vocab on every decode step, and a whitespace stall in
schema-constrained decode is fixed. Gemma4-unified gains a per-request image-token
budget. No breaking changes.

Added

• Per-request + CLI image-token budget for Gemma4-unified vision. A
  image_max_tokens request field (and matching CLI flag) caps soft image
  tokens per request; default 280, ceiling 1120. Lets callers trade vision
  fidelity for prefill cost on the unified any-to-any path. (#181, closes #180)

Fixed

• Schema-constrained decode whitespace loop. Under response_format: json_schema, enum / scalar leaves accepted insignificant whitespace in
  states where it must be rejected (inside a literal, inside a string, at the
  root scalar start), letting temp=0 decode loop on \n. The allow-mask now
  matches whitespace per-leaf-state and rejects raw control chars (0x00..=0x1f)
  inside strings. (#183)

Performance

• json_schema constrained decode no longer deep-clones the schema per
  vocab token. The allow-mask probe reset a scratch SchemaGrammar ~152K
  times per decode step, deep-copying the immutable parsed schema each reset.
  The schema is now held behind Arc (Object.props, Union, Array.items),
  so entering a container/property/union branch is a refcount bump and the
  per-token reset reuses buffers in place. Per-step cost on the production path
  drops ~8–25× (was 20–40× heavier than the json_object engine; now
  comparable). Tool / function-calling agents pay this directly. (#184,
  closes #182)
• json_object constrained decode allow-mask reset is scratch-reused. The
  JsonGrammar reset became a state copy + Vec clear/extend (the stack frame
  is Copy) instead of a fresh clone per vocab token — ~2× on the per-step
  probe. The two engines now share one fill_allow_mask kernel over a
  ProbeGrammar trait. (#183)

rMLX 0.2.6

f32-KV-leak class hardening. The --kv-quant none KV cache no longer widens to
f32 on the Qwen3 path, and the leak is now structurally closed for every
architecture. Headline: Qwen3-dense (Bonsai-8B-2bit) none decode is ~+32…+87 %
across 4k–64k and now beats the mlx-lm reference at every context, with KV
residency halved. No breaking changes.

Fixed

• Qwen3 dense (Qwen3ForCausalLM) --kv-quant none KV stored f32, not bf16.
  Bonsai ships RMSNorm weights and quant scales/biases as fp16; bf16 activations
  × fp16 params promoted the residual — and the K/V projection outputs — to f32,
  so the cache stored f32 (4 B/element). Casting all Qwen3 float params to bf16
  at load (bf16_param) keeps the stream and the cache bf16. On Bonsai-8B-2bit:
  none-KV halved (≈0.53× the f32 MB), decode +32 / +47 / +68 / +82 / +87 % at
  4k / 8k / 16k / 32k / 64k, and prefill ~0.55×. (#168)
• Qwen3.6 MoE (Qwen3_5MoeForConditionalGeneration) hardened to bf16-param
  parity, including the GatedDeltaNet norm + conv1d weights; audited clean for
  the same f32-KV leak. (#171)

Added

• Model-agnostic bf16 floor at the KV-cache store boundary. The
  --kv-quant none cache casts K/V to bf16 at the single store choke point, so
  no architecture can store f32 there regardless of upstream dtype — a durable
  backstop for the per-arch fixes. Bytes-per-element invariant test wired into
  make model-check. (#169)
• CI gate make check-no-scalar-f32-leak flags unguarded scalar_f32( in
  arch-layer code (the f32-leak idiom). Surfaced and fixed 13 latent leaks across
  gemma3, gemma4 vision/audio, jina, bitnet, and dflash. (#170)

Dependencies

• safetensors 0.7→0.8, rusqlite 0.32→0.40, miniz_oxide 0.8→0.9, plus the
  cargo-minor-patch group; CI actions/checkout 6→7 and Swatinem/rust-cache.
  (#162–167)

Security

• memmap2 0.9.10 → 0.9.11, clearing RUSTSEC-2026-0186 (unsound out-of-bounds
  offset/len in advise_range / flush_range). rMLX maps safetensors
  read-only and does not call the affected functions, so it was not reachable —
  bumped to keep the advisory gate clean.

Docs

• Bonsai-8B (2-bit) full rMLX KV-quant matrix + sibling-backend champions. (#177)

rMLX 0.2.5

Prefill / time-to-first-token fix for the MoE families, plus a baseline
correction. Headline: Qwen 3.6 prefill is ~4× faster at short context and now at
mlx-lm parity. No breaking changes.

Performance

• Qwen 3.6 (Qwen3.5-MoE) prefill is ~4× faster at short context. The
  GatedDeltaNet recurrence flipped from the gated_delta_step_gpu Metal kernel
  to a lazy ops-graph at T≥256, which pinned the prefill chunk at 64 — a 4k
  prompt ran ~64 forward passes where mlx-lm runs ~2. Making the GDN always use
  the kernel (a byte-for-byte port of mlx-lm's gated_delta_kernel; chaining
  across chunks is f32-state-exact) unblocked raising the prefill chunk to 2048
  (mlx-lm's prefill_step_size). Warm-TTFT on Qwen3.6-35B-A3B-8bit (kv-none):
  4k 4240→1065 ms (4.0×), 8k 9008→2136 ms (4.2×), 16k 19489→4712 ms (4.1×);
  decode unchanged, no Metal watchdog through 64k. gated_delta_prefill_ops is
  retained as the test-only kernel-equivalence oracle. (#155)
• Gemma 4 prefill chunk raised 512 → 1024. A real-model sweep found 1024 the
  shared TTFT sweet spot: e4b 4k +6% / 8k +4.5%, 26b-a4b +17%; decode flat, no
  watchdog. chunk=2048 regresses the e4b dense path (a sliding-window /
  exec-unit cliff above 1024 = 2×window), so the shared gemma4 default
  stays 1024. (#155)

Documentation

• Prefill/TTFT is at mlx-lm parity, not "40–50× slower". The earlier
  "~40–50× slower than mlx-lm / 4k TTFT 144 ms / 28000 tok/s" framing was a
  non-physical baseline (the cited prompt-throughput exceeds the M5-Max
  bandwidth ceiling). A direct mlx-lm 0.31.3 run on the same Qwen3.6-35B-A3B-8bit
  snapshot + prompts measures 2711–3606 prompt tok/s vs rMLX's ~3050 — mlx-lm is
  only ~1.1–1.2× faster. README, docs/models/qwen3.6/rMLX.md, and
  docs/models/qwen3.6/SIBLINGS.md retract the claim. (#155)
• Gemma 4 e4b QAT complex-image vision is a checkpoint limitation, not a bug.
  Investigated degenerate / hallucinated output from the e4b-it-qat-mxfp4 and
  -qat-nvfp4 snapshots on high-detail screenshots (#153). The e4b QAT
  snapshots share a byte-identical SigLIP vision_tower and clipped-linear
  bounds with e4b-it-mxfp8; the unquantized qat-bf16 checkpoint degrades on
  dense images identically to the fp4 variants, and the mlx_vlm Python
  reference reproduces the same failure on the same snapshots. So this is an
  intrinsic quality limit of the e4b QAT checkpoint on complex images, not an
  fp4-dequant defect — rMLX output is reference-faithful. No code change;
  docs/MODELS.md now documents the behavior and recommends e4b-it-mxfp8 for
  complex-image OCR. (#153)

rMLX 0.2.4

Vision, KV, and embedding-lookup bug-fix batch for Qwen3-VL and Gemma 4, plus a
/metrics/cache recording/docs fix and a Homebrew bottle build+publish flow.
Highlights: Qwen3-VL large images now work end to end (KV sized from --max-ctx;
the O(seq²) embedding lookup that tripped the Metal GPU watchdog is gone), and
Gemma 4 image grounding is fixed by placing image tokens inside the user turn. No
breaking changes.

Added

• Homebrew bottle build+publish flow. scripts/release/build_bottle.sh +
  make bottle drive brew bottle against an installed keg, rename the local
  bottle to the GitHub-Release asset name, and emit the ready-to-paste
  bottle do block; documented as a release-time step in docs/RELEASING.md.
  The committed formula stays source-build until a real bottle is uploaded, so
  existing tap installs are unaffected. (#143, #139)

Fixed

• /metrics/cache TTFT empty for non-streaming completions. Both
  non-streaming paths (generate_blocking, OpenAI + Anthropic) measured TTFT
  but never pushed it into the in-memory ttft_store ring — only the streaming
  path did, so ttft stayed [] for non-streaming traffic. The ring is now
  written on both paths. docs/SERVER.md is realigned to the endpoint's actual
  shape (models[], itl, tokens_in/out), dropping the never-emitted
  prompt_cache / last_itl keys. (#142, #141)
• Gemma 4 image grounding (degenerate / image-independent output). The
  per-image token block was spliced after BOS but before the user-turn opener,
  leaving the image outside the user message; the model then ignored it. Image
  blocks are now spliced inside the (final) user turn via a shared
  splice_image_block, matching the HF/mlx-vlm placeholder substitution. Fixes
  the reported e4b QAT-fp4 degeneration (the soft tokens were correct all along)
  and a latent flakiness that affected all Gemma 4 image requests; Qwen3-VL is
  unified onto the same path. (#144, #140)
• Qwen3-VL ignored --max-ctx; large images failed with a slice_update
  broadcast. The image and text generate paths built KV with the bare 4096
  default and never bracketed prefill, so any prompt over 4096 tokens (a large
  image tiles to thousands of soft tokens) overran the fixed buffer. Both paths
  now size the KV ring from the effective --max-ctx and chunk the prefill;
  an over-cap prompt returns a clean context_overflow instead of the broadcast
  panic. (#145, #138)
• Qwen3-VL large images hit the Metal GPU watchdog. The quantized embedding
  lookup used an O(seq²) eye(seq) @ w identity-matmul on CPU (plus a GPU↔CPU
  round-trip); embedding the whole augmented prompt for a large image produced a
  single command buffer that overran the ~10 s watchdog. Replaced with on-device
  take + dequantize (O(seq)); added query-tiled ViT attention as a faithful
  defense for very large single images. (#147, #146)
• Qwen3.6 (qwen3_5_moe) embedding lookup carried the same O(seq²)
  eye(seq) @ w-on-CPU trick (plus an unsafe block); ported to the same
  on-device take + dequantize. Numerically faithful, removes a per-step CPU
  round-trip. (#149, #148)

Performance

• Qwen3-VL: large images (e.g. 2560×2560 → 6400 soft tokens) now complete
  end-to-end instead of aborting the process at the Metal GPU watchdog. (#145, #147)

Tested

• New CI-gated tests: image-token placement (in-turn, last-turn, multi-image,
  after-BOS fallback), ViT attention tiling equals a single SDPA, and
  qwen3_5_moe embed_lookup numeric equivalence across both dtype arms (the
  prior coverage was #[ignore] + env-gated). Real-model proofs across Qwen3-VL
  (KV + large-image), Gemma 4 e4b QAT-fp4 vision, and Qwen3.6 (decode-TPS
  same-session A/B: no regression).

rMLX 0.2.3

Multi-model registry hardening. Two --registry serving bugs fixed: the
multimodal encoder-output cache no longer leaks vision/audio features across
models, and eager model preload now respects --max-loaded-models. No breaking
changes.

Fixed

• Multimodal encoder-output cache cross-model leak. In --registry
  multi-model mode the vision/audio encoder cache was keyed on the
  post-preprocess content hash only, so a cached image encoding produced for one
  model (projected to its hidden_size) was returned to a different model for
  the same image — a vision-feature shape mismatch (HTTP 503) when the hidden
  sizes differed. The cache key now folds in a stable per-model signature, so
  entries are never shared across models; same-model repeats still hit. (#132)
• Registry eager-preload ignored --max-loaded-models. rmlx serve --registry preloaded every model at startup even with a smaller resident cap,
  paying the full load cost for models that were immediately evicted (a
  ~5-minute boot for a 13-model registry). Preload is now bounded to at most
  --max-loaded-models entries (the alphabetically-first ids, since the
  registry is id-sorted); the rest load on demand. (#133)

Changed

• README.md refreshed to 0.2.3 with an accurate "What works" summary, and
  docs/CLI.md documents that the multimodal cache key now includes model
  identity (no cross-model sharing) and that registry preload is bounded to the
  resident cap.

rMLX 0.2.2

Multimodal release. Whisper transcription works end to end (decode correctness

• long-form) behind a new model-agnostic rmlx transcribe CLI; the dense
  Gemma 4 12B gemma4_unified any-to-any architecture is now supported for image
  and audio input; the standard Gemma 4 family gains native audio input through
  the serve path; and the unified vision color-fidelity bug is fixed. Plus
  release-signing and CI-hardening housekeeping. No breaking changes.

Added

• rmlx transcribe <audio> --model <snapshot> [--format vtt|srt|json|txt] —
  model-agnostic audio transcription CLI, arch-dispatched on config.json
  (Whisper today, a clean seam for future ASR). Decodes any container to 16 kHz
  mono internally (enabled symphonia isomp4+aac, so .m4a works). The HTTP
  endpoint and the CLI share one long-form engine. (#119)
• Gemma 4 12B unified (gemma4_unified) image + audio input. The dense
  any-to-any 12B has no SigLIP/Conformer tower — vision and audio are
  early-fusion via soft tokens projected straight into the shared 48-layer LM.
  Faithful encoder-free ports of Gemma4UnifiedVisionEmbedder (host patchify +
  3×3 merge → patch_ln1 → quantized patch_dense → factorized 2D pos-emb →
  embed_vision) and Gemma4UnifiedAudioFeatureExtractor (raw 16 kHz waveform
  → fixed 640-sample frames → embed_audio). Dispatched off is_unified_arch;
  the standard e4b/26b/31b SigLIP path is unchanged. (#120)
• Gemma 4 native audio input through the serve path. The Conformer
  audio_tower + embed_audio projector + USM feature extractor now load at
  startup alongside the vision tower, and input_audio parts are decoded → mel
  → AudioEncoder → soft tokens scattered at <|audio|>, mirroring the vision
  flow. Submitting audio to a model without an audio tower (or combining image +
  audio) returns a clear 503 — no silent drop. (#122)

Fixed

• Whisper transcription was empty / garbage. large-v3 has 100 language
  slots, shifting every special token +1 vs the v1/v2 layout the constants
  assumed — so TOK_TRANSCRIBE pointed at <|translate|> and the
  timestamp-begin hard-stop fired on <|notimestamps|>. Corrected the
  special-token layout and added the missing in-loop logit filters
  (SuppressBlank, SuppressTokens derived generally from the tokenizer, and a
  faithful ApplyTimestampRules). Long-form decode bounds are derived from
  n_text_ctx at runtime so the positional table can't overflow. Full 48-min
  real recording at temp 0 → normalized WER ≈ 0.079, deterministic. (#119)
• Gemma 4 12B unified vision color corruption. The encoder-free path read
  image soft tokens causally, but gemma4_unified conditions each image's
  soft tokens with bidirectional attention (the SigLIP path hides this by
  pre-integrating the image in its ViT). A per-prefill bidirectional overlay,
  keyed off the <start_of_image>/<end_of_image> markers and merged
  element-wise into each layer's causal/SWA mask, fixes color naming and layout;
  gated on has_image so text prefill is untouched. (LayerNorm eps also
  corrected to the PyTorch nn.LayerNorm default 1e-5.) A 100%-uniform
  achromatic fill still reads as one level — an inherent property of the
  encoder-free projection (patch_ln1 normalizes the absolute level away),
  documented in docs/MODELS.md. (#127)
• --probe-smoke false BrokenPunctLoop on instruction-tuned snapshots.
  The probe fed a bare (no-chat-template) instruction; chat models degenerate on
  such out-of-distribution input (the mlx-lm reference reproduces it
  identically) — a probe artifact, not a 4-bit dequant bug. The smoke seed is
  now rendered through the snapshot's chat_template.jinja when present, falling
  back to the bare seed for base models; each entry point keeps its own canonical
  BOS resolver (no hardcoded id). (#121)

Security

• Pin CI actions (actions/checkout, dtolnay/rust-toolchain,
  Swatinem/rust-cache) to commit SHAs, add keyless cosign release signing
  (make release-sign), and drop a stale RustSec advisory ignore. (#116)

Changed

• scripts/release/source_sha256.sh --write now also bumps the formula url
  version, not just the sha256 (previously left the formula pointing at the old
  tag's tarball). (#118)
• docs/RELEASING.md documents the formula url-bump and Dependabot
  migration-push gotchas. (#115)

rMLX 0.2.1

Correctness + maintenance release. Closes a systemic KV-cache head-scramble
class that affected every flat quantized KV codec, hardens the SSD KV tier
and prompt cache, makes the single-MLX Metal claim self-heal after a crashed
holder, and unifies the per-architecture model code onto shared seams (decode
loop, loader, Architecture dispatch). Plus a round of dependency bumps. No
breaking changes.

Fixed

• Systemic KV head-scramble class. Every flat quantized KV codec wrote its
  buffer sequence-major but reshaped it head-major on dequant — agreeing only
  when batch × kv_heads == 1, and scrambling per-head K/V on any multi-append
  (decode after a multi-token prefill, or after an SSD hydrate) when
  kv_heads > 1 (grouped-query attention). Fixed family-wide with a canonical
  sequence-major layout (transpose on append + on dequant) and an explicit
  Array::contiguous before each custom MSL kernel, which reads its input by
  raw linear index and so cannot honor a lazy transpose. Covers QuantK
  (#103), QuantV / TurboSym-K / paged-K handoff (#108), the Iso/Rotor
  rotation codecs (#109), and PlanarQuant K/V plus its packed-K decode kernels
  (#110).
• SSD KV tier. Spill + restore now carry the bf16 K/V payload for
  KvQuant::None layers (#88); SSD-hydrated entries are excluded from the
  exact-hit fast path so a hydrate cannot be mistaken for an exact prompt-cache
  hit (#87); a Gemma 4 entry hydrated with an empty SWA layer degrades to a
  full re-prefill instead of decoding from a hole (#90).
• Prompt cache unified across architectures. A single model-agnostic
  consume engine replaces the per-arch hydrate/reuse glue and is retrofitted
  onto five architectures, so the SSD-hydrate / prefix-reuse correctness fixes
  above hold identically on every model (#98).
• GPU default stream on every inference entry. The image, speculative,
  audio, and embeddings blocking-thread entries now establish the thread-local
  GPU stream the text path already had, fixing intermittent
  no Stream(gpu, N) failures off the text path (#104). The adaptive
  prefill-chunk fallback resolves the loaded architecture instead of assuming
  Gemma 4 (#68).
• Metal claim self-heals. A stale claim left by a crashed holder is
  auto-reclaimed once the holder PID is proven dead (re-probed under the file
  lock); SIGTERM/SIGINT now shut the server down gracefully and release the
  claim (#112).
• Array::to_bytes evaluates before reading the data pointer, closing a
  lazy-eval race in the only reader of the raw MLX array buffer (#101).
• MetalKernel::new frees its input vector when output-name conversion fails
  (#60); the Planar3 V codec uses one packing path on CPU and GPU (#102) and
  warms its MSL kernels at precompile (#59); the resident-bytes estimator
  models Iso/Rotor sidebands exactly (#58); chunked_prefill exits prefill on
  every cache after a failure (#57); f16 negative subnormals no longer decode
  to -0.0 (#56); the tensor-view loader distinguishes not-found from I/O /
  parse failures (#4b5ea54 → see history).
• Gemma 4 loading: unquantized bf16 and affine-int4 (QAT) snapshots load;
  affine biases pass through the MoE expert gather_qmm; the perplexity scorer
  prepends BOS to every sliding window.

Changed

• Shared decode loop. Qwen 3, Qwen 3.5-MoE, Gemma 4, and Gemma 3 now run on
  one decode loop (per-arch copies removed); ProbeStep / SmokeVerdict live
  in the shared loop.
• Shared loader seam. All architecture loaders (Gemma 4 / 3, Qwen 3 /
  3.5-MoE / 3-VL-MoE, Laguna) adopt load_util::Weights — an index-first,
  header-truth tensor fetch; AWQ byte-math moved to rmlx-quant; a single
  read_raw_config helper replaces six per-loader clones.
• Architecture dispatch. Auto-KV default, KV-byte reporting, and
  prompt-cache stats now dispatch through the Architecture trait rather than
  arch-specific branches.
• Shared fused-QK setup scaffold (q8 / turbo-K3 / turbo-K4 / iso dispatchers
  ported onto it); arch modules construct arrays via
  Array::from_{i32,f32}_slice per docs/FFI.md; refuses_qwen_moe renamed to
  k_below_8bit (it is a codec property, not an arch rule).

Dependencies

• tokenizers 0.20 → 0.23 (encode/decode add-special / skip-special semantics
  preserved; verified on Gemma 4, Qwen 3.6, and Bonsai tokenizers) (#97).
• toml 0.8 → 1.1 (#94), tikv-jemallocator 0.6 → 0.7 (#96),
  criterion 0.5 → 0.8 (dev / benches) (#95), uuid 1.23.2 → 1.23.3 and
  time 0.3.47 → 0.3.49 (#93).

Tested

• Full KV-codec regression re-sweep after the head-scramble fixes: every codec
  class (QuantK/V, Iso/Rotor, Planar including its live fused-QK kernel) is
  within ±5 % of its recorded best decode cell on Bonsai, Gemma 4-e4b, and
  Qwen 3.6 — no decode regression. GPU round-trip tests assert each layout flip
  reconstructs true head-major K/V at quant noise (with pre-fix scramble
  controls).
• Tokenizer correctness re-proven on three tokenizer families (SentencePiece +
  BPE) at temp 0.

rMLX 0.2.0

Gemma 4 decode is now competitive with mlx-lm across the whole family, Gemma 4
speculative decoding (MTP) works end to end, the KV ring grows lazily with
per-request KV / context hot-swap, KV-cache metrics report live sizes, and the
env-var surface is cleaned up — breaking for shell configs that set removed
vars directly (see Removed).

Added

• Per-request KV-quant + --max-ctx hot-swap on a resident model — switch
  the KV codec or context ceiling per request without reloading the model. (#26)
• Per-layer KV net-benefit estimator — warns when a KV codec costs more
  resident bytes than it saves on a given layer mix (general across arches). (#34)
• Five env-var-only knobs promoted to proper --flag / env= pairs (the flag
  always takes precedence): --log-cap-mb, --yarn-factor,
  --yarn-original-max, --session-cache-max-sessions, --prompts-dir.

Fixed

• Gemma 4 speculative (MTP) functional end to end. Dispatch routes
  --draft-kind mtp by draft arch family and rejects a plain-gemma4 draft
  cleanly (#23); the assistant SWA mask uses array mode instead of the rejected
  additive mode (#24); a verify-step SWA mask off-by-one in both the producer
  and consumer branches is fixed (#32); and the loader supports both assistant
  LM-head variants — sparse centroid-routed (e2b/e4b) and plain tied-head
  (26b/31b) (#49). All four Gemma 4 sizes load and run coherent under MTP.
• Gemma 4 decode kept bf16 end to end. gelu_tanh f32 constants plus the
  embed / per-layer scales no longer promote the dense activation stream to f32
  (#44), and the MoE router's strong-f32 root-size scalar no longer leaks f32
  into the routing weights and the downstream KV (#51). Net: e2b/e4b beat mlx-lm
  decode, 26b-a4b MoE closed from −10…−28 % to −4…+1 %, and global --kv-quant none KV is halved (bf16) on every model.
• --max-ctx is a virtual ceiling — the KV ring grows on demand, so a high
  ceiling no longer penalizes small-prompt decode. (#25)
• Rotation / K-only KV codecs precompile their MSL kernels at load and are
  truthfully classified Metal vs CPU (no silent host-CPU fallback). (#36)
• Qwen3.6-MoE SSD-hydrated prefix skips prefill via a hydrated-tail path — a
  cache hit no longer re-runs the full prefill. (#9)
• Live KV-cache metrics — kv_cache_bytes reports the real resident size
  (was always 0) and counts the filled prefix, not the --max-ctx ceiling.
  (#33, #39)

Performance

• MoE prefill ~4× faster on gemma4-26b and Qwen3.5-MoE via sorted-index
  expert gather (contiguous per-expert access in gather_qmm) — 26b 128k cold
  TTFT ~403 s → ~117 s. (#46)

Tested

• Falsified the 6× SWA-KV claim: windowed SWA KV is window-bounded, not
  full-context (#35, #40).
• Full Gemma 4 and Qwen 3.6 KV × context bench matrices (per-model decode /
  TTFT / KV-size across all codecs) recorded under docs/models/.

Changed

• Env-var surface cleanup (chore/env-var-cleanup). Five previously
  env-var-only knobs are now proper --flag / env= pairs so the flag always
  takes precedence: --log-cap-mb (RMLX_LOG_CAP_MB), --yarn-factor
  (RMLX_YARN_FACTOR), --yarn-original-max (RMLX_YARN_ORIGINAL_MAX),
  --session-cache-max-sessions (RMLX_SESSION_CACHE_MAX_SESSIONS),
  --prompts-dir (RMLX_PROMPTS_DIR).
• docs/CLI.md env-var section restructured: split into User / operational
  and Internal / advanced subsections, with flag / default / description
  columns for every entry.
• docs/TESTING.md: added RMLX_KV_TEST_MODEL, RMLX_DRAFT_TEST_MODEL,
  RMLX_VL_TEST_MODEL, RMLX_TEST_MODEL to the specialised test-model table;
  added a Test behaviour toggles table covering RMLX_SKIP_GPU,
  RMLX_REGEN_GOLDENS, RMLX_E2E_*, RMLX_REGISTRY_TEST,
  RMLX_NIAH_KV_QUANT, and the *_STRICT flags.
• .env.example expanded to document all user-facing env vars: runtime data
  vars (RMLX_HOME, RMLX_METRICS_DB), all five newly-promoted flag-envs,
  audio path vars, RMLX_MM_CACHE_BYTES, RMLX_SESSION_CACHE_MAX_SESSIONS,
  draft compat keys, and prefill chunk tuning.
• Dependency bumps: safetensors 0.4 → 0.7, symphonia 0.5 → 0.6.

Removed

The following env vars no longer have live readers in the Rust codebase.
This is a breaking change for any shell config that set them directly —
use the replacement flag instead.

Removed variable	Replacement
RMLX_KEEP_ALIVE	--idle-timeout-secs
RMLX_PROMPT_CACHE_MAX_BYTES	--prompt-cache-ram-gb
RMLX_PAGED_KV	--paged-kv
RMLX_KV_PAGE_SIZE	--paged-kv-page-tokens

The following debug / internal vars were dropped with no user-facing
replacement (they had no stable semantics across releases):

• RMLX_SPEC_K — undocumented experimental speculative-lookahead override.
  Its only value was the default; lookahead K is now fixed at 4. The
  independent --draft-block-size flag still controls the draft round size.
• RMLX_MTP_DUMP, RMLX_DFLASH_DEBUG — folded into tracing events; use
  --log debug or RUST_LOG=rmlx=debug instead.
• RMLX_GIT_SHA — was read for the metrics drainer's git_sha annotation but
  nothing ever set it (always None); the annotation now reuses the same
  git rev-parse helper the run ID uses, so it is populated in a git checkout.
• RMLX_METAL_AVAILABLE, RMLX_METAL_CAPTURE — doc-only, never implemented.
• RMLX_METRICS_LOCK — doc-only, never implemented (WAL handles concurrency).
• RMLX_GPU_RESIDENT_ISO, RMLX_SPARSE_V_KERNEL, RMLX_SPARSE_V_THRESHOLD —
  deep perf/kernel toggles, now hardcoded to their proven-best defaults
  (off, on, 1e-6); the override env was removed (no perf change).
• RMLX_OMODELS_DIR — bench-script alias renamed to the canonical
  RMLX_O_MODELS_ROOT.