ThunderMLX v0.3.6 — data-plane idle ping: flat ~350ms turn TTFT

Turn starts are now instant — at any think-time

TTFT after idle: 4.5–5s → ~350ms flat (measured at 6s, 12s, and 25s think-time gaps in a resident session).

Root cause

Apple's RDMA provider puts the Thunderbolt data-plane queue-pairs to sleep after ~1–2 seconds of pipeline inactivity, and they take ~3 seconds to wake — which landed exactly on interactive think-time between chat turns (prefill measured 2.5× slower on the first post-idle request). No GPU/matmul keepwarm can reach those QPs; only real pipeline traffic does.

Fix

The 1 Hz coordinated keepwarm tick now carries a tiny symmetric send/recv pair through the actual pipeline queue-pairs (rank0 sends first, rank1 receives first — deadlock-free by construction). The flag rides the op payload, so both ranks always agree regardless of env state. Failure policy is die-fast into the proven guard → teardown → auto-relaunch path rather than limping into a desynced collective. Kill switch: MLX_M3_KEEPWARM_DATAPLANE_PING=0.

Also in this release

• Keepwarm cadence retuned for the measured 1–2s decay window (1s interval, 0.5s idle gate, large-cache throttle removed) + request-start warmup enabled
• Gateway: sticky-oMLX grace window (interactive oMLX sessions defer M3 auto-start; capped at 30s by default) with M3-wakeup attribution logging to identify background resurrection culprits
• Launcher refuses mixed-code launches (self-syncs rank1) — prevents a data-plane message off-by-one from version skew
• Keepwarm transaction releases the generation lock only while still the recorded owner (stale-steal safety)
• Dashboard: live prefill tok/s during the prefill phase (was 0 until completion)

ThunderMLX v0.3.5 — append-aware image cache

Screenshot workflows no longer pay a full conversation re-read per capture. When a request's image set strictly extends the cached one, an append verdict verifies from rank-symmetric inputs (consensus-proven descriptor plus per-image sha256 manifest) that every cached image is byte- and layout-identical, then reuses the verified prefix — a new partial-media input mode feeds only the appended images' pixels alongside the suffix tokens. Changed, reordered, or replaced images fall back to the proven cold rebuild; rank-divergent verdicts collapse to a symmetric full prefill via the prefix-plan consensus with no new collectives. Field results on a live 100k+ agent session: captures that previously cost a ~3-minute full re-read now prefill sub-1k-token suffixes in seconds, across consecutive appends on both ranks in lockstep. Also in this line: runtime updated to mlx-vlm 0.6.6 with the validated custom MLX core preserved.

ThunderMLX v0.3.4 — reasoning-recall loop gate

The recall feature re-inserts prior-turn reasoning into the model prompt for KV alignment (agent clients drop think blocks from transcripts). When a contained copy-spiral's reasoning entered that store, greedy decode regenerated it verbatim every turn and each copy was re-stored — a self-sustaining echo producing byte-identical thinking streams while tool actions progressed normally. The output stream itself was audited clean; the defect was input-side priming. Store and recall now both refuse reasoning the repetition detector flags, so a contained loop dies with its turn instead of echoing indefinitely. Healthy reasoning recall and its cache alignment are unchanged.

ThunderMLX v0.3.3 — antidoom loop containment, hardened

The decode repetition-loop guard now actually covers the streaming path: its long band (121-1,200 char units x5 copies) needed more history than the SSE loop retained, so paragraph-scale loops were undetectable exactly where agent traffic lives. The scan window is now 6,000 chars on both paths, the repeating unit must contain a letter in any script (CJK loops visible) with two distinct characters (base64 runs, zero hashes, numeric fillers, and markup walls no longer truncate legitimate output), containment stops record the identical sequence on both ranks (no post-containment divergence rebuild), and the EOS injection rides the validated synchronized-sampler helper. Live-validated: a streamed 306-char paragraph loop contained at exactly 5 copies with clean lockstep and a warm follow-up turn. Caught its first real production loop within hours of shipping.

ThunderMLX v0.3.2 — rank-symmetric prepare path

Closes an entire distributed wedge class: collectives pair by order across ranks, and the SSD-restore coherence gate ran three all_sums only on ranks whose local cache state took the SSD path — any per-rank cache skew could cross-pair mismatched collectives into a zero-progress pipeline stall (reproduced live before the fix). The prepare phase now issues an identical collective sequence on both ranks under any local cache state: restores are rank-local best-effort reconciled by the unconditional prefix-plan consensus, prepare exceptions vote before aborting so the peer aborts in lockstep, the image cache gate keys on a symmetric any-rank-has-image exchange, prewarm cleanup is symmetric, and divergences self-heal by re-aligning per-rank SSD artifacts instead of looping. Enables the admission guard's full eviction ladder, including live-cache release under memory pressure.

ThunderMLX v0.3.1 — prefill admission guard

Prevents the memory-wedge class where a large cold prefill collides with a nearly-full GPU wired ceiling. Before any cold prefill of 8k+ tokens, each rank locally projects the KV working set against its wired limit and, under deficit, trims the Metal freed-buffer pool with full telemetry (wired/limit/deficit/actions). Fail-open by design: never blocks a request, adds zero collectives, and measures no decode or prefill cost when memory is roomy. Validated with offline unit suites plus live cluster tests: no-op at the real ceiling, correct fail-open under an impossible synthetic ceiling, tools 4/4, needle recall, decode at baseline.

ThunderMLX v0.3.0 — lossless high-context decode speedup

Single-pass exact sparse-selection kernel — bit-exact (60/60 parity), identical model outputs, selection cost 18.7ms→4.0ms/token @200k (4.7x). Real agent decode with native tools: 50k 22.8→24.4 (+7%), 80k 21.5→23.9 (+11%), 150k 17.7→22.4 (+26%), 200k 15.6→21.5 (+38%). Chat/prefill/TTFT/cache unchanged by design and measurement. Field-validated in a live 178k-token agent session (7,374-token write at 20.6 tok/s flat, 63 tool calls, zero retries). Enable: MLX_M3_MSA_SELECT_V2=1. Full methodology + honest observations: ops/fable_lab/RESULTS.md.

ThunderMLX v0.2.0

Long-form thinking, without channel loss

ThunderMLX v0.2.0 fixes the remaining MiniMax-M3 Q4 failure where a long standalone artifact could begin inside native reasoning, never emit </mm:think>, and leave clients with reasoning but no final answer.

• Detects only complete, tool-free standalone HTML requests and the model's own explicit planning-to-implementation declaration.
• Injects the native thinking-close token through rank 0's existing sampled-token synchronization, so both pipeline ranks remain in lockstep and generation continues in the content channel.
• Stops at the first complete </body></html> boundary, suppressing a duplicated second document while synchronized EOS propagates.
• Uses MiniMax's published thinking defaults when clients omit sampling: temperature 1.0, top-p 0.95, no repetition penalty.
• Raises the portable default output budget and hard ceiling to 32,768 tokens.

Loop containment

The existing exact-span decode guard is now documented and configurable as an AntiDoom-inspired runtime safeguard. It scans the rolling tail every 12 tokens and ends only confirmed byte-identical semantic loops through synchronized EOS. It does not alter normal logits and is explicitly not upstream AntiDoom FTPO/LoRA training. See docs/ANTIDOOM.md.

Operations and visibility

• Adds advanced dashboard settings for thinking sampling, artifact transition, and loop-scan cadence.
• Adds repeatable exact-prompt and offline transition probes.
• Keeps the native mlx-vlm 0.6.5 tool path, split 38,22, MSA profile, cache policy, networking, and model gateway unchanged.

Qualification results

Reference hardware: M3 Ultra 256GB + M5 Max 128GB over Thunderbolt/JACCL, MiniMax-M3-4bit.

• Exact Shadow Syntax prompt: 2,158 reasoning characters followed by one complete 52,020-character HTML document, 26.43 tok/s decode.
• 200.5K context: 309.14 prompt tok/s cold and 24.52 tok/s cached decode, reusing 200,474/200,518 prompt tokens.
• Short post-restart tool decode: 32.15-32.29 tok/s; thinking began streaming at 0.64s on the hot turn.
• Non-stream standalone artifact: one complete document with reasoning/content separation at 30.68 tok/s.
• True stop/start released wired memory to 4GB/3GB; a 34-tool agent session then restored 20,645/20,682 tokens from SSD.
• Passed OpenAI tool calls, Anthropic Messages, Codex Responses, Claude Code extended loops, real ZCode/OpenCode workflows, images, cancellation, multi-session isolation, hot cache, SSD cache, and OpenWebUI-shaped stress with no new server failures.

Upgrade

git pull origin main
open ./M3_Stop.command
open ./M3_Start.command

Existing private machine configuration remains in .env.local and is not part of the release.

ThunderMLX v0.1.0

First stable ThunderMLX release. This promotes the tested Beta 9 baseline together with subsequent SSD-session preservation, long native-tool completion, catalog cleanup, and native tool-loop recovery improvements.

Future releases follow Semantic Versioning: patch releases for compatible fixes, minor releases for compatible features, and major releases for breaking changes.
Full Changelog: v0.1.0-beta.9...v0.1.0