v0.9.2 — anonymous telemetry, routing observability, persistent settings

Added

• Anonymous community telemetry, on by default. The router sends one summary a day: which models ran, request and token counts, latency percentiles, and error counts by category. Never prompts, never completions, never raw error text, never your hostname. Opt out with FLEET_NODE_TELEMETRY=false, or from the dashboard's new Community Telemetry section. A one-time notice is printed the first time herd-node starts, before anything is ever sent, and the opt-out is honoured on that first run — a node that has opted out writes no files at all, not even the identifier. Every field that can be sent is published at ollamaherd.com/telemetry; that page is the contract, and the payload builder mirrors it under tests that fail the build if the two drift.

  Identity is a random install_id UUID stored in ~/.fleet-manager/install_id — delete it and you are a new install, delete it and opt out and you are gone. It is never derived from anything about the machine; tests assert it is neither the hostname in any form nor a hash of it, so it cannot be turned into a stable fingerprint later.

  Reporting is per herd, not per machine. The router is the only component that knows a fleet is one fleet, so it sends one payload containing a devices[] row per node (chip, memory, cores, agent and Ollama version, that node's request share). Fleet totals are derived from those rows server-side rather than sent as scalars, so a total can never disagree with its parts. Each device carries a device_id derived from its node_id but hashed and salted with the herd's install_id: stable enough to aggregate across days, and impossible to reverse to a hostname or correlate across herds.

• Name your herd for the public leaderboard — a second, separate opt-in on top of telemetry. Set it in the dashboard or via FLEET_NODE_HERD_NICKNAME. Telemetry alone is never public; a nickname is the only field that is, and the UI says so before you set one. Names are limited to 30 characters of letters, numbers, spaces, -, _, ., validated where they are typed rather than only in the browser.

• X-Fleet-Affinity makes session routing visible. Every scored endpoint now reports matched when a request went back to the node already holding that conversation, or new when it did not — and omits the header entirely on routes that never score, because "did not apply" and "missed" are different facts.

  It reports the routing decision, not a cache hit, and deliberately so: Ollama folds llama.cpp's cache_n back into prompt_n (ollama/ollama#16428), so prompt_eval_count cannot yield a hit ratio and inferring one from it produced a false "zero prefix-cache reuse" report here once already. Time-to-first-token is the honest proof: a matched turn shows a large drop.

• usage.prompt_tokens_details.cached_tokens is reported on backends that actually measure it (MLX), and omitted — never zeroed — on backends that cannot (Ollama). Zero means "measured, nothing reused"; absent means "cannot measure". Conflating them is a bug vLLM shipped and SGLang still has.

• The Ollama and mlx-lm versions are collected in the heartbeat and surfaced to the router, so a fleet can answer "which runtimes are actually out there?" before a version-gated model or a changed default lands.

Changed

• Session affinity now decays with queue depth. A pinned node contributes the full bonus when idle and progressively less as its queue grows, so a warm but saturated node stops winning while an idle peer sits free. Every production router does this — NVIDIA Dynamo, Ray Serve, SGLang — and a flat bonus is the bug vLLM's production-stack shipped loadaware routing to fix. Signal 3 already prices congestion, so this shrinks the bonus rather than adding a second penalty, and because decay can only shrink, affinity still never outweighs a thermal warning.

• MLX prompt cache raised from 4 to 10, matching mlx_lm.server's own default. The previous value had no recorded rationale and sat below upstream, silently halving how many conversations could keep their KV cache warm — which is the actual limit on how many sessions affinity can honour.

Fixed

• Telemetry now sends a missed day on startup instead of sleeping past it. The scheduler slept until the next 00:05 UTC before its first send, so start time of day decided whether an install reported at all: a router started at 00:10 UTC waited 23h55m, and one restarted daily after 00:05 never sent. Both are indistinguishable from "nobody uses this" on the receiving end. Found because our own fleet ran 12 hours of clean uptime with zero automatic sends.

• The published telemetry opt-out worked in name only. Moving the sender from the node to the router silently repointed which environment variables it read (ServerSettings carries a different prefix), so FLEET_NODE_TELEMETRY=false — the opt-out documented on the website — stopped disabling anything, with no error anywhere, because "off" and "unset" are indistinguishable in a boolean default. Both that and FLEET_NODE_HERD_NICKNAME are now read correctly, with tests pinning the names as a published contract.

• Dashboard toggles now survive a restart. POST /dashboard/api/settings only mutated settings in memory, so every Feature Toggle silently reverted on the next start. They are now written to ~/.fleet-manager/env with a line-based writer that preserves the comments and hand edits in that file, and the response reports not_persisted / restart_required instead of implying an effect it cannot deliver. Merely annoying for auto_pull; it would have been a broken promise for a telemetry opt-out.

• The gotomy.ai platform panel is removed from Settings — that service is shut down.

Removed

• The account-based platform telemetry opt-ins are untouched, but the dashboard's platform-connection UI is gone along with the service it targeted.

v0.9.1 — session affinity, two health checks, three fixes

A fixes-and-hardening release on top of 0.9.0 — no breaking changes, no new dependencies. Three reliability bugs found while operating the local fleet, plus a new routing signal and two health checks that came out of the same soak.

Added

• Session affinity — an 8th scoring signal. A conversation is now pinned to the node already holding its warm prefix cache, so turn N+1 reuses the processed prompt (llama.cpp's get_common_prefix skips already-seen tokens) instead of re-prefilling ~30K tokens on a fresh node — which also removes the decode interference that a cold prefill inflicts on any stream co-resident on the new node. Keyed by an explicit session:<id> request tag when present, else client_ip|model; TTL 900s. The bonus (20) sits below thermal (50) so a hot node never wins on affinity alone. See docs/fleet-manager-routing-engine.md § Signal 8.
• decode_degraded health check (#39). Watches per-model TPOT — (latency − time_to_first_token) / (completion_tokens − 1) — against each model's own rolling baseline, so it isolates genuine decode contention from queueing and prefill. Fires WARNING when decode slows against a node's own history rather than a fleet-wide constant.
• pin_cannot_fit health check (#40). Surfaces a pinned model that can never load on its node because the pinned set can't physically co-reside, instead of letting the preloader retry it forever.

Changed

• Queue concurrency is capped at what the backend will actually decode. Per-node:model concurrency is now min(memory_slots, OLLAMA_NUM_PARALLEL) — Ollama admits only num_parallel requests into decode at once, so provisioning more queue workers than that just built a second queue in front of the real one. Nodes report num_parallel in the heartbeat; the router mirrors it the same way it already mirrors the hot-model cap.

Fixed

• FLEET_NODE_NODE_ID now actually sets the node id. herd-node passed its empty --node-id default explicitly into NodeSettings, and an explicit kwarg shadows pydantic-settings' env lookup — so the env var was inert and the node fell back to socket.gethostname(). On macOS the hostname is network-derived when the static HostName is unset, so it silently changed between networks (bb at home → Neons-Mac-Studio travelling) and orphaned every pin bound to the old id. The same shadowing affected FLEET_NODE_ROUTER_URL; both are fixed.
• Image requests carrying image / input_image content parts no longer fall back to a blind text model. The image-detection guard only recognised image_url; requests using the other two shapes slipped past it, routed to gpt-oss, and 400'd — while the guard built to prevent exactly that sat inert. The detector now matches the full superset the runtimes accept and skips non-dict messages.
• A stale dashboard tab can no longer wedge the router. The /dashboard/events SSE loop now checks request.is_disconnected() each iteration and wraps the stream so an exception is logged and returned rather than escaping, while CancelledError still propagates — a disconnected tab stops generating work instead of accumulating it on the event loop.

v0.9.0 — OpenAI Codex support, image routing, and the first release since 0.7.0

The first release since 0.7.0, and it contains breaking changes. Nothing between 0.7.0 and this was ever published, so upgrading from 0.7.0 lands all of 0.8.0, 0.8.1, and 0.8.2 at once — see those sections below for the full detail. This section is the upgrade guide: what breaks, and what's new enough to matter.

Versioned 0.9.0 rather than 0.8.2 deliberately: a 0.7.0 → 0.8.2 jump reads like "two patches on a release I already have," which would invite a casual deploy straight into removed env vars and a retired header. Under 0.x SemVer, MINOR is where breaking changes belong.

⚠️ Breaking — read before upgrading from 0.7.0

• Legacy single-server MLX env vars are gone. FLEET_NODE_MLX_AUTO_START, FLEET_NODE_MLX_AUTO_START_MODEL, FLEET_NODE_MLX_URL, FLEET_NODE_MLX_KV_BITS, FLEET_NODE_MLX_PROMPT_CACHE_SIZE/BYTES, FLEET_NODE_MLX_DRAFT_MODEL, FLEET_NODE_MLX_NUM_DRAFT_TOKENS. Migrate to a one-entry FLEET_NODE_MLX_SERVERS array. (FLEET_MLX_ENABLED and the server-side FLEET_MLX_URL fallback remain; the herd/herd-node CLI, FLEET_* internals and ~/.fleet-manager/ are unchanged.)
• X-Fleet-Model is retired in favour of X-Fleet-Served-Model. X-Fleet-Fallback also changed meaning: it is now an always-present "true"/"false" boolean, not a model name emitted only on substitution.
• Queue-full is now 429, not 503. Ollama's "maximum pending requests exceeded" is no longer retried (retrying a saturated node amplified the flood).
• POST /fleet/pin can now refuse. 409 when the pinned set can't physically co-reside ("force": true overrides), 400 for an unknown node_id. Previously every pin was accepted silently — which is how a 307 GB pinned set on a 512 GB box produced hours of thrash.
• Image requests can now fail instead of silently succeeding. A request carrying images will no longer fall back to a model that can't see them; it fails loudly rather than returning a confident answer about an image the model never received.
• Backend client errors surface as themselves. A 4xx from Ollama (e.g. "model does not support tools") now reaches you as that 4xx with the backend's message, instead of an opaque 500.
• The built-in default anthropic_model_map is now empty. It previously hard-coded qwen3-coder:30b / qwen3:32b / qwen3:14b — model names a given deployment may never have pulled. Deployments that relied on that built-in default (never set FLEET_ANTHROPIC_MODEL_MAP) now get auto-routing instead (best loaded model per tier), which is strictly more likely to resolve to something they actually have. Explicit maps set via env are unaffected. See the auto-routing feature below.

Headline features

• OpenAI Codex support — a native Responses API at /v1/responses. Codex removed Chat Completions in Feb 2026 (wire_api = "chat" is gone), so this is the only endpoint current Codex can speak. Agentic coding is verified end-to-end against a real codex-cli 0.145.0-alpha.18: it ran pytest, read sources, created a module from scratch, patched files via apply_patch, and reached green on its own. Zero configuration — a gpt-5-codex/gpt-5.6-sol id auto-routes to the best coding model you have loaded, the same resolver Claude Code uses. See docs/guides/codex-integration.md.

  Getting there meant bridging a gap that only appears with local models: Codex's tool descriptions document an API its tool schema doesn't expose, and local models call what the prose names. Herd now normalises three cases automatically, each logged at WARNING:

  • Tools hidden inside an additional_tools input item (the sol/terra/luna "Responses-Lite" slugs, including the ChatGPT Desktop default) are extracted, and the grammar-constrained custom exec tool is bridged to something Ollama function-calling can express. Without this the model has nothing callable and rationalises the failure — openai/codex#31894.
  • A top-level exec_command call — a nested tool only reachable from inside code-mode JavaScript — is rewritten as a custom_tool_call on its host tool. Passing it through makes Codex stop with no error displayed.
  • An apply_patch tool call is rewritten as an exec_command heredoc. apply_patch is a binary Codex injects on the sandbox PATH, not a tool; calling it as one returns unsupported call: apply_patch, and the session can then read and execute but never write.

• Images route to a model that can see them, on every endpoint. An image-bearing request auto-selects a vision-capable model even when the conversation's model is a code-tuned one, and image content now reaches Ollama as its images list instead of being silently dropped in translation. A dropped image is worse than a dropped tool call: it produces a fluent, specific, wrong answer while every server-side metric reports success.

• Distributed MLX inference — run one model across multiple Macs via mlx.launch (ring over LAN today; jaccl/Thunderbolt 5 targeted). The herd sees one endpoint whether one Mac or four are behind it.

• Fleet control API — GET /fleet/limits, POST /fleet/pin (with wait for readiness), DELETE /fleet/pin/{model}.

• mlx: models reachable over the OpenAI endpoint, not just Anthropic — OpenAI-only clients get the fast backend instead of a slow fallback.

• Canonical X-Fleet-* headers on every proxied response, so a caller can always tell what actually ran.

• Per-request strict mode (X-Fleet-No-Fallback) and a per-client concurrency cap (FLEET_CLIENT_MAX_IN_FLIGHT, default off).

• FLEET_ANTHROPIC_MODEL_MAP is now optional — Claude Code works with zero configuration. A claude-* id with no explicit mapping is resolved to the best currently-loaded local model for its tier (coding models preferred for Claude Code's workload; a loaded vision model chosen automatically for image requests), falling back to the best on-disk model, then a configured default. So a fresh install routes to whatever the user pulled — no hand-written map that has to match your downloads, and no map entry silently pointing at a model you never pulled. Explicit map entries still win as per-alias overrides; set FLEET_ANTHROPIC_AUTO_ROUTE=false to require an explicit map (the pre-0.9 behaviour). See docs/reference/anthropic-auto-routing.md.

Notable fixes

• finish_reason is recorded on every trace, so a turn that ends mid-task is distinguishable from one that exhausted its token budget without eyeballing completion_tokens.
• A num_ctx override that cannot apply now says so, once, instead of logging like it worked. FLEET_NUM_CTX_OVERRIDES sets the context a cold load comes up with; it cannot shrink a resident model without forcing an unload/reload. Previously it logged an "injected" line and a "stripped" line per request — 393 pairs in nine hours — which read as a working feature doing nothing.
• OpenAI function calling was dropped in both directions on /v1/chat/completions; tool calls now survive the round trip.
• /v1/models emits the schema Codex actually decodes. Codex validates against its own undocumented, strictly-typed schema and fails the whole decode on the first problem — visibility: "public" alone (not in its list/hide/none enum) emptied the model picker, which pushes ChatGPT Desktop onto its Lite slugs. supports_vision is now reported per model rather than hardcoded false. The field set is not converged; see docs/issues.md.
• Image requests are never answered by a blind model (the 2026-04-23 incident class — see below).
• Failed-request traces no longer vanish, so the dashboard's success rate stops hiding failures.
• Model sizes come from Ollama's real /api/tags data instead of being guessed from the name — the guess called a 290 GB model "10 GB" and defeated the memory gate.
• Models are now sized by what they actually cost in RAM — weights plus KV cache. Every "will this fit?" decision previously counted on-disk weights and ignored the KV cache, which scales with context and routinely dwarfs the weights: qwen3-coder:30b is 18.6 GB of weights and 122.9 GB resident at its default 262K context, so the gate was under-counting it by 5.4×. The router now learns each model's KV cost per token from heartbeat data it was already receiving ((resident − weights) / context_length) and predicts the real footprint at the context the model will actually run with. The preloader gate, /fleet/pin admission and the scorer all share one estimator. Models the fleet has never observed keep their previous sizing — evidence tightens these gates, guesswork doesn't. See docs/issues/model-sizing-ignores-kv-cache.md.
• Preloading warms a model at the same num_ctx requests will use. It previously warmed at the model's default and let the first real request reload it at the override — pointless churn, and it made the model's cost unknowable at load time.
• The hot-model cap is no longer hardcoded to 3 — nodes report their own, and free_slots follows it.

Recommended alongside this release

Upgrade Ollama to 0.32.1. Not required, but measured on the same hardware: glm-4.7-flash 13.7 → 77.8 tok/s (the glm4moelite expert-offload bug is fixed upstream), gpt-oss:120b 50.9 → 74.5, and prefix caching demonstrably works. See docs/plans/ollama-0.32-upgrade-and-mlx-evaluation.md.

---

v0.7.0 — Native text embedding server (fastembed), Node Models dashboard

Native text embedding backend and full-fleet Node Models dashboard. The embed timeout incident from June 1 exposed a fundamental contention problem: OLLAMA_NUM_PARALLEL=2 means a running gpt-oss:120b inference holds both Ollama slots, so nomic-embed-text requests queue indefinitely inside Ollama regardless of available hardware (14% CPU, 291 GB free RAM). The structural fix routes text embedding out of Ollama entirely — a dedicated fastembed server on port 11439 handles nomic-embed-text via ONNX Runtime with zero inference slot contention. Verified on the local fleet: 573 embed requests over 24h, avg 792ms, 0.0% error rate, no timeouts. Dashboard now shows cards for every model backend (Ollama, MLX, native fastembed, vision embedding) with live per-model stats, renamed from "Request Queues" to "Node Models" to reflect the expanded scope.

Added

• Native text embedding server (fastembed, port 11439). A dedicated FastAPI server runs nomic-ai/nomic-embed-text-v1.5-Q (130 MB int8, 768 dims, 8192 token context) via ONNX Runtime — no PyTorch, no Ollama. The router intercepts /api/embed calls for nomic-embed-text before they reach Ollama and proxies them to the best available node's text embedding server. fastembed>=0.4.0 added to the existing --extra embedding group (same uv sync --extra embedding command as vision embeddings — one command enables both). Model weights download automatically on first request (~130 MB from HuggingFace) and are cached in ~/.fleet-manager/models/text-embedding/. Zero contention with LLM inference slots. See src/fleet_manager/node/text_embedding_server.py, src/fleet_manager/node/text_embedding_models.py, src/fleet_manager/server/routes/text_embedding_compat.py.

• 4 new health checks (32 → 36 total):

  • embed_error_rate — WARNING at ≥5 embed errors/hour, CRITICAL at ≥25/hour. Closes the observability gap that let 202 embed timeouts accumulate undetected for 1.5h on June 1.
  • text_embedding_backend_missing — WARNING when nomic-embed-text weights are cached on disk but fastembed isn't installed. Fix: uv sync --extra embedding.
  • text_embedding_ollama_bypass — WARNING when nomic-embed-text is available in Ollama but the native server isn't running, meaning embed requests still contend for LLM inference slots. Includes Apple Silicon platform note.
  • nomic_loaded_in_ollama — INFO when the native server is running and handling all embed traffic but nomic-embed-text remains loaded in Ollama's hot set, consuming VRAM and a model slot unnecessarily.

• TextEmbeddingModel and TextEmbeddingMetrics heartbeat fields. Node heartbeats now report text_embedding (available models + cached status), text_embedding_port, and text_embedding_status (backend_available, cached_model_count). The router registry copies these fields to NodeInfo and /fleet/status includes them in node serialization.

• Dashboard "Node Models" — cards for all backends. Renamed from "Request Queues". Now shows a card for every model receiving traffic, not just Ollama-queued models: Ollama (grey badge), MLX (purple), native fastembed (green), and vision embedding (cyan). Cards for instant backends (fastembed, vision) show 24h completed/failed counts and avg latency from a 60s-TTL trace DB cache rather than live queue depth. "DL ON DEMAND" amber chip appears when model weights are not yet cached. Stats counter in the header includes all backends, not just Ollama queues.

Fixed

• Embed timeout visibility and resilience. The /api/embed handler previously had no record_trace() calls — all embed outcomes (success and failure) were invisible to the dashboard and health checks. Added trace recording for all paths. Added a retry loop (1s then 3s backoff) on ReadTimeout before returning 504, matching the retry behavior of the LLM streaming path. The 504 response includes a human-readable hint when the timeout is likely caused by a model download in progress.

• embed_error_rate SQL uses model LIKE '%embed%' (not tags LIKE '%embed%'). Using the tags column caused false positives — the local VOD processing pipeline tags its LLM requests with "embed" as a pipeline stage label. Restricting to model column matches only actual embedding models.

• filelock DEBUG log suppression in fastembed download path. fastembed's HuggingFace download emits ~20 DEBUG lines per file lock during the initial model download. Suppressed with logging.getLogger("filelock").setLevel(logging.WARNING) in the text embedding server.

Changed

• CLAUDE.md updated: architecture table + routes updated for new text embedding modules; current state reflects nomic-embed-text → native fastembed; health count 32→36; test count 986→1006; --extra embedding description updated to mention fastembed; skills/ grep updated.

v0.6.2 — Trace store resilience, WAL hardening, log rotation race fix

Reliability hardening for the trace_store SQLite layer and structured logging — both addressing a real production incident on the local fleet that ran undetected for ~4 days. The headline finding: a long-running read transaction held off WAL checkpoints, the WAL grew to 2.5 GB, and the 5-second busy_timeout couldn't absorb the resulting writer contention. ~40,000 background record_trace tasks failed with database is locked over May 10-15 while requests themselves still succeeded — observability was the only visible casualty (dashboard reqs_24h quietly dropped to 0). Adjacent: the daily log rotation handler raced between herd and herd-node writing to the same file, leaving one day's log growing for the entire incident window. Both are fixed; both now have health checks or architectural separation to prevent silent recurrence. Initial fix (busy_timeout + retry + autocheckpoint, committed 2026-05-15) reduced failure amplitude but didn't eliminate it — see the follow-up Part C + A fix below, which addresses the structural cause and was verified clean under live load on 2026-05-16.

Fixed

• TraceStore and LatencyStore now use a dedicated read connection (Part C in docs/plans/trace-store-read-connection-and-checkpoint.md). Each store opens two aiosqlite connections: _db for writes, _read_db for every dashboard analytics + scoring read path. Two purposes — (1) aiosqlite serializes operations per-connection through one background thread, so a slow read on the shared connection blocks queued writes for the read's duration; with separate connections they run concurrently in separate threads. (2) Read snapshots pin the WAL checkpoint barrier; on a separate connection the writer's view of the WAL can advance independently. Combined effect verified on the local fleet 2026-05-16 — a 30-concurrent-write + 120-dashboard-poll burst held the WAL at 410 KB peak vs 103 MB on the same workload before this change. PRAGMA query_only=1 on the read connection rejects accidental writes immediately rather than silently competing with the writer. 7 new tests cover routing (reads → _read_db, writes → _db), query-only enforcement, and connection lifecycle.

• Explicit periodic PRAGMA wal_checkpoint(PASSIVE) every 10 seconds on each store's writer connection (Part A in same plan). Defense-in-depth on top of wal_autocheckpoint=100 — autocheckpoint is tied to write volume, so under bursty traffic the WAL can sit at 99 pages for a long time while readers accumulate snapshots; by the time the 100th page write fires autocheckpoint, those snapshots have pinned the checkpoint barrier so far back that very little can advance. Tying checkpoints to wall-clock makes them fire in the gaps between reader snapshots rather than only when a write lands on a threshold. PASSIVE is non-blocking, so this is safe to run on a tight cadence. Logged at DEBUG every tick; promotes to INFO if a tick comes back contested with >100 unwritten pages — surfaces sustained contention without flooding the log under healthy operation. 3 new tests cover the return shape, closed-connection safety, and error-swallow behavior (must never crash the background task).

• TraceStore and LatencyStore SQLite write resilience. PRAGMA busy_timeout bumped from 5s → 30s in both stores so a transient WAL checkpoint stall can't immediately fail writes. TraceStore.record_trace now retries on database is locked errors with exponential backoff (200ms → 800ms → 2s, 3 attempts) before giving up — so the busy_timeout absorbs short contention and the retry loop covers longer stalls, for ~90s cumulative patience before a trace is declared lost. Added PRAGMA wal_autocheckpoint=100 to both stores to bound WAL growth even when an external reader is slow. Failures after all retries are exhausted are counted in a rolling deque so the new health check (below) can surface them without operators having to grep logs. See the 2026-05-15 observation in docs/observations.md for the full incident timeline.

• Daily log rotation race between herd and herd-node. Both processes previously called setup_logging with the same default file path (~/.fleet-manager/logs/herd.jsonl) and registered their own TimedRotatingFileHandler. At UTC midnight, one process would rename herd.jsonl → herd.jsonl.YYYY-MM-DD and the other would keep writing to the renamed inode via its still-open file descriptor for the rest of the file's life. Observed in the wild: a single day's log grew to 131 MB while peer days were 6 MB; the orphaned file kept receiving writes for five days after its supposed rotation. Fix: setup_logging(log_name=...) is now parameterized; router uses herd (default, back-compat) and node uses herd-node. Each process owns its rotation. Cross-file audits (grep -c '"level": "ERROR"' ~/.fleet-manager/logs/herd*.jsonl*) still work via glob.

Added

• trace_store_write_failures health check (now 32 distinct checks). Reads TraceStore.get_write_failure_count(window_s=300) and emits a WARNING at 1+ failures in the last 5 minutes, CRITICAL at 50+. Closes the observability black hole that hid the May 10-15 incident — the only visible signal before this was dashboard reqs_24h=0 for a router that was clearly serving traffic, which is easy to dismiss as "nobody's running anything right now." 10 new tests in tests/test_server/test_trace_store_resilience.py cover retry-on-locked-then-succeed, retry-exhaustion-then-record-failure, non-lock-errors-don't-retry, window-pruning semantics, and the severity threshold transitions.

Changed

• CLAUDE.md "Gotchas" — two new entries to keep future operators (and AI agents reviewing logs) from repeating the failure modes that delayed detection of this incident. (1) JSONL log scans must use '"level": "ERROR"' with a space after the colon — json.dumps writes whitespace by default and the no-space variant silently matches zero lines. A wrong grep pattern was the root cause for ~4 days of "clean fleet" soak reports during the incident. (2) Trace DB write failures are invisible from the dashboard because record_trace is fire-and-forget — explicit pointer to the new health check + the three most common root causes (long-running read, disk-full, stale db-shm/-wal).

v0.6.1 — MLX supervisor hardening, speculative decoding, vision embedding honesty

Reliability + observability hardening on top of 0.6.0's multi-MLX foundation. The big wins: an MLX supervisor that can no longer get stuck restarting an orphan-port forever (orphan reap + crash-window quarantine), vision embedding chips that stop lying when the backend isn't installed, a brew install ollama-herd path that actually works, a tunable FLEET_MLX_MAX_INFLIGHT_PER_MODEL for operators willing to trade memory for batched throughput, dashboard color semantics that finally agree with the product's "idle hardware is waste" thesis, and speculative decoding live on the dedicated 30B compactor for an immediate Claude-Code-summarization-pass speedup. Source-read research in docs/research/mlx-lm-stability-and-concurrency.md confirmed v0.31.3 is the right pin; v0.31.2 would have traded our quarantine-able bug for two uncontainable ones.

Added

• Speculative decoding enabled on the dedicated context compactor (port 11441, Qwen3-Coder-30B-A3B-Instruct-4bit). Re-tested on our pinned mlx-lm==0.31.3 and confirmed it works on standard transformer MoEs with a Qwen3-1.7B-4bit draft (≈94 tok/s on M3 Ultra, no ArraysCache error). Live config in ~/.fleet-manager/env adds "draft_model":"mlx-community/Qwen3-1.7B-4bit","num_draft_tokens":4 to the 30B-A3B server entry only — every Claude Code request's pre-summarization pass benefits, which is the hot path. Not enabled on port 11440 (Qwen3-Coder-Next-4bit, the main coding model): the Qwen3-Next architecture uses Mamba/SSM-style linear attention layers that mlx-lm represents with a non-trimmable ArraysCache, so spec decoding still hits ml-explore/mlx-lm#1081 at speculative_generate_step:531. The earlier "always blocked on ArraysCache" framing was wrong — the bug is architecture-specific, not version-specific. Updated post-mortem in docs/issues/mlx-speculative-decoding-blocked.md. Per-spec draft_model + num_draft_tokens were already plumbed through MlxServerSpec.from_dict in 0.6.0 — this release is config-only on the production fleet.

Fixed

• FLEET_MLX_MAX_INFLIGHT_PER_MODEL env var — tunable per-model concurrent-request cap on the MLX proxy. Default 1 (strict serialization, matching historical behavior). Bump to 2 or 3 to let mlx_lm.server's BatchGenerator process multiple requests in one inference pass — empirically validated 2026-04-27 on the local fleet (3 concurrent requests took 0.55s wall vs 1.10s sum, confirming real parallelism). The default stays conservative because each in-flight request carries its own KV cache state, so 2× concurrent 100K-token prefills is 2× memory pressure; concurrent paths in mlx_lm.server have historically been bug magnets (#965, #1166 — both fixed in v0.31.3, but the pattern is real). Operator opt-in only. Full source-read + live test in docs/research/mlx-lm-stability-and-concurrency.md. 5 new tests cover defaults, clamping (negative/zero values floor to 1), the 2-concurrent-acquires-then-third-blocks behavior, and per-model semaphore independence. Confirmed in the same research that mlx-lm v0.31.3 is the right pin even though it's where we hit #1208 — downgrading to v0.31.2 would trade our quarantine-able bug for two uncontainable ones (#1166 Qwen3-Next concurrent crash + #1181 thread-local stream crash).

• MLX supervisor now detects and SIGKILLs orphan mlx_lm.server processes on the configured port before spawning its own. A previous herd-node session killed via pkill -9 -f "bin/herd-node" (without also killing mlx_lm.server) leaves the MLX subprocesses alive — Popen(start_new_session=True) makes them survive their parent's death; they get reparented to launchd and keep holding ports 11440 / 11441. The next supervisor startup couldn't bind, exited rc=1 in ~2 seconds, and the quarantine guard kicked in against a process that didn't exist while the orphan kept serving requests. Result was 17 hours of mlx_server_quarantined warnings against a fleet that was actually serving traffic, just not under any supervisor's control. New find_orphan_mlx_pids_on_port() (psutil-based, identity-strict — only kills processes whose cmdline mentions mlx_lm.server AND whose net_connections show binding to our port). MlxSupervisor.start() calls it before its own Popen and SIGKILLs anything found, with a loud WARNING explaining what was reaped. 8 new tests for the strict identity check (right cmdline + right port = kill; wrong port = leave alone; non-mlx process on right port = leave alone; permission errors handled). Operator restart recipe in CLAUDE.md updated to pkill -9 -f "bin/herd|mlx_lm.server" so the manual path doesn't reproduce the trap. See observation in docs/observations.md 2026-04-27.

• MLX supervisor now quarantines a crash-looping subprocess instead of restarting it forever. On 2026-04-26 a stuck-state in mlx_lm.server v0.31.3 (latest) triggered a 420-restart, 2.5-hour crash loop on the local fleet — the supervisor's _monitor correctly detected each crash and restarted at the (capped) 60s exponential-backoff cadence, but had no upper bound on how many times it would keep doing that. Now: ≥5 crashes within a 5-minute rolling window switches the supervisor into "quarantined" state with a 10-minute restart interval, and surfaces a CRITICAL mlx_server_quarantined health-check recommendation pointing at logs + likely causes. Quarantine clears automatically once a restart stays up for the full window — so transient bursts still recover gracefully; only persistent failures get throttled. Tunable via _QUARANTINE_FAILURE_COUNT / _QUARANTINE_WINDOW_S / _QUARANTINE_RESTART_INTERVAL in mlx_supervisor.py. 8 new tests for the threshold + windowing logic. Underlying mlx-lm bug filed upstream as ml-explore/mlx-lm#1208. See observation in docs/observations.md 2026-04-26.

• Vision embedding chips no longer lie about availability when onnxruntime is missing. The dashboard previously rendered DINOv2 / SigLIP / CLIP chips on a node card whenever the model weights were cached on disk, regardless of whether the embedding backend (onnxruntime) could actually load them. Operators saw "available" chips and assumed the service worked; the first real /embed call returned HTTP 500. The collector now probes for onnxruntime import on every heartbeat and refuses to advertise vision-embedding models when the backend isn't loadable, so the chips disappear (honest) instead of misleading. Paired with a new vision_backend_missing health check (WARNING) that fires when weights ARE cached but the backend is missing — operators see "Vision embedding backend not installed on <node>. Run uv sync --extra embedding ..." in the Recommendations panel instead of silently-disappearing chips. Also closes the recurring root cause: the local-deploy snippet in CLAUDE.md was uv sync (without --extra embedding), which is destructive — every restart stripped the embedding deps. Updated to uv sync --all-extras so optional capabilities stay resident across restarts. New vision_embedding_status: dict field on heartbeat + NodeState carries {backend_available, cached_model_count} so future health checks have a clean signal to read. 9 new tests in tests/test_server/test_health_vision_backend.py covering both fires-on-missing and silent-when-fine paths plus the collector probe directly.

• brew install ollama-herd now actually works. The Homebrew formula at geeks-accelerator/homebrew-ollama-herd had been broken throughout 0.5.x — Homebrew's pip install --no-binary :all: policy forced source builds for pydantic-core, which required Rust to bootstrap maturin, which the formula didn't depend on; six pyproject.toml deps (cryptography, cffi, pycparser, tiktoken, regex, websockets) were also missing from the formula's resource blocks; and pydantic-core was version-mismatched against the bundled pydantic (2.45.0 vs the required 2.41.5). Fix shipped to the tap as geeks-accelerator/homebrew-ollama-herd@71856f3 (no PyPI republish needed). Verified end-to-end on macOS Apple Silicon: clean fresh-user install in ~5 minutes, all critical imports clean, both herd and herd-node CLIs functional. The release checklist in CLAUDE.md was updated to make the brew end-to-end install test a non-negotiable gate so this class of failure can't recur. Background and post-mortem in docs/observations.md (entry: 2026-04-25).

Added

• Platform-aware thermal signal — new ThermalMetrics on the heartbeat with state (nominal / warning / unknown), temperature_c, and source fields. Linux nodes now report real peak temps from psutil.sensors_temperatures() (scanning coretemp / k10temp / zenpower / cpu_thermal drivers) and flag warning above 85°C. macOS and Windows honestly report unknown — Apple Silicon's machdep.xcpm is Intel-only, powermetrics requires sudo, and pmset -g therm only reports past events; psutil.sensors_temperatures() isn't implemented on macOS at all. The dashboard's .bar-thermal overlay now uses the reported signal when available and falls back to the CPU≥95% proxy only when state is unknown, so Linux operators get first-class thermal detection and macOS operators keep the existing behavior with a clean seam for future upgrades. See src/fleet_manager/common/system_metrics.py::get_thermal_metrics.
• /dashboard/color-states dev route — renders the utilization bar in every Axis B warning state side-by-side (normal / memory warning / memory critical / CPU thermal) plus gradient sweeps for CPU (cyan→purple) and memory (soft-blue→deep-purple). Uses the live dashboard CSS + JS so any change to production styling reflects here automatically. Linked from docs/guides/dashboard-color-reference.md and used by the marketin...

v0.6.0 — Multi-MLX servers, Claude Code reliability, layered context management

Multi-MLX, Claude Code reliability, and layered context management. The long-context failure modes that made Claude Code CLI feel broken around 30K tokens on local Qwen3-Coder models are systematically addressed: tool-schema fixup for the llama.cpp#20164 optional-param trap, mechanical tool-result clearing with stable-cut prefix-cache preservation, LLM-based compactor with force_all escape, pre-inference 413 cap, MLX wall-clock timeout. Multi-MLX-server support lets a single node run main + dedicated-compactor models side-by-side without Ollama eviction risk. Tool-use reliability layer repairs malformed JSON tool-call arguments. Ollama watchdog removed after it caused production incidents.

Added

• Multi-MLX-server support — the node agent can now spawn N mlx_lm.server subprocesses on N ports simultaneously, with per-server memory-pressure gate, per-URL health reporting, and multi-node aggregation. Closes docs/issues/multi-mlx-server-support.md. Config shape: FLEET_NODE_MLX_SERVERS='[{"model":"mlx-community/Qwen3-Coder-Next-4bit","port":11440,"kv_bits":8},{"model":"mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit","port":11441,"kv_bits":8}]'. Legacy single-server config (FLEET_NODE_MLX_AUTO_START_MODEL + FLEET_NODE_MLX_URL) is synthesized into a one-entry list when the new var is unset — no breaking change.
  • MlxSupervisorSet (node/mlx_supervisor.py) — owns N child MlxSupervisor instances, parallel start/stop, one failure doesn't block the others, per-child status snapshots (healthy/starting/unhealthy/memory_blocked/stopped) published in the heartbeat.
  • Memory-pressure startup gate (memory_gate_ok(), estimate_model_size_gb()) — before spawning each server, estimates weight size from the HuggingFace disk cache and compares to psutil.virtual_memory().available. Refuses to start when the total (model + headroom) won't fit. FLEET_NODE_MLX_MEMORY_HEADROOM_GB default 10 GB. Surfaces the skip reason in the heartbeat so the operator sees WHY on the dashboard, not just that the server is down.
  • Multi-node aggregation — FLEET_NODE_MLX_BIND_HOST=0.0.0.0 exposes MLX servers on the LAN. Heartbeat carries per-server {port, model, status, model_size_gb, kv_bits, last_ok_ts}; the router's NodeRegistry.resolve_mlx_url(model) walks every online node and returns the LAN URL of whichever healthy server hosts the model. MlxProxy now takes an optional url_resolver callable with a per-URL httpx.AsyncClient cache, so a slow server's connection pool can't back-pressure into a fast one. Back-compat: legacy base_url positional still works.
  • Dashboard per-URL health table — each node card renders a compact MLX servers table showing port, short model name, colour-coded status, size in GB, and time-since-last-healthy. Drops in below the model chip row; absent on nodes without MLX configured.
  • Two new health checks: mlx_memory_blocked (WARNING when a server skipped start due to memory gate) and mlx_server_down (CRITICAL when a server that should be healthy has failed; WARNING when stuck in starting). Fix hints point at the three most common causes: missing weights, wiped --kv-bits patch, port collision from a leftover subprocess.
  • Context compactor to a dedicated MLX server — enables FLEET_CONTEXT_COMPACTION_ENABLED=true with FLEET_CONTEXT_COMPACTION_MODEL=mlx:mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit so summarization runs on an 80B-class MoE (3B active) model without competing for the main coding model's MLX process. Shipped side-by-side on the Mac Studio: Next-4bit @ 41.8 GB on port 11440 + 30B-A3B @ 16 GB on port 11441, both hot, ~260 GB RAM headroom remaining.
  • 45 new tests: MlxServerSpec.from_dict validation, memory gate accept/reject/unknown-size paths, HF cache walk, MlxSupervisorSet parallel orchestration + one-failure isolation + duplicate-port dedup + healthy-models filter, registry resolution (single node, multi-node, bare/prefixed, offline/unhealthy skip, full-map aggregation), MlxProxy resolver priority + exception fallback + per-URL client cache isolation + unresolvable-URL error, and two per-server health check emitters.

Fixed

• Per-port MLX log files — mlx_lm.server subprocesses now write to ~/.fleet-manager/logs/mlx-server-<port>.log (e.g. mlx-server-11440.log) instead of sharing a single mlx-server.log. Uncovered during post-deploy investigation of a timed-out request: with both MLX subprocesses appending to the same log file, per-server crash diagnosis was effectively impossible. Single-server deploys get a stable mlx-server-11440.log path; the old shared mlx-server.log is left untouched for archival value but no longer written to. Also updated the module docstring path reference so tail -f ~/.fleet-manager/logs/mlx-server-*.log does the right thing for any deploy shape.

• FLEET_MLX_WALL_CLOCK_TIMEOUT_S guidance — default stays at 300s (reasonable for most workloads), but the configuration reference now explicitly calls out that long Claude Code sessions (2000+ messages) on Qwen3-Coder-Next-4bit routinely run 200-245s and need 600 to avoid edge-case 300.5s-type timeouts. No default change; the existing env var has always been tunable — just making the tuning knob discoverable for the workload that most benefits from it.

• Four field-survey-driven Claude Code CLI enhancements (P1–P4). Landed after a competitive audit of 13+ open-source Claude Code proxies (musistudio/claude-code-router 32.8k⭐, nicedreamzapp/claude-code-local, and others). Full research in docs/research/claude-code-proxy-techniques-survey.md; priority matrix + rationale in docs/plans/claude-code-enhancements-from-field-survey.md.

  • P1 — Expanded JSON repair patterns. tool_call_repair.py now falls through to a four-pattern regex catalog (Pattern A: parameter=key>value, Pattern B: <parameter_key>value, Pattern C: malformed "arguments" objects, Pattern D: single-arg tool inference via an 8-entry defaults table for Bash/Read/Write/Glob/Grep/WebFetch/WebSearch/TodoWrite) when json-repair produces schema-invalid output. Adapted from nicedreamzapp/claude-code-local's recover_garbled_tool_json. Still schema-gated — no repair substitutes unless it passes _structurally_valid_against_schema. 10 new tests covering each pattern + end-to-end single-arg inference. The repair cascade is now strict-parse → json-repair → regex-patterns → pass-through original.
  • P2 — FLEET_ANTHROPIC_TOOLS_DENY. Comma-separated list of Claude Code tool names to strip from every /v1/messages request before translation (e.g. "WebSearch,WebFetch,NotebookEdit"). Saves 200–600 prompt tokens per turn depending on which tools get removed. Pairs with client-side permissions.deny in .claude/settings.json — client-side only blocks execution, this removes the definitions from the wire entirely. Names matched exactly (case-sensitive). 7 new tests covering empty deny, single/multi strip, whitespace tolerance, exact-not-substring, strip-everything.
  • P3 — Size-based model escalation. FLEET_ANTHROPIC_SIZE_ESCALATION_TOKENS + FLEET_ANTHROPIC_SIZE_ESCALATION_MODEL let operators auto-route prompts over N tokens to a different (larger) model without changing the normal FLEET_ANTHROPIC_MODEL_MAP. Example: map Sonnet to qwen3-coder:30b for fast turns, escalate to mlx:Qwen3-Coder-Next-4bit above 50K tokens. Trades small-request throughput for large-request quality where it matters. Pre-route token count uses the same _total_tokens() used by context management so it's consistent with clearing/compaction decisions.
  • P4 — Warm-prompt preload on MLX supervisor start. After mlx_lm.server passes its health check, MlxSupervisor fires a fire-and-forget 1-token request to prime the prompt cache with the system prompt prefix. Based on waybarrios/vllm-mlx's reported 1.3–2.25× TTFT improvement on the first real request. Non-fatal on failure (DEBUG log). Confirmed live after deploy — mlx_lm.server warmup complete — prompt cache primed appears ~15s after supervisor start.
  • P6 — Documentation. New "Stability techniques for long-context local sessions" section in docs/guides/claude-code-integration.md covering permissions.deny pairing, the 80/20 token-range rule, fresh-session cadence, and the new size-escalation knobs. Three new env vars documented in docs/configuration-reference.md.

• Per-tier model routing: claude-haiku-* → gpt-oss:120b (Ollama); claude-sonnet-* / claude-opus-* → mlx:Qwen3-Coder-Next-4bit. FLEET_ANTHROPIC_MODEL_MAP edit only — zero code change. Lets Claude Code users trade speed for quality per-invocation (claude --model claude-haiku-4-5). Haiku goes to a smaller hot Ollama model (fast, already pinned) while heavier tiers stay on the 80B MoE. Different model families on different tiers also diversifies failure modes — if Qwen3 has a bad day, haiku still works. See docs/plans/claude-code-performance-improvements.md §#4.

• Tool-call JSON repair: server/tool_call_repair.py + metrics. Local coding models occasionally emit tool_use.input with minor syntax errors (trailing commas, unescaped quotes, missing brackets). Without repair, Claude Code's strict SDK parser rejects these and the session errors. New module uses the json-repair library (added to core deps, pure Python ~100KB) to attempt recovery, validates the repaired dict against the tool's input_schema, and only substitutes the repaired version if it passes structural check. Never hides real failures silently — every repair attempt logs at WARNING, and `tool_repair: {attempts, succes...

v0.5.2 — SSE watchdog, Fleet Intelligence bug fixes, connection failure tracking

Highlights

• SSE watchdog — dashboard auto-reconnects after 10s of silence, preventing stale node status after network drops
• Connection failure tracking — node agent tracks connection failures, health engine surfaces them, Fleet Intelligence includes them in briefings
• Fleet Intelligence enriched — per-model traffic breakdown, per-node disk space, all health warnings, previous briefing continuity
• 2 bug fixes — report.score AttributeError and avg_latency_ms KeyError in Fleet Intelligence prompt (briefings were silently failing)
• 17 health checks total

Fixes

• Dashboard header stats no longer go stale (innerHTML race condition)
• Dashboard SSE connection established before footer DOM exists
• Fleet Intelligence prompt now correctly accesses report.vitals.health_score and avg_ttft_ms

See CHANGELOG.md for full details.

Install: pip install ollama-herd==0.5.2

v0.5.0 — Smart benchmarks, dynamic context, Fleet Intelligence, 8-tab dashboard

Highlights

• Smart benchmark system — auto-discovers fleet, fills available memory with recommended models, benchmarks LLM chat, embeddings, and image generation simultaneously
• Dynamic num_ctx management — measures actual token usage, auto-adjusts context windows to free KV cache memory
• Fleet Intelligence — LLM-powered briefings that analyze fleet health using the fleet's own models
• /api/pull endpoint — pull models through the router, auto-selects best node by available memory
• Dashboard overhaul — gradient progress bars, animated health ring, model badges by type, in-place SSE updates
• 16 health checks total

Fixes

• Context recommendation uses total tokens (prompt+completion) instead of prompt-only
• Node card DOM flashing eliminated with in-place SSE updates
• Fleet Intelligence prompt restricted to real commands only

See CHANGELOG.md for full details.

Install: pip install ollama-herd==0.5.0

v0.4.1 — Thinking model support, queue depth API, 15 health checks

Highlights

• Thinking model support — auto-detects DeepSeek-R1, QwQ, phi-4-reasoning and inflates token budgets to prevent empty responses
• Queue depth API — GET /fleet/queue for client-side backoff decisions
• KV cache bloat detection — health check detects when OLLAMA_NUM_PARALLEL is too high
• Stream reliability checks — client disconnect and incomplete stream tracking
• 15 health checks total

Fixes

• Embeddings proxy was incorrectly routing through chat pipeline
• shutil.which() couldn't find mflux/DiffusionKit installed via uv tool
• Client disconnects and incomplete streams now tracked correctly

See CHANGELOG.md for full details.

Install: pip install ollama-herd==0.4.1