v0.9.0 — OpenAI Codex support, image routing, and the first release since 0.7.0

The first release since 0.7.0, and it contains breaking changes. Nothing between 0.7.0 and this was ever published, so upgrading from 0.7.0 lands all of 0.8.0, 0.8.1, and 0.8.2 at once — see those sections below for the full detail. This section is the upgrade guide: what breaks, and what's new enough to matter.

Versioned 0.9.0 rather than 0.8.2 deliberately: a 0.7.0 → 0.8.2 jump reads like "two patches on a release I already have," which would invite a casual deploy straight into removed env vars and a retired header. Under 0.x SemVer, MINOR is where breaking changes belong.

⚠️ Breaking — read before upgrading from 0.7.0

• Legacy single-server MLX env vars are gone. FLEET_NODE_MLX_AUTO_START, FLEET_NODE_MLX_AUTO_START_MODEL, FLEET_NODE_MLX_URL, FLEET_NODE_MLX_KV_BITS, FLEET_NODE_MLX_PROMPT_CACHE_SIZE/BYTES, FLEET_NODE_MLX_DRAFT_MODEL, FLEET_NODE_MLX_NUM_DRAFT_TOKENS. Migrate to a one-entry FLEET_NODE_MLX_SERVERS array. (FLEET_MLX_ENABLED and the server-side FLEET_MLX_URL fallback remain; the herd/herd-node CLI, FLEET_* internals and ~/.fleet-manager/ are unchanged.)
• X-Fleet-Model is retired in favour of X-Fleet-Served-Model. X-Fleet-Fallback also changed meaning: it is now an always-present "true"/"false" boolean, not a model name emitted only on substitution.
• Queue-full is now 429, not 503. Ollama's "maximum pending requests exceeded" is no longer retried (retrying a saturated node amplified the flood).
• POST /fleet/pin can now refuse. 409 when the pinned set can't physically co-reside ("force": true overrides), 400 for an unknown node_id. Previously every pin was accepted silently — which is how a 307 GB pinned set on a 512 GB box produced hours of thrash.
• Image requests can now fail instead of silently succeeding. A request carrying images will no longer fall back to a model that can't see them; it fails loudly rather than returning a confident answer about an image the model never received.
• Backend client errors surface as themselves. A 4xx from Ollama (e.g. "model does not support tools") now reaches you as that 4xx with the backend's message, instead of an opaque 500.
• The built-in default anthropic_model_map is now empty. It previously hard-coded qwen3-coder:30b / qwen3:32b / qwen3:14b — model names a given deployment may never have pulled. Deployments that relied on that built-in default (never set FLEET_ANTHROPIC_MODEL_MAP) now get auto-routing instead (best loaded model per tier), which is strictly more likely to resolve to something they actually have. Explicit maps set via env are unaffected. See the auto-routing feature below.

Headline features

• OpenAI Codex support — a native Responses API at /v1/responses. Codex removed Chat Completions in Feb 2026 (wire_api = "chat" is gone), so this is the only endpoint current Codex can speak. Agentic coding is verified end-to-end against a real codex-cli 0.145.0-alpha.18: it ran pytest, read sources, created a module from scratch, patched files via apply_patch, and reached green on its own. Zero configuration — a gpt-5-codex/gpt-5.6-sol id auto-routes to the best coding model you have loaded, the same resolver Claude Code uses. See docs/guides/codex-integration.md.

  Getting there meant bridging a gap that only appears with local models: Codex's tool descriptions document an API its tool schema doesn't expose, and local models call what the prose names. Herd now normalises three cases automatically, each logged at WARNING:

  • Tools hidden inside an additional_tools input item (the sol/terra/luna "Responses-Lite" slugs, including the ChatGPT Desktop default) are extracted, and the grammar-constrained custom exec tool is bridged to something Ollama function-calling can express. Without this the model has nothing callable and rationalises the failure — openai/codex#31894.
  • A top-level exec_command call — a nested tool only reachable from inside code-mode JavaScript — is rewritten as a custom_tool_call on its host tool. Passing it through makes Codex stop with no error displayed.
  • An apply_patch tool call is rewritten as an exec_command heredoc. apply_patch is a binary Codex injects on the sandbox PATH, not a tool; calling it as one returns unsupported call: apply_patch, and the session can then read and execute but never write.

• Images route to a model that can see them, on every endpoint. An image-bearing request auto-selects a vision-capable model even when the conversation's model is a code-tuned one, and image content now reaches Ollama as its images list instead of being silently dropped in translation. A dropped image is worse than a dropped tool call: it produces a fluent, specific, wrong answer while every server-side metric reports success.

• Distributed MLX inference — run one model across multiple Macs via mlx.launch (ring over LAN today; jaccl/Thunderbolt 5 targeted). The herd sees one endpoint whether one Mac or four are behind it.

• Fleet control API — GET /fleet/limits, POST /fleet/pin (with wait for readiness), DELETE /fleet/pin/{model}.

• mlx: models reachable over the OpenAI endpoint, not just Anthropic — OpenAI-only clients get the fast backend instead of a slow fallback.

• Canonical X-Fleet-* headers on every proxied response, so a caller can always tell what actually ran.

• Per-request strict mode (X-Fleet-No-Fallback) and a per-client concurrency cap (FLEET_CLIENT_MAX_IN_FLIGHT, default off).

• FLEET_ANTHROPIC_MODEL_MAP is now optional — Claude Code works with zero configuration. A claude-* id with no explicit mapping is resolved to the best currently-loaded local model for its tier (coding models preferred for Claude Code's workload; a loaded vision model chosen automatically for image requests), falling back to the best on-disk model, then a configured default. So a fresh install routes to whatever the user pulled — no hand-written map that has to match your downloads, and no map entry silently pointing at a model you never pulled. Explicit map entries still win as per-alias overrides; set FLEET_ANTHROPIC_AUTO_ROUTE=false to require an explicit map (the pre-0.9 behaviour). See docs/reference/anthropic-auto-routing.md.

Notable fixes

• finish_reason is recorded on every trace, so a turn that ends mid-task is distinguishable from one that exhausted its token budget without eyeballing completion_tokens.
• A num_ctx override that cannot apply now says so, once, instead of logging like it worked. FLEET_NUM_CTX_OVERRIDES sets the context a cold load comes up with; it cannot shrink a resident model without forcing an unload/reload. Previously it logged an "injected" line and a "stripped" line per request — 393 pairs in nine hours — which read as a working feature doing nothing.
• OpenAI function calling was dropped in both directions on /v1/chat/completions; tool calls now survive the round trip.
• /v1/models emits the schema Codex actually decodes. Codex validates against its own undocumented, strictly-typed schema and fails the whole decode on the first problem — visibility: "public" alone (not in its list/hide/none enum) emptied the model picker, which pushes ChatGPT Desktop onto its Lite slugs. supports_vision is now reported per model rather than hardcoded false. The field set is not converged; see docs/issues.md.
• Image requests are never answered by a blind model (the 2026-04-23 incident class — see below).
• Failed-request traces no longer vanish, so the dashboard's success rate stops hiding failures.
• Model sizes come from Ollama's real /api/tags data instead of being guessed from the name — the guess called a 290 GB model "10 GB" and defeated the memory gate.
• Models are now sized by what they actually cost in RAM — weights plus KV cache. Every "will this fit?" decision previously counted on-disk weights and ignored the KV cache, which scales with context and routinely dwarfs the weights: qwen3-coder:30b is 18.6 GB of weights and 122.9 GB resident at its default 262K context, so the gate was under-counting it by 5.4×. The router now learns each model's KV cost per token from heartbeat data it was already receiving ((resident − weights) / context_length) and predicts the real footprint at the context the model will actually run with. The preloader gate, /fleet/pin admission and the scorer all share one estimator. Models the fleet has never observed keep their previous sizing — evidence tightens these gates, guesswork doesn't. See docs/issues/model-sizing-ignores-kv-cache.md.
• Preloading warms a model at the same num_ctx requests will use. It previously warmed at the model's default and let the first real request reload it at the override — pointless churn, and it made the model's cost unknowable at load time.
• The hot-model cap is no longer hardcoded to 3 — nodes report their own, and free_slots follows it.

Recommended alongside this release

Upgrade Ollama to 0.32.1. Not required, but measured on the same hardware: glm-4.7-flash 13.7 → 77.8 tok/s (the glm4moelite expert-offload bug is fixed upstream), gpt-oss:120b 50.9 → 74.5, and prefix caching demonstrably works. See docs/plans/ollama-0.32-upgrade-and-mlx-evaluation.md.

---