v0.9.1 — session affinity, two health checks, three fixes
A fixes-and-hardening release on top of 0.9.0 — no breaking changes, no new dependencies. Three reliability bugs found while operating the local fleet, plus a new routing signal and two health checks that came out of the same soak.
Added
- Session affinity — an 8th scoring signal. A conversation is now pinned to the node already holding its warm prefix cache, so turn N+1 reuses the processed prompt (llama.cpp's
get_common_prefixskips already-seen tokens) instead of re-prefilling ~30K tokens on a fresh node — which also removes the decode interference that a cold prefill inflicts on any stream co-resident on the new node. Keyed by an explicitsession:<id>request tag when present, elseclient_ip|model; TTL 900s. The bonus (20) sits below thermal (50) so a hot node never wins on affinity alone. Seedocs/fleet-manager-routing-engine.md§ Signal 8. decode_degradedhealth check (#39). Watches per-model TPOT —(latency − time_to_first_token) / (completion_tokens − 1)— against each model's own rolling baseline, so it isolates genuine decode contention from queueing and prefill. Fires WARNING when decode slows against a node's own history rather than a fleet-wide constant.pin_cannot_fithealth check (#40). Surfaces a pinned model that can never load on its node because the pinned set can't physically co-reside, instead of letting the preloader retry it forever.
Changed
- Queue concurrency is capped at what the backend will actually decode. Per-
node:modelconcurrency is nowmin(memory_slots, OLLAMA_NUM_PARALLEL)— Ollama admits onlynum_parallelrequests into decode at once, so provisioning more queue workers than that just built a second queue in front of the real one. Nodes reportnum_parallelin the heartbeat; the router mirrors it the same way it already mirrors the hot-model cap.
Fixed
FLEET_NODE_NODE_IDnow actually sets the node id.herd-nodepassed its empty--node-iddefault explicitly intoNodeSettings, and an explicit kwarg shadows pydantic-settings' env lookup — so the env var was inert and the node fell back tosocket.gethostname(). On macOS the hostname is network-derived when the staticHostNameis unset, so it silently changed between networks (bbat home →Neons-Mac-Studiotravelling) and orphaned every pin bound to the old id. The same shadowing affectedFLEET_NODE_ROUTER_URL; both are fixed.- Image requests carrying
image/input_imagecontent parts no longer fall back to a blind text model. The image-detection guard only recognisedimage_url; requests using the other two shapes slipped past it, routed togpt-oss, and 400'd — while the guard built to prevent exactly that sat inert. The detector now matches the full superset the runtimes accept and skips non-dict messages. - A stale dashboard tab can no longer wedge the router. The
/dashboard/eventsSSE loop now checksrequest.is_disconnected()each iteration and wraps the stream so an exception is logged and returned rather than escaping, whileCancelledErrorstill propagates — a disconnected tab stops generating work instead of accumulating it on the event loop.