You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Also in this release
Behaviour change: the admission gate is on by default (#81). Every
engine-backed backend now gets an auto-tuned admission gate (migration 0059
adds the auto / explicit / off modes). The gate is replica-aware: it divides
its limit across --admission-replicas / FASTLLM_ADMISSION_REPLICAS
(default 2). It halves at most once per 10s cooldown, and is restored when
the engine goes silent or stops being detected. Tuning fields apply only in
explicit mode.
Spill on high time-to-first-token (#80). A new max_ttft_ms_per_backend setting moves traffic to another target when the
local backend's TTFT is above the limit. It reads TTFT from the engine's
histogram and falls back to the proxy's own measurement. The console's
FIRST TARGET picker now offers model pools, not only provider models.
The stall detector no longer ejects healthy backends under load (#79).
It now judges liveness from the engine's step counter and the exact KV-cache
gauge.
Changed
Behaviour change: Kubernetes Service discovery is opt-in. The node agent
used to turn every LoadBalancer Service into candidates and send each a
plain-HTTP GET /models every cycle, which filled the logs of unrelated
TLS services with handshake errors. A Service is now a candidate only with fastllm.io/advertise or fastllm.io/discover: "true". LoadBalancer
engines that were auto-discovered must now be annotated fastllm.io/discover: "true" or they stop registering. Pod discovery
(hostPort, host network) is unchanged.
Fixed
A rebooted host comes back into its pools. Discovery drops a host's
attachment while the host is down and creates a new one when it returns.
Before this fix, a pool member that named that attachment was deleted
along with it and never recreated. After one reboot, a pool balancing
across two Sparks used only one, and so did every frontend model routing
through that pool. Now the member is bound to its provider's name
(migration 0058). It waits while the host is down, routing nowhere and not
widening onto other providers, and re-attaches once the host serves the
model again. Members lost before this release are not restored and need
re-adding once.
Two proxies no longer talk each other out of a dead backend. A replica
that withdrew its ejection because the fleet disagreed reported the backend
healthy in its next health report, though it had not reached it. With two
replicas, each took the other's withdrawal as evidence. A stopped engine
went in and out of rotation every 20 seconds, and requests landing in the
"in" half failed. A withdrawn ejection now reports unhealthy until the
replica's own probe passes.
Changed
The control plane's database moved onto the cluster's shared Postgres. fastllm-control now reads FASTLLM_DATABASE_URL from the fastllm-db-shared Secret (uri) instead of a dedicated fastllm-pg CNPG Cluster, which is retired. Data was migrated with pg_dump. The shared
instance runs Postgres 17.6 where the per-app cluster ran 18, so a schema
re-check after upgrading is worth the minute; nothing in the application
depends on an 18-only feature.