Skip to content

v0.3.6

Latest

Choose a tag to compare

@piwi3910 piwi3910 released this 30 Oct 08:23
· 14 commits to main since this release
f30463f

Also in this release

  • Behaviour change: the admission gate is on by default (#81). Every
    engine-backed backend now gets an auto-tuned admission gate (migration 0059
    adds the auto / explicit / off modes). The gate is replica-aware: it divides
    its limit across --admission-replicas / FASTLLM_ADMISSION_REPLICAS
    (default 2). It halves at most once per 10s cooldown, and is restored when
    the engine goes silent or stops being detected. Tuning fields apply only in
    explicit mode.
  • Spill on high time-to-first-token (#80). A new
    max_ttft_ms_per_backend setting moves traffic to another target when the
    local backend's TTFT is above the limit. It reads TTFT from the engine's
    histogram and falls back to the proxy's own measurement. The console's
    FIRST TARGET picker now offers model pools, not only provider models.
  • The stall detector no longer ejects healthy backends under load (#79).
    It now judges liveness from the engine's step counter and the exact KV-cache
    gauge.

Changed

  • Behaviour change: Kubernetes Service discovery is opt-in. The node agent
    used to turn every LoadBalancer Service into candidates and send each a
    plain-HTTP GET /models every cycle, which filled the logs of unrelated
    TLS services with handshake errors. A Service is now a candidate only with
    fastllm.io/advertise or fastllm.io/discover: "true". LoadBalancer
    engines that were auto-discovered must now be annotated
    fastllm.io/discover: "true" or they stop registering. Pod discovery
    (hostPort, host network) is unchanged.

Fixed

  • A rebooted host comes back into its pools. Discovery drops a host's
    attachment while the host is down and creates a new one when it returns.
    Before this fix, a pool member that named that attachment was deleted
    along with it and never recreated. After one reboot, a pool balancing
    across two Sparks used only one, and so did every frontend model routing
    through that pool. Now the member is bound to its provider's name
    (migration 0058). It waits while the host is down, routing nowhere and not
    widening onto other providers, and re-attaches once the host serves the
    model again. Members lost before this release are not restored and need
    re-adding once.
  • Two proxies no longer talk each other out of a dead backend. A replica
    that withdrew its ejection because the fleet disagreed reported the backend
    healthy in its next health report, though it had not reached it. With two
    replicas, each took the other's withdrawal as evidence. A stopped engine
    went in and out of rotation every 20 seconds, and requests landing in the
    "in" half failed. A withdrawn ejection now reports unhealthy until the
    replica's own probe passes.

Changed

  • The control plane's database moved onto the cluster's shared Postgres.
    fastllm-control now reads FASTLLM_DATABASE_URL from the
    fastllm-db-shared Secret (uri) instead of a dedicated fastllm-pg CNPG
    Cluster, which is retired. Data was migrated with pg_dump. The shared
    instance runs Postgres 17.6 where the per-app cluster ran 18, so a schema
    re-check after upgrading is worth the minute; nothing in the application
    depends on an 18-only feature.