Skip to content

v0.7.0

Latest
Latest

Choose a tag to compare

View all tags
@flyworker flyworker released this 16 Oct 07:24
v0.7.0
5423814
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Disable a model without removing it. "disabled": true in models.json keeps a model out of service across restarts: it is left out of the startup registration and every re-registration, and the health checker and self-check no longer probe it (#184, #194, #196, #198).
  • Routing view. A new dashboard view traces traffic from where it entered the node (hub, health probe, self-check, local gateway) through each model to the backend that served it, including work a model server did that never passed through the node (#191, #193, #195, #197).
  • Request routes. Each recorded request now carries the route it took, and the dashboard shows the context window the hub assumes for each model (#188).
  • Embeddings through the local gateway. POST /v1/embeddings is forwarded and recorded like chat requests (#199).

Security

  • The CLIProxyAPI management key no longer leaves the machine. The usage sampler treated any endpoint whose /metrics answered 404 as a possible CLIProxyAPI and sent it a management request carrying CLIProxyManagementKey. If models.json points at a remote API, that API received the key every minute. The key is now sent only to loopback addresses, and remote endpoints are not sampled. If you have configured a remote endpoint and set CLIProxyManagementKey, rotate the key: change remote-management.secret-key in each proxy's config, restart the proxies, and update CLIProxyManagementKey (#201).

Changes

  • Only models registered with Swan Inference are health-checked. Local-gateway backends — a local_endpoint, or a local-only model's endpoint — are no longer probed; their status on the dashboard is the outcome of the last request the gateway sent them. Local-only models are ready as soon as they are configured (#201).

Fixes

  • A stale read of the ready model set could drop a model from a registration (#182).
  • Local-only models were included in the health report sent to Swan Inference, which put them back on the routing table; the node then refused every request routed to them (#190).
  • The self-check now reads llama.cpp's per-slot context window rather than the total (#181).
  • A hub request for a local-only model no longer raises a critical error-rate alert; it raises a single warning that the hub is still routing that model to the node (#180).
  • Proxy usage recorded under a backend's served name is merged into the model ID (#176).

Observability

  • The node logs why a model leaves or joins its registration, and the dashboard shows registration drops; the startup list is not counted as a drop (#187, #189).
  • Proxy accounts: token consumption per model (#173, #175), a warning when an account's usage figures are stale (#192), and disabled models and unreachable endpoints are left out of the report (#183, #185).

Upgrading

computing-provider update

Then restart the daemon and, if you run it, the dashboard. No configuration changes are required; disabled is optional.

Full changelog: v0.6.1...v0.7.0

Assets 8
Loading