obleth-gateway v0.8.0

Know your deployment is sound, and prove it: an agentic model-testing console (Charo), a graded system scorecard (obench score), honest health for every model type, Slurm state you can trust at a glance, and benchmark traffic kept out of your numbers.

• Charo grows into an agentic model-testing console. Charo — Charon, the ferryman — carries an operator's prompt to any configured model and brings the answer back with its toll: latency, token counts, and a trace of which boons actually fired. It now runs a real agent loop over a tool framework (an admin-gated deterministic tool-run surface, plus a streaming brain-and-tools loop with confirm-to-run handoff and an iteration cap), and gets a dedicated /dashboard/charo workspace with chat, run history, settings, and a tools rail alongside the existing corner panel. Its brain model, enabled tools, and benchmark caps are configurable in Settings.
• Run a load benchmark from the chat. Charo ships a run_benchmark tool — a concurrency-ramp executor with cap enforcement, knee detection (error + p99 latency gates), percentile/step summaries, a routing-identity config fingerprint, and a blended score with grade and findings — rendered inline as a capacity-curve card.
• obench score — a graded readiness scorecard for the whole gateway. A new subcommand runs six sections — capacity ramp, gateway overhead (proxy tax), streaming quality (jitter + stalls), overload behavior, resilience (health-probe MTTD/MTTR via fault injection), and fair-share dynamics (Jain index, convergence, starvation) — rolls them into a weighted, letter-graded scorecard, stores a baseline, and diffs later runs for regressions. Runnable from the interactive TUI wizard too. The fixture backend gains a runtime POST /control for fault injection so the resilience section can measure real detect/recover times.
• Every model type now gets an honest health signal. Text-to-speech and transcription models are verified with real minimal inference probes (one character of speech; a 0.1-second silence clip). Image models are checked against the upstream's model catalog. Previously these types could only ever show "unchecked" — or worse, sit falsely unhealthy when mis-typed.
• Configuration mistakes no longer masquerade as outages. When a probe is rejected but the upstream's catalog still lists the model, the model is marked degraded with a pointed message ("model_type may be misconfigured") instead of counting toward failure alerts. A model genuinely missing upstream still alerts, now with catalog evidence in the message.
• Fixing a model's connection takes effect immediately. Changing a model's API base, upstream id, or type clears the old failure streak and alert state and re-checks within seconds — no more stale "down" badges after a config fix. Creating or editing a model also pre-flights the config, warning when the upstream doesn't list the model id, the catalog can't be verified (wildcard pass-through), or the model type isn't recognized — the save always succeeds; the warnings tell you what to fix.
• Busy models are no longer probed needlessly. The passive traffic window now follows the model's check interval, so any model with recent successful traffic is settled from the usage ledger for free. Wildcard upstream catalogs (/models returning *) are explicitly treated as unverifiable and can never produce a false healthy badge.
• Frozen Slurm replica state is now visible, loudly. The provisioner reports each reconcile tick's outcome to the gateway. When it can't reach Slurm (or is idle), the model's Replicas panel shows a warning with the failure reason and how long states have been frozen, state badges gray out with a ?, and Settings → Slurm distinguishes "running" from "running but failing since X" — previously a week-old "Healthy" pill was indistinguishable from a live one. If a successful reconcile hasn't happened for 10 minutes while Slurm provisioning is enabled, a deduplicated alert fires (with a recovery notice when it clears).
• Zombie Slurm jobs self-heal. A replica whose Slurm job still reports RUNNING but whose server is dead is restarted automatically, on either of two signals: 3 consecutive failed provisioner probes (OBLETH_PROVISIONER_RESTART_AFTER_FAILURES, 0 disables), or the gateway's real-inference endpoint check staying unhealthy — the latter catches servers that still answer metadata requests but hang on actual inference. Capped at one restart per model per tick so a probe-side network problem can never mass-cancel a fleet. Replica rows now say "updated Xh ago" (never a liveness "seen"), draining rows say why, and disabling Slurm in Settings warns that running jobs are not cancelled by it.
• Synthetic-tenant tagging keeps test traffic out of the numbers. Tenants can be flagged synthetic (obench seeds its fixture tenants that way); their traffic is recorded as benchmark traffic and, together with health probes, excluded from usage and cost stats by default (include_internal=true opts back in). Benchmark traffic never enters the permanent daily rollup.
• Reach the dashboard from any host. A new TRUSTED_ORIGINS setting (comma-separated origins; * on trusted private networks only) lets better-auth accept logins from a LAN IP or alternate hostname, not just the exact BETTER_AUTH_URL — fixing the invalid-origin login failure on self-hosted Docker/K8s deploys. Wired through .env.example, docker-compose, and the Helm chart.