obleth-gateway v1.0.0

The 1.0 release: identity-provider tokens on the data plane, a security hardening pass across egress, filesystem, and browser policy, and a dedicated Playground for model comparison.

⚠️ Required action when upgrading. The Compose stack now publishes the Management API (9180), metrics (9091), Prometheus, and Jaeger host ports on 127.0.0.1 only. If you reached any of those from another machine, add your own port mapping in docker-compose.override.yml or route through the edge proxy. The data plane, dashboard, and edge proxy ports are unchanged. Kubernetes deployments are unaffected.

• Sign in with your identity provider's tokens. The data plane accepts short-lived JWT access tokens from configured OIDC issuers (OBLETH_JWT_ISSUERS) alongside API keys. Tokens are verified offline against the issuer's cached signing keys; a verified identity maps to a per-user identity key under the configured tenant, created on first use, so tenant budgets, schedules, model allowlists, and disable all apply as they do to any key. Usage rows record the token's device id. Identity keys appear on the Keys pages with an identity badge.

• Additional security hardening. Recipe lookup and exports reject paths and symlinks outside the configured directory. AWS's IPv6 metadata endpoint is blocked even under broad upstream allowlists.

• Dashboard scripts run by nonce. Every dashboard page now carries a per-request Content Security Policy that allows scripts only by nonce; inline script and eval permissions are gone from production. Development keeps eval for live reload only.

• Destination policy covers every save and restore. The energy Prometheus URL, Slack webhook, and slurmrestd URL are checked against the upstream destination policy when saved, and a configuration restore is rejected before any write if a model, endpoint, or MCP server in it points at a blocked address. The Slurm status, resource, and provisioner clients no longer follow redirects.

• PKCE for SSO providers. Set pkce: true on an OIDC_PROVIDERS entry to send a code challenge with the authorization request.

• DASHBOARD_PASSWORD_HASH removed. The setting was offered in the example environment files but was never read; the break-glass password has always come from DASHBOARD_PASSWORD, which the dashboard stores hashed. Nothing to change unless you set the unused variable, which had no effect.

• Compose binds management and telemetry ports to localhost. The Management API, metrics, Prometheus, and Jaeger host ports are published on 127.0.0.1 only; the data plane, dashboard, and edge proxy are unchanged. The compressor image refreshes pip before installing packages.

• Security review fixes. Pending administrators can no longer use the built-in authentication admin API. Upstream CIDR allowlists cannot reopen link-local/cloud-metadata addresses, and the shared gateway HTTP client and energy test client no longer follow redirects to unvalidated destinations. Configure final upstream URLs directly.

• Dependency security refresh. Update dashboard packages, Rust telemetry, and benchmark dependencies; use Node 24 LTS for dashboard images and CI, and HAProxy 3.2 for the Compose edge proxy. Keep authentication on the patched 1.6 series pending its separate OAuth callback/client migration.

• Tenant creation preserves your draft. Moving between setup sections keeps all inputs, submits the full configuration, and retains your work after a failed request. Clearer section navigation and validation bring you to fields that need attention.

• A dedicated Playground for chat and model comparison. Select one to four models in a shared conversation, with each question followed by independent response cards. Keep Charo’s guided tools, image support, capability checks, and benchmarks; save sessions in your browser, adjust generation settings, retry or stop responses, and continue with a chosen model.

• Disconnected streams remain accounted for. Client cancellation records a 499 usage entry and reconciles budgets using the admission estimate when final usage is unavailable. Completion bookkeeping continues if the client disconnects during settlement.

• Streaming output monitoring works across token boundaries. Log-only guardrails decode SSE content and reasoning, including streamed tool-loop responses, with bounded buffers and explicit warnings when scanning is skipped.

• Telemetry recovery uses bounded memory and disk. Spill segments replay in checkpointed batches with retry backoff. New spill stops at 256 MiB or 1,024 files, with errors and dropped-record counters when storage is exhausted. Existing WAL files remain readable.

• Routing preserves known health during refresh failures. A failed health query no longer re-enables unhealthy models or models under maintenance.

• Idle group capacity is redistributed instead of sitting unused. Hierarchical group shares were hard ceilings: a backlogged group could not use slots a quieter group was leaving idle, so a contended gateway ran well below its concurrency limit with requests waiting. Shares now cap contended demand only, and spare capacity goes to whoever is queued. A group returning to demand reclaims its share as the borrowed requests finish. On a five-tenant benchmark this lifted utilization from 62% to 99% and throughput by roughly 60%. Weighted mode was already unaffected.

• Fair-share debt is weight-adjusted in every mode. The per-tenant share_score on the fairshare view divided served tokens by tenant weight under the weighted algorithm but returned the raw token count under the hierarchical default, even though admission ranks tenants inside a group on the weight-adjusted value. The field now reports the score the scheduler actually uses, matching its documented definition.

obleth-gateway v0.9.6

A faster, tidier dashboard: the overview leads with what needs attention, heavy pages load lighter, and destructive actions confirm consistently.

• Overview redesigned around action. The landing page leads with gateway health, live work, and 24-hour traffic and cost tiles, plus a "Needs attention" panel listing unhealthy routes, admission backlog, and tenants below fair share — each linking straight to the right page. Top models keep the latency signals needed for quick triage, and the sidebar groups pages into Overview, Operations, Access, and Configuration.
• The dashboard's summary poll is far lighter. A new aggregate endpoint returns entity counts and windowed usage totals in one call, so the status footer's 30-second poll on every page no longer downloads the full tenant, key, and model lists — a large key fleet was previously deserialized just to display a count.
• The models page opens fast on large fleets. Per-model health history and endpoint lists load when a model card is expanded instead of three-requests-per-model up front, and the settings page fetches all of its sections in parallel.
• Smoother live pages. The fast scheduler poll no longer re-renders the traffic chart and model tables every two seconds, report charts stop re-animating on every filter change, navigation shows a loading skeleton instead of freezing on the previous page, and an unexpected render failure lands on a styled error screen with a retry.
• Destructive actions confirm consistently. Deleting keys, tenants, models, endpoints, MCP servers, and templates now uses a styled in-app dialog stating the consequence; deleting every key matching a filter keeps the typed two-step confirmation.
• The request log's "traced only" filter now applies. It previously returned unfiltered rows.
• Numbers read the same everywhere. Compact counts, durations, and shortened IDs share one format across pages, replacing per-page variants that disagreed on casing and thresholds.

obleth-gateway v0.9.5

Slurm replica endpoints stop depending on flaky per-request DNS, and replicas whose endpoint vanished out-of-band now heal themselves.

• Node address overrides. Slurm settings gain a node hostname → IP override list, editable in the dashboard, for clusters where the pods running obleth resolve compute-node names unreliably. The provisioner registers each replica's endpoint by IP and probes by IP, so neither health checks nor proxied requests depend on per-request DNS — a single missed lookup previously surfaced as an instant 502 upstream request failed. Edits take effect on the next provisioner tick without a restart; leave the list empty to keep resolving node names through DNS.
• Existing endpoints migrate to their resolved address in place. Endpoints already registered by node name are rewritten to the resolved IP on the next tick — keeping their name, priority, weight, and enabled flag — so running replicas stop depending on DNS immediately after upgrading, without being re-provisioned.
• Node names resolve with retry and a cache even without overrides. Un-aliased hostnames are resolved at promotion with a short retry, a per-node success cache, and a last-known-address fallback, so one transient DNS miss no longer decides where an endpoint points. Model warm-up requests also target the resolved address.
• Replicas with a vanished endpoint re-register instead of serving a phantom. A replica marked healthy whose endpoint was removed out of band — a manual delete in the Reliability tab, or a cancellation that only half-landed — used to sit "healthy" forever while the model ran one endpoint short. The provisioner now detects the dangling reference and registers a fresh endpoint on the next tick.

obleth-gateway v0.9.4

The v0.9.3 self-heal fix now actually reaches deployments, and failed job cancellations say why.

• Deployment defaults no longer undo the v0.9.3 self-heal fix. The docker-compose fallback, the Helm chart default, and the env example all still pinned OBLETH_PROVISIONER_RESTART_AFTER_FAILURES=3, silently overriding the binary's new default — so provisioners deployed from them kept cancelling healthy replicas after ~45 seconds of probe flaps. All three now default to 20 (~5 minutes of sustained failure at the default tick interval). If you set the variable to 3 yourself, remove or raise it when upgrading.
• Cancel failures report the real reason. When slurmrestd refuses to cancel a job, the provisioner now logs the response body (e.g. Access/permission denied from slurm_kill_job2) instead of only the HTTP status — a bare 500 previously hid actionable causes like a JWT user / job owner mismatch.

obleth-gateway v0.9.3

Slurm-provisioned replicas stop getting killed in a loop, and jobs with a walltime submit cleanly.

• Healthy replicas are no longer cancelled on transient probe blips. A busy single-threaded inference server (e.g. llama.cpp) that briefly missed a health probe and passed the next used to accumulate toward the self-heal threshold and get cancelled — then resubmitted, flap, and die again in a loop. The failure counter now decays on every passing probe, so only a sustained outage restarts a replica; the default window is ~5 minutes (OBLETH_PROVISIONER_RESTART_AFTER_FAILURES, now 20). The gateway's separate inference-based zombie check is unchanged.
• Jobs with a time limit submit without a slurmrestd error. A managed model's walltime (e.g. 0-04:00:00) is now converted to the integer minutes slurmrestd expects, instead of being sent as a date string that failed the submit with a 500. Unparseable values are omitted so the partition default applies.

obleth-gateway v0.9.2

Managed-model config catches bad inputs before Slurm does, provisioning-error notices can be dismissed, and a dead replica pool can no longer ride a stale success to "healthy".

• Managed-model settings validate before saving. The Placement and Service fields now check their inputs client-side — time limit format, port range, replica counts, node/CPU numbers — and highlight the offending field with an inline hint, instead of forwarding a malformed value and surfacing an opaque slurmrestd 500. Each field also carries a short format hint.
• Provisioning-error banners are dismissible. When the provisioner rejects a job (bad account / partition / QoS), the model's error banner can now be cleared once you've fixed the cause — it returns on its own if the next launch also fails.
• A model whose replicas have all died no longer reports "healthy". For Slurm-provisioned / dynamic-endpoint models, a recent passive success from the usage ledger could stand in as the health verdict even after every replica behind it went away — a success served just before the pool emptied. The passive shortcut now only settles a pool that still has a live endpoint serving; an empty or fully-dead pool reports its own reality.

obleth-gateway v0.9.1

Benchmarks report tokens-per-second, and benchmark traffic finally stays out of your real usage numbers.

• Tokens-per-second across the benchmark suite. Every concurrency step now reports aggregate output tok/s and per-stream decode rate (p50/p10) — in the obench CLI summary, the live TUI, the capacity scorecard, and Charo's inline benchmark card.
• Benchmark and test traffic no longer pollutes usage stats. Charo's model-test console and in-dashboard benchmarks reach the gateway through a reserved internal identity, which is now marked synthetic — so its requests are tagged as benchmark traffic and excluded by default from the overview's request, token, and tokens-per-second figures. Previously a single capacity run could bury a model's real numbers under thousands of synthetic requests. Existing pre-upgrade rows age out of the rolling window on their own; internal traffic stays viewable on demand.
• Charo hides models' hidden reasoning. Chain-of-thought that some models emit inline (<think>…</think>) is now stripped from Charo's answers and from the transcript sent back upstream, so you see the reply and its tool cards, not the scratchpad.
• The chat panel, refined. Assistant replies render as clean chat bubbles, the mascot imagery is retired for a tighter layout, and the typing indicator sits in-bubble so a pending reply never reads as dead air.

obleth-gateway v0.9.0

Charo grows from a testing console into a working colleague: guided activities, direct model chat, MCP verification, documentation-grounded answers with citations, and a redesigned chat panel.

• Guided activities, opened conversationally. Testing a model's capabilities, chatting with a specific model, and benchmarking are now step-by-step workflow cards in the chat thread — pick a model and options inline. Ask Charo in plain language ("test gemma4") and it opens the right workflow itself.
• Probe a model's capabilities from the chat. The capability test fires each configured boon through the gateway — quick ping, tools/web search, forced JSON, vision — with live pass/warn/fail rows, per-test output, and the request trace. The vision probe now requires a real image you attach (picker or drag-and-drop) instead of a bundled placeholder.
• Chat with any model directly. A raw, persona-free line to the model you pick — a banner shows who you're talking to, and exiting returns you to Charo.
• MCP servers verified end-to-end. A test_mcp tool runs the real MCP handshake through the gateway and lists each server's tools; the dashboard MCP tab auto-probes servers and gains a Test button. Deleting an MCP server now strips its grant from every model that had it.
• Ask the docs. Charo answers how-to and configuration questions grounded in the official documentation, with cited source pages linked under the answer — and says so plainly when the docs don't cover something.
• The chat panel, redesigned. Assistant replies render markdown properly; results hang off a clean rail instead of stacked gray boxes; answers stream below their sources so nothing hides off-screen; scrolling up mid-stream no longer gets yanked back; images attach via paperclip or drag-and-drop; compact type fits more in the small window. The panel is titled Gateway Chat.
• Charo sounds like a person. Personality calibration, a stop button for in-flight runs, a typing indicator, and greetings no longer deflected as chit-chat.
• Dashboard fixes. Model edit forms no longer revert to stale values on save; the container's .next/cache is writable by the runtime user.

obleth-gateway v0.8.1

Health badges you can trust: non-chat models stop showing a false "degraded", recovered models clear themselves, and Charo opens as a pop-out modal from anywhere.

• Embedding, TTS, transcription, and image models no longer show a false "degraded". The scheduled health worker was probing every model against the chat completions endpoint regardless of its type, so any non-chat model was rejected (HTTP 404) and left sitting "degraded — model_type may be misconfigured" even when correctly configured and serving. Scheduled checks now probe each model's real modality endpoint. (Manual "Check now" and bulk checks were already correct — only the background sweep was affected.)
• Recovered models clear themselves instead of staying stuck "unhealthy". A window of stale upstream errors with no recent successes could stand in as the health verdict and suppress the very active probe whose success would have cleared it — pinning a recovered model "unhealthy" until the window aged out. Only an observed success now settles a check for free from the usage ledger; anything short of that defers to a live probe as ground truth.
• Charo opens as a pop-out modal from anywhere. The dedicated /charo page is replaced by a centered pop-out modal, so the model-testing console can be summoned over whichever dashboard view you're on without navigating away.

obleth-gateway v0.8.0

Know your deployment is sound, and prove it: an agentic model-testing console (Charo), a graded system scorecard (obench score), honest health for every model type, Slurm state you can trust at a glance, and benchmark traffic kept out of your numbers.

• Charo grows into an agentic model-testing console. Charo — Charon, the ferryman — carries an operator's prompt to any configured model and brings the answer back with its toll: latency, token counts, and a trace of which boons actually fired. It now runs a real agent loop over a tool framework (an admin-gated deterministic tool-run surface, plus a streaming brain-and-tools loop with confirm-to-run handoff and an iteration cap), and gets a dedicated /dashboard/charo workspace with chat, run history, settings, and a tools rail alongside the existing corner panel. Its brain model, enabled tools, and benchmark caps are configurable in Settings.
• Run a load benchmark from the chat. Charo ships a run_benchmark tool — a concurrency-ramp executor with cap enforcement, knee detection (error + p99 latency gates), percentile/step summaries, a routing-identity config fingerprint, and a blended score with grade and findings — rendered inline as a capacity-curve card.
• obench score — a graded readiness scorecard for the whole gateway. A new subcommand runs six sections — capacity ramp, gateway overhead (proxy tax), streaming quality (jitter + stalls), overload behavior, resilience (health-probe MTTD/MTTR via fault injection), and fair-share dynamics (Jain index, convergence, starvation) — rolls them into a weighted, letter-graded scorecard, stores a baseline, and diffs later runs for regressions. Runnable from the interactive TUI wizard too. The fixture backend gains a runtime POST /control for fault injection so the resilience section can measure real detect/recover times.
• Every model type now gets an honest health signal. Text-to-speech and transcription models are verified with real minimal inference probes (one character of speech; a 0.1-second silence clip). Image models are checked against the upstream's model catalog. Previously these types could only ever show "unchecked" — or worse, sit falsely unhealthy when mis-typed.
• Configuration mistakes no longer masquerade as outages. When a probe is rejected but the upstream's catalog still lists the model, the model is marked degraded with a pointed message ("model_type may be misconfigured") instead of counting toward failure alerts. A model genuinely missing upstream still alerts, now with catalog evidence in the message.
• Fixing a model's connection takes effect immediately. Changing a model's API base, upstream id, or type clears the old failure streak and alert state and re-checks within seconds — no more stale "down" badges after a config fix. Creating or editing a model also pre-flights the config, warning when the upstream doesn't list the model id, the catalog can't be verified (wildcard pass-through), or the model type isn't recognized — the save always succeeds; the warnings tell you what to fix.
• Busy models are no longer probed needlessly. The passive traffic window now follows the model's check interval, so any model with recent successful traffic is settled from the usage ledger for free. Wildcard upstream catalogs (/models returning *) are explicitly treated as unverifiable and can never produce a false healthy badge.
• Frozen Slurm replica state is now visible, loudly. The provisioner reports each reconcile tick's outcome to the gateway. When it can't reach Slurm (or is idle), the model's Replicas panel shows a warning with the failure reason and how long states have been frozen, state badges gray out with a ?, and Settings → Slurm distinguishes "running" from "running but failing since X" — previously a week-old "Healthy" pill was indistinguishable from a live one. If a successful reconcile hasn't happened for 10 minutes while Slurm provisioning is enabled, a deduplicated alert fires (with a recovery notice when it clears).
• Zombie Slurm jobs self-heal. A replica whose Slurm job still reports RUNNING but whose server is dead is restarted automatically, on either of two signals: 3 consecutive failed provisioner probes (OBLETH_PROVISIONER_RESTART_AFTER_FAILURES, 0 disables), or the gateway's real-inference endpoint check staying unhealthy — the latter catches servers that still answer metadata requests but hang on actual inference. Capped at one restart per model per tick so a probe-side network problem can never mass-cancel a fleet. Replica rows now say "updated Xh ago" (never a liveness "seen"), draining rows say why, and disabling Slurm in Settings warns that running jobs are not cancelled by it.
• Synthetic-tenant tagging keeps test traffic out of the numbers. Tenants can be flagged synthetic (obench seeds its fixture tenants that way); their traffic is recorded as benchmark traffic and, together with health probes, excluded from usage and cost stats by default (include_internal=true opts back in). Benchmark traffic never enters the permanent daily rollup.
• Reach the dashboard from any host. A new TRUSTED_ORIGINS setting (comma-separated origins; * on trusted private networks only) lets better-auth accept logins from a LAN IP or alternate hostname, not just the exact BETTER_AUTH_URL — fixing the invalid-origin login failure on self-hosted Docker/K8s deploys. Wired through .env.example, docker-compose, and the Helm chart.