Ume LLM Gateway 0.8.5

Fix: Claude on an Azure AI Foundry provider failed with 401. The gateway sent the provider's key to the resource's
Anthropic API in the api-key header, which that API does not read; it now sends it in x-api-key. A Bearer token is
sent as before. Rolling update: yes.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.8.5@sha256:ad3291087b979077b790c15d17cd2290e9bbff1c9f1f4de92bdd405a1aa23f5e
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.8.5@sha256:113e713d167e2240cde9877e34a1a85b5f89ba5f608f415aac5e674268ec4831
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.8.5@sha256:bc0f256560b39adb705527a6fbf69f00dcbe66c594f467025a4602dfb039de4a
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.8.5@sha256:5b35bb2042d7f571694e362637a3229d69997cf2c24c28d7410ec6abffd14b48

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.8.5/ume-llm-gateway-0.8.5.tar.gz
tar -xzf ume-llm-gateway-0.8.5.tar.gz && cd ume-llm-gateway-0.8.5
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.8.4

Model prices follow the open models.dev list instead of being kept by hand on every installation, and models get
descriptions and capabilities from it; one Azure AI Foundry provider now serves Claude too; known provider types need no
capability boxes; teams can manage their own keys; audio can be priced per second, minute or hour; and a sizing guide and
one benchmark baseline. Migrations AudioDurationPrecision, TeamCodes, ModelListPrices and ModelBaseModel; read the
0.8.4 section of docs/upgrade-guide.md before upgrading (prices of existing models can change). Rolling update: yes.

Model prices and details from models.dev. Prices for Anthropic (also Claude on Foundry), Ollama Cloud, OpenAI and
OpenRouter models no longer have to be kept up to date by hand on every installation. Once a day the admin API reads the
open models.dev list and adds a changed list price to the price history of every model that follows
it, from the time of the check (earlier usage keeps its cost). Input, output, cached input, cache write and audio-token
prices come from the list; a duration or per-image price is kept. Following the list is on by default and can be turned
off per model (Price from models.dev) or per provider (Prices from models.dev); a model that follows the list cannot be
given a price by hand, and a model that is not in the list is priced by hand until it is. Providers › Models shows each
model's price mode and can filter by it, the price history shows each price's source, and changes are audited as
system:price-list. The same list fills in a model's empty description (new, shown in the catalogue), capabilities and
context window, and discovery uses it for what the provider does not report. Settings › Model prices shows the last
update and has Check now. Configure with Admin:PriceListUrl (a copy of the file for an admin API without internet
access, or empty to turn it off) and Admin:AutoUpdatePrices. Azure OpenAI and Foundry deployments get descriptions, capabilities and context
windows from the list too (looked up by the model behind the deployment), and its price when they have none; Azure's own
price is otherwise kept.

No capability boxes for known providers. What a provider serves now follows from its type: OpenAI, Azure OpenAI and
Foundry the whole OpenAI API, Anthropic chat and Messages, Ollama chat and embeddings, Ollama Cloud chat, and Claude on
Foundry chat and Messages. The provider form shows what the type supports instead of eight boxes to get right; only an
OpenAI-compatible server still has Endpoints this server offers (default chat, streaming and embeddings). Which model
gets a request still follows the model's kind. Rolling update: yes.

One Azure AI Foundry provider for every model, Claude included. A Foundry provider now discovers the project's Claude
deployments and the gateway sends them to the resource's Anthropic API (…/anthropic/v1, same key), while the other
deployments go to …/openai/v1; admins no longer add a second, Anthropic-type provider per Foundry resource. Models get a
Base model (the model behind a deployment, set by discovery) that decides this and finds the model in models.dev.
Existing Anthropic providers at a Foundry address keep working. Migration ModelBaseModel. Rolling update: yes.

The configuration export now includes model capabilities. Migration
ModelListPrices. Rolling update: yes.

Sizing. Production setup has a Sizing section, and the website's Benchmarks page a What to run it on section:
for up to about 50 messages/s, one host with 2 vCPU, 4 GB RAM and 20 GB disk plus usage data (about 560 bytes per
request kept), or 1–1.5 vCPU and 1.5 GB for the gateway and admin API with your own Postgres and Redis. Measured with
the new benchmarks/compare/sizing/run.sh: the gateway limited to 0.5 vCPU and 512 MB, PII detection and the content
guard on, real-provider timing and mixed traffic; it carried 100 streamed messages/s at 236 MB without adding latency.
The website no longer publishes Streaming findings (it stays in docs/), and the start page says how the headline
figures were measured: in a Docker container limited to 4 vCPUs.

One benchmark baseline. Every performance figure in the docs and on the website now comes from one run (2026-10-27,
Results (baseline 2026-10-27) in docs/performance.md). Older measurements are no longer quoted anywhere:
docs/benchmark-2026-10-13.md is removed, and docs/performance.md, docs/streaming-findings.md and the improvement
plan are rewritten around the new run. benchmarks/compare uses one CPU layout for run.sh, scale.sh and
latency.sh, gives the fake LLM four threads without telemetry and Redis its own core without disk snapshots, and every
report ends with a harness check showing how busy the fake LLM, Postgres and Redis were, so a run limited by the harness
is visible. For 6 vCPUs scale.sh borrows one thread each from the fake LLM and Postgres (checked not to limit the
gateway). Gateway, 4 vCPUs: 0.97 ms added latency at one client, 17 888 streamed and 16 805 non-streamed requests/s at
the peak, 0.21 ms CPU per message; 20 550 streamed requests/s on 6 vCPUs. The eneo comparison now uses eneo's latest
release, v2.2.3, through a service API key, measured on the same harness the same day (ENEO_VERSION picks the image;
the stream probe can use the API key too): eneo peaks at 51 streamed messages/s on 4 vCPUs at 79 ms CPU per message. The
website and the platform comparison slides (HTML and PDF) use these numbers. Rolling update: not affected (benchmarks
and docs only).

Teams manage their own keys. A new role, team-admin, lets team members create, change, rotate and revoke their
team's keys themselves, with every key setting a department admin has, without access to the rest of the department.
Which teams comes from a new teamCodes claim (Oidc:TeamClaim, UME_OIDC_TEAM_CLAIM), matched against a new
Team code on each team (Organisation › Edit team, code on /api/teams, unique, migration TeamCodes). With one
AD group per team, set the team claim to the group claim and use the group names as team codes; the role itself can
come from a group too (UME_OIDC_GROUPS_TEAM_ADMIN). Team admins see Keys and the Catalogue only: no stored secrets,
team settings, budgets or usage. A team code grants its keys only: a viewer or department admin who is also a team
admin gets the team's keys on top of what they had, nothing more. The dev Keycloak realm has a team-admin user for the
seeded team Digitalisering & AI. See Team scope in docs/identity-and-roles.md. Rolling update: yes.

Input audio can be priced per second, minute or hour. Providers price audio by duration in different units
(Whisper per minute, Azure speech and on-prem servers often per hour, some per second), and the price form only took USD
per minute. Audio input by duration in the price form now has a unit (second, minute or hour); the gateway still
stores the price per minute (audioPerMinuteUsd, now with ten decimals so a price per hour or second is kept exactly),
and the price history and model list show it in the unit it reads best in. A model with a duration price and no audio
input token price no longer also bills its audio input tokens as text tokens: before, a realtime model priced per hour
was charged for the duration and for the audio tokens at the text input price. Migration AudioDurationPrecision.
Rolling update: yes.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.8.4@sha256:69fd32deffc3534499c343573de116d2cd1e4e2189ba6e5ca7739e8b8abb05d5
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.8.4@sha256:d1703d93ccad52d4a463281b810f3e4a59ec2c3ac9f8052fe442d76900776416
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.8.4@sha256:3ea3eaf017b12d025428d5152a6cfc0d937b0cd2196d368d74f8bc2eca210e71
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.8.4@sha256:d0463b9a735652ecaa5f2a91272effc4b7973467f46b969c8992a4c8e26b05b8

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.8.4/ume-llm-gateway-0.8.4.tar.gz
tar -xzf ume-llm-gateway-0.8.4.tar.gz && cd ume-llm-gateway-0.8.4
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.8.3

Agents that call tools through Chat Completions work on gpt-5.6 and GPT-6 (gpt-6-astra, -sol, -luna, gpt-6.1) again:
the gateway bridges those requests to the Responses API. No configuration, no schema change. Rolling update: yes.

Tool calls work on gpt-5.6 and GPT-6 through Chat Completions. On these models a Chat Completions request with
function tools failed with "Function tools with reasoning_effort are not supported … use /v1/responses", whatever the
client sent: they reason by default, and gpt-6-astra and gpt-6.1-sol also reject reasoning_effort: "none". The gateway
now serves such a request through the provider's Responses API and returns the answer, or the stream, in Chat Completions
form, as LiteLLM and Bifrost do, so agents keep working with reasoning on and nothing needs configuring. Known model names
(gpt-5.6 and later, GPT-6 and later) are bridged at once; a deployment with a name of its own is bridged when the provider
first answers with that error, and remembered until restart. Requests without tools and requests with
reasoning_effort: "none" are unchanged. See Tool calls on gpt-5.6 and GPT-6 in docs/providers-and-models.md.
Rolling update: yes.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.8.3@sha256:be12c06b550cc57fa60963658ba8a85dd08be11ec69d2a4864fb116b0472d654
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.8.3@sha256:da5bcf04f0b3e5687f346aaa14d5454e2819a363e8e75d54a065e95141a8da85
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.8.3@sha256:f1137b08a099e924b8d6d00e20e15687255b5e27d11c01f345752b8b4507634e
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.8.3@sha256:ef83211dfafcc8ace5bc3a3c8e24006398346916f9b4b1d3344c06a1839cf1e4

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.8.3/ume-llm-gateway-0.8.3.tar.gz
tar -xzf ume-llm-gateway-0.8.3.tar.gz && cd ume-llm-gateway-0.8.3
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.8.2

Provider speed on Overview, Health & operations and Provider stability is now the provider's own time to first token,
and failed requests can be listed from every error count. One admin API rename (p50LatencyMs/p95LatencyMs become
p50FirstTokenMs/p95FirstTokenMs; see the 0.8.2 section of docs/upgrade-guide.md). Rolling update: yes.

Provider speed is the provider's time to first token. The p50/p95 per provider on Overview, Health & operations and
Provider stability was the whole request's latency, so long answers and failed attempts at other providers made a fast
provider look slow. The gateway now records, for each streamed answer, the time from sending the request to the provider
that answered until its first event (ProviderFirstTokenMs; Data API and request details providerFirstTokenMs, metric
ume.gateway.provider.time_to_first_token), and the provider figures use it. Answers that are not streamed have no
comparable figure and are not counted. The admin API fields are renamed p50FirstTokenMs/p95FirstTokenMs (see the
upgrade guide). Rolling update: yes.

Failed requests are easy to find. Usage › Requests and costs has an Outcome filter (All errors or one outcome),
plus model and provider filters, and shows the error code under each outcome. Each Errors count in the summary lists
that row's failed requests, and the Usage dashboard's Requests card and Overview's error figures link to them.
GET /api/usage/requests takes errors=true, model and provider.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.8.2@sha256:7e440a63b84ff22baab8779d6ad9adf5f65af16d871360099db5a223fec18c6f
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.8.2@sha256:d6832c569de04556e63ba777a5a898e65e666989a8509ff856a1ca831c6b49e1
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.8.2@sha256:838e14e201ee2c6294652ba9449ef8718e0a6a1ca1a2f277a92bb9d685281fd8
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.8.2@sha256:d6ac8e2c58524801bb0123c36c5f5fdebdee9d61f9e3064809beffde7c72a8c5

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.8.2/ume-llm-gateway-0.8.2.tar.gz
tar -xzf ume-llm-gateway-0.8.2.tar.gz && cd ume-llm-gateway-0.8.2
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.8.1

Costs in USD only: SEK and the exchange rate are removed from every interface (breaking for anything that reads the
SEK headers, webhook fields or Data API fields; see the 0.8.1 section of docs/upgrade-guide.md). The admin UI works on
iPhone again (menu, search), and the routing rule test says when a model does not exist. Rolling update: yes.

Costs in USD only; SEK and the exchange rate removed. Prices were already in USD, and budgets and the admin UI
already counted and showed USD; the exchange rate under Settings only fed SEK copies of the same amounts. It is gone,
with every SEK output: the x-ume-cost-sek and x-ume-budget-remaining-sek response headers, spentSek/limitSek in
budget alert webhooks, costSek in the Data API's usage records and aggregates, limitSek on its budgets and its
/v1/catalog/exchange-rates, the ume.gateway.cost metric (SEK; ume.gateway.cost.usd stays), the admin API's
/api/settings/exchange-rate, the gateway's exchange-rate readiness check and the Gateway:FallbackSekPerUsd
setting. Breaking for anything that reads those; see the upgrade guide. Rolling update: yes.
The database keeps the old CostSek columns (no longer written) and ExchangeRates table for one more release.

Admin UI on iPhone. The phone menu opened as a strip showing only its title in Safari on iOS; it now opens at full
height. Search no longer zooms the page in when it opens (iOS zooms into fields with text smaller than 16px, so fields
use 16px there), and its results fit above the on-screen keyboard, so the last ones can be reached.

Fixes. Routing rules › Test a request showed a model name that does not exist as a disabled model; it now says
Does not exist. The Request logging status on Settings is shorter (Off: problem requests only), so it fits on a
320px screen. The browser tests (tests/e2e) follow the current admin UI again: the Requests tab on Usage, the
New rule button next to New rule from template, English error messages, target-size checks limited to the open
dialog, and a new sign-in when an earlier test signed the cached session out.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.8.1@sha256:d65fd519655699920a6e2ddf9a9a9fee0518998e1a5d8a937e9c0fcabe175798
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.8.1@sha256:8bedaef9fd5ec0556e7c5e887a5dddfe875a2731b876d9ba6258bf7d259268ac
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.8.1@sha256:3dce0e118f19855e7f2e2c36233f5d69d4d65ebb69ce047cfd511f89d7766811
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.8.1@sha256:d901dd2e31d97e9e7a4e752747d9236dba7cd0bde6bdfb33ce858d39a6d78b21

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.8.1/ume-llm-gateway-0.8.1.tar.gz
tar -xzf ume-llm-gateway-0.8.1.tar.gz && cd ume-llm-gateway-0.8.1
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.8.0

A clearer admin UI with a menu grouped by task, search into pages and a working phone layout; costs in US$ throughout
the admin UI; a regrouped, phone-friendly website; and fewer npm packages with limits on what they can do. Rolling update:
yes (no migrations). One format change: the usage CSV export has kostnad USD instead of kostnad SEK (see the 0.8.0
section of docs/upgrade-guide.md).

Admin UI costs in USD. Overview (spend this month, daily spend, spend by department), Usage (totals, grouped tables,
request details) and the Catalogue prices showed Swedish kronor; they now show US$, like budgets. The usage CSV export
has kostnad USD instead of kostnad SEK, and the admin API's usage summaries and catalogue report USD. The Data API,
gateway response headers and budget webhooks still include SEK. See the upgrade guide.
The daily spend chart on Overview shows all 30 days, also those without spend (with one day of usage, that day's bar
filled the whole card).

Website: docs menu by task, and a working phone layout. The documentation menu is grouped by what you came to do:
Get started (now with Identity provider and roles), Build with the API, Set up the gateway, Run it, Security and
compliance, Performance and Reference. On phones and tablets the page dropdown is replaced by a docs menu that shows where
you are, plus an On this page list; the menu closes with Esc or a tap outside, and on narrow phones the theme switch
moves into it so the header fits at 320px. Fixed sideways scrolling on narrow phones (the SDK sample on the home page and
long code in headings). Admin UI menu paths in the docs follow the new menu (Models & routing › Routing rules). The
manual test checklist is removed; manual testing is tracked elsewhere.

Fewer npm packages, and limits on what they can do. The admin UI no longer uses Reka UI: dialogs, drawers, search
and the phone menu use the browser's <dialog> element, which takes 17 packages out of the code that runs in the
browser (and the unused axe-core out of the admin UI's dev tools). Each npm project has an .npmrc that turns off
install scripts, skips versions published in the last 7 days when packages are added or updated, and refuses
dependencies from git; npm ci and the lockfiles are unchanged. undici (used by the unit tests) updated for
GHSA-4qv8-2p9j-2685. See npm packages.

Admin UI: clearer menu, search into pages, and a working phone layout. The side menu is grouped by task: Monitor,
Access, Models & routing, Cost & compliance and System (Catalogue moved to Models & routing, PII incidents to Cost &
compliance, Settings to System; Operations is now Health & operations, as on the page). Search (Ctrl K) also finds
cards and tabs inside pages and everyday words (retention, circuit, department), works with the arrow keys and opens
full screen on phones. On phones a Menu tab opens every page, the theme, the session and Sign out, and the end of
a page is no longer hidden behind the bottom bar. Fixes: the session timer lost its hours ("h 34 min left"), the "Skip to
main content" link showed over the header on phones, and the daily spend and provider stability charts had no height
(Tailwind classes using spacing steps that did not exist produced no CSS). The time labels under narrow
charts (the Usage cards) no longer run together, and the Overview circuit alert says Open Health & operations.
Screenshots on the website and in docs/images are retaken with the new menu. See Admin UI.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.8.0@sha256:4229fa97d764acc7c1a9483148bdda555d78df252aeb5981740c927dc493692d
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.8.0@sha256:d81d9167e4790d9f5301addff3107cd5bf4374212a50baac010d42b2873537aa
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.8.0@sha256:c007920a5acac6e7c7ab64cb8085abfe925ea13e4860d44b0b06ccfb6e64474d
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.8.0@sha256:ec463dcb89e3cd87502d2a5daf3a7f0337acbbe8c97c5fbacb8ab6fa619f6153

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.8.0/ume-llm-gateway-0.8.0.tar.gz
tar -xzf ume-llm-gateway-0.8.0.tar.gz && cd ume-llm-gateway-0.8.0
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.7.1

Fix release: admin sign-in failed with a 500 on deployments with the shipped Redis ACL. Rolling update: yes (no
migrations). Redeploy the Redis ACL (see the 0.7.1 section of docs/upgrade-guide.md).

Fix: admin sign-in failed with a 500 (RedisServerException). Admin sessions in Redis (0.6.4) are written with a
time to live, which StackExchange.Redis sends as SETEX/PSETEX. The shipped ACL only allowed set, so every sign-in
failed with NOPERM after the identity provider returned. deploy/redis/users.acl.example (and the generated Portainer
stack) now allow setex and psetex; the forced circuit-breaker open (StringSetAsync with a duration) needed them too.
Redeploy the stack (Portainer) or rerun init-deployment.sh and restart Redis (Compose) to regenerate users.acl.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.7.1@sha256:9ae6b87ebe31be9c7c3b529edf2bac37d41ddb7485996aa2b7a9f0f864268c23
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.7.1@sha256:99326ede3805fcc32f8bb95e2127350f49434eeef7f82784d1852c00e9a250fb
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.7.1@sha256:767ad47a8b810326a81ad0833dd9511b727dd838767aff3f4024c7b22efa529d
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.7.1@sha256:6d2923d9b5847e31348eedf192ddbc856e30714cd8bf4712ec0693aedb8c8b08

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.7.1/ume-llm-gateway-0.7.1.tar.gz
tar -xzf ume-llm-gateway-0.7.1.tar.gz && cd ume-llm-gateway-0.7.1
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.7.0

An on-prem content guard, token quotas for sharing on-prem capacity, no usage lost while the database is down, and
gateway improvements from a comparison with agentgateway (Retry-After cooldown, least-loaded balancing, retries, sliding
rate limits with a concurrency cap, request policies, prompt caching, cache-write prices). Rolling update: yes (five
additive migrations). Read the 0.7.0 section of docs/upgrade-guide.md first: the Redis ACL must be redeployed for the
usage journal, per-minute limits slide and a failed last candidate is retried once, and token quotas and
/v1/messages/count_tokens should wait until the rollout is finished.

Content guard (on-prem). A per-key check for prompt injection and harmful content (Keys > Content guard: Off,
Record, Block; admin API contentGuardPolicy). Rules in the gateway catch common prompt-injection phrasings in English
and Swedish and your own patterns (Gateway:ContentGuard:BlockedPatterns); an optional guard model (Llama Guard or IBM
Granite Guardian) on your own on-prem server checks meaning (Gateway:ContentGuard:Model, Format), and complete
answers too with CheckAnswers. The gateway never uses a guard model that is not on an on-prem provider. Record keeps the
categories on the usage record (Usage, Content guard), adds x-ume-guard: flagged and logs gateway.content.flagged;
Block answers content_blocked. Keys with the guard off do no extra work. See docs/content-guard.md.
Migration ContentGuard.

No usage lost when the database is down. When Postgres refuses a batch of usage records, the gateway keeps it in a
usage journal in Redis and writes it when Postgres is back, instead of holding it in memory: the queue no longer fills,
so requests are not held up and records are no longer dropped after 10 s. Another gateway instance takes over the batches
of one that stopped. Gateway:UsageJournal:Mode: Spill (default; no work while Postgres is healthy), Always (every
batch is journaled first, so a crash loses nothing) or Off; MaxRecords (1 000 000) caps it. The operations status adds
journaledBatches. The Redis ACL needs zrangebyscore (read-only). Measured: no change in throughput (audit, 64 clients).

Token quotas. A budget can count tokens instead of USD (Budgets > New budget > Counts: Tokens), optionally only for
requests served by providers of one residency, such as On-prem models only: the way to share on-prem capacity, whose
models cost nothing, between teams and keys. Same alerts, same 402 when used up, a fallback to another residency is not
charged. Reserved in the same Redis call as cost budgets (no extra round trip). Admin API unit, limitTokens,
residency, spentTokens; alerts and the webhook carry token amounts. Migration TokenBudgets.

Improvements taken from a comparison with agentgateway (docs/agentgateway-comparison.md).

Retry-After cooldown. A provider answer with Retry-After or retry-after-ms (a 429 from Azure OpenAI or OpenAI)
puts that model deployment on cooldown for that long, shared by all gateway instances through Redis: it goes last, like
an open circuit, so requests stop paying a wasted round trip to a deployment that is out of quota. It no longer counts
toward the provider's circuit breaker, so a quota on one deployment no longer closes the provider's other deployments.
Capped by CircuitBreaker:MaxRetryAfter (default 5 minutes; 00:00:00 turns it off). No extra Redis round trip.

Least-loaded balancing (opt-in). Gateway:LoadBalancing=LeastLoaded picks the first of several targets with the
same priority by power of two choices over each instance's moving averages of latency and success and its requests in
flight, so a slow, failing or saturated deployment gets less traffic. The default stays Weighted. See
routing rules.

Retries. The last candidate of a request is retried once, after a short jittered delay, when it fails with a
connection error or 502/503/504 (not a timeout, a 429 or a client error): a route with one target no longer fails on a
network blip. Gateway:Retry:MaxRetries (1; 0 turns it off) and Gateway:Retry:BaseDelayMs (250). x-ume-fallbacks and
the usage record count every extra attempt; the all_providers_failed message counts attempts.

Smoother rate limits and a concurrency cap. Requests and tokens per minute are counted over a sliding minute (the
previous minute weighted by how much of it the last 60 seconds cover), so a client can no longer send twice its limit
across a minute boundary; Retry-After says when one more fits; refused requests no longer count. Keys get Requests at
once (maxConcurrentRequests): requests in progress across all instances, so one busy key cannot saturate a shared
on-prem server. Same single Redis round trip. Migration KeyConcurrencyLimit.

Request policies per team and key. A cap on output tokens, a system prompt placed before the application's own,
default parameters and forced parameters (admin UI: Request policy on the team and key forms; admin API requestPolicy).
A key's values win over its team's. Applied after the PII policy and the routing rules, before the budget estimate (which
the cap tightens). Migration RequestPolicies.

Token counting and context windows. POST /v1/messages/count_tokens is passed to Anthropic providers, free (the
Anthropic SDKs offer it; it answered 404 before). A model whose context window the prompt clearly exceeds is skipped, so a larger
fallback model takes long prompts; with none left the answer is 400 context_length_exceeded before anything is sent.

Prompt caching. OpenAI-format requests translated for Claude get Anthropic cache markers on the tools and the system
prompt (Gateway:PromptCaching: Static by default, Conversation or Off). The header x-ume-session (or OpenAI's
prompt_cache_key) keeps a conversation on the same target, so vLLM and Ollama prefix caches are reused.

GenAI metrics (opt-in). With Gateway:GenAiMetrics=true the gateway also records the OpenTelemetry GenAI
semantic-convention metrics (gen_ai.client.token.usage, gen_ai.client.operation.duration,
gen_ai.server.request.duration, gen_ai.server.time_to_first_token) for standard GenAI dashboards. Off by default, so
the request path does no extra work. Always: ume.gateway.cost.usd; ume.gateway.cost (SEK) stays until SEK is removed.

Flaky tests outside the pull request pipeline. Tests that depend on timing or on races between the shared fixtures'
parallel test classes are marked [Trait(TestCategory.Name, TestCategory.Flaky)]. dotnet test still runs every test
locally, and the weekly Flake hunt runs them too; Verify POC runs with --filter-not-trait "Category=Flaky". Four
tests are marked, each with the reason (see Tests that do not flake in the local development guide).

Faster Claude streams through the OpenAI API. The translated stream writes its chunks with Utf8JsonWriter: 683 → 291
µs and 1 270 → 379 KB per 300-chunk stream.

Cache-write prices. A model price can set Cache write (USD per 1M tokens) for providers that charge for writing
the prompt cache. Anthropic cache_creation_input_tokens were billed at the input price; with a cache-write price
(1.25x input for Claude) they are billed correctly. Admin API and config export: cacheWriteInputPerMillionUsd
(optional); Data API prices feed: the same field. Migration CacheWritePrice adds a nullable column.

One verify run per pull request. Verify POC ran twice for every push to a branch with a pull request (on the push
and on the pull request). It now runs on pull requests, on pushes to main, on release tags and by hand (Actions tab,
Run workflow), and a newer push cancels the run still in progress for the same pull request.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.7.0@sha256:ec41258c5094b60f77d7cf4d491bd25cccf6bb7c7bc8581c9c92bfb38f663098
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.7.0@sha256:b9eb08afc0696f7caa7eb4b9e365cd436ae247107b890beea2607e5735acce3f
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.7.0@sha256:cf85559b72105333ef50b655fcf1de87c4dd390c7ed1b5c21b62cae2d75ff570
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.7.0@sha256:336f4d3f442774f2de6ac0efff4df75340a4ca61e23acaa12c22a6c065bc7acd

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.7.0/ume-llm-gateway-0.7.0.tar.gz
tar -xzf ume-llm-gateway-0.7.0.tar.gz && cd ume-llm-gateway-0.7.0
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.6.4

Security hardening from an internal security assessment (findings H1–M4, L2, L4, L5, A3, A5, A6, LG1, LG2, LG4 and LG5),
and a live audio fix. Rolling update: yes (two additive migrations). Read the 0.6.4 section of docs/upgrade-guide.md
first: provider tools and storage are off for every key until an admin allows them, console logs are JSON, everyone is
signed out once, and existing Compose installations should renew the internal CA.

Provider credentials stay with their address. Changing a provider's base URL to another scheme, host or port now
requires entering the credential again (400, field credential); the stored credential is never sent to a new address.
Another path on the same host keeps it. The same applies to configuration import. Before, an edit of the address alone
sent the stored API key to whatever host was entered.

Runtime image 10.0.13. The service images are built on aspnet:10.0.13-noble-chiseled-extra (two critical and five
high ASP.NET Core CVEs in 10.0.12). CI now scans the pinned base image and fails on fixable HIGH/CRITICAL findings.

PII guard reads more of the request. Anthropic plain-text and custom-content documents (document blocks with a source
of type text or content) are scanned like messages; before, a personnummer in such a document passed a key with policy
Block. The request's user field and a message's name are scanned too (tool and function names are not). Text is
compared in a canonical form, so full-width and other non-ASCII digits, no-break spaces, zero-width and other invisible
characters and Unicode dashes no longer hide a personnummer; a redacted value is written in that form.

Admin sessions end when you sign out. Sessions are kept on the server (Redis, encrypted), the cookie only names one:
signing out ends the session for every copy of the cookie. A session lasts at most Admin:MaxSessionHours (12) from
sign-in, however often it is extended; after that the user signs in at the identity provider again, which also picks up
role changes. GET /bff/user and POST /bff/session/extend return sessionLimitAt; extending at the limit answers 409.
Everyone is signed out once by the upgrade.

Floods of made-up keys are throttled. After Gateway:AuthFailuresPerMinute (60) failed authentications in a minute
from one client address (IPv6 per /64), the gateway stops looking keys up for that address until the minute ends: keys
it already has cached keep working, others get 429 rate_limited with Retry-After. 0 turns it off.

Repeated JSON properties are refused. A request body such as {"model":"a","model":"b"} gets 400 invalid_request
(it was a 500 with no usage record). On live audio such an event is answered with an error event, not passed on.

Provider refusals are trimmed. When a provider refuses a request (4xx), the client gets only its message, type,
code and param in the client API's error shape with the gateway's request_id; inner errors, deployment and
organisation names stay out.

A saved change is not reported as failed when Redis is down. The admin API used to answer 500 after a change was
saved if it could not tell the gateways; now it retries once, logs a warning and answers with
Ume-Change-Propagation: delayed, and the admin UI says the gateways apply it when their cache refreshes.

JSON console logs. Outside development the services log JSON (set Logging__Console__FormatterName=simple for plain
text), and model names and Data API paths are logged with control characters replaced, so a client cannot forge log lines.

API reference CSP and switch. /scalar on the gateway, Data API and admin API gets a strict Content-Security-Policy
(scripts only with a per-request nonce). Gateway:PublishApiReference and DataApi:PublishApiReference (default on) turn
the public reference and OpenAPI document off.

Database roles hold only what they use. app-roles.sql now revokes and grants again in one transaction on every
migration-service start, naming each table: the gateway role cannot read the audit log or Data API key hashes or add Data
Protection keys (the gateway no longer generates them; the admin API does), and the admin role can change the settings
row only column by column, without UsageRetainedFrom. A test compares the effective grants, so a new table is readable
by no service until it is granted on purpose.

Conformant internal CA. The generated CA has keyUsage (certificate and CRL signing), and service certificates have
keyUsage and authorityKeyIdentifier, so strict TLS clients (Python 3.13, newer Go and Java) accept the chain.
init-deployment.sh --renew-ca replaces an older CA; the Portainer bootstrap does it on its own. The setup now tells you
to move ca.key off the host.

Provider tools and storage need the key's permission. Two new key settings, both off for every key: Allow provider
tools (web search, file search, code interpreter, MCP, web fetch and other tools the provider runs itself; otherwise only
function tools, else 400 tool_not_allowed) and Allow provider storage (store: true, Responses background and
conversation; otherwise 400 storage_not_allowed, and Responses API requests are sent with store: false). Live audio
refuses mcp tools the same way. Both are in the key form and detail, the admin API (allowProviderTools,
allowProviderStorage), the audit log and the Data API key inventory, and are copied on rotation; refusals are
gateway.request.refused events. Migration ProviderFeaturePolicy.

Admin activity is attributable. Audit entries now record the user's name, roles and departments at the time, the client
address behind the trusted proxies (Admin:AuditSourceAddress, Full by default), a session id given at sign-in (never the
cookie), the request's trace id and the user agent; the audit page shows them and the Data API audit feed passes them on
(migration AuditActorContext). Configuration and usage exports are audit entries. New security events: admin.signin,
admin.signout, admin.session.extended, admin.access.denied (at most once a minute per user, with a count) and
admin.data.read (audit log, request list, PII incidents, Data API keys), with suggested SIEM alerts in
docs/security-events.md.

Live audio sessions stopped by the gateway close with the right code. When the budget ran out, the provider's answer to
the gateway's close could reach the client first: the client got close code 1000 instead of 1008 and the session could be
recorded as a normal close instead of BudgetExceeded. The gateway's own close and outcome now always win.

Website: the icons on the "Control over AI" cards are outlined like the rest of the site, without a fill.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.6.4@sha256:19f08b45067cf6b8f0c60bb072479dcae0972207ff856f1ec5581d5ea95de756
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.6.4@sha256:38442d0d668e864319a9deeedccc685a3ef8c4b660e0f26c07ed7979b393836c
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.6.4@sha256:6e6642d2bf5296e84edbf243ecfe60ff7ab521f89d313ab4ebcfd34983229a95
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.6.4@sha256:74040ce12994334020f42fa9050421d805faea5506f0ee06d33c956d00605ba0

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.6.4/ume-llm-gateway-0.6.4.tar.gz
tar -xzf ume-llm-gateway-0.6.4.tar.gz && cd ume-llm-gateway-0.6.4
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.

Ume LLM Gateway 0.6.3

:latest image tags. Releases also tag the service images :latest on ghcr.io (pre-releases do not), so a Portainer
stack can follow the newest release without editing image variables; see Portainer in the runbook.

Images

GATEWAY_IMAGE=ghcr.io/thekim1/llmgateway/gateway:0.6.3@sha256:7a7bffbde4fa335f903265e7c216bbc681219c5b611b348d1941e9260bfeac79
ADMINAPI_IMAGE=ghcr.io/thekim1/llmgateway/adminapi:0.6.3@sha256:9d1318e5e374eb1654c30ee8faffd5ffc0855417e6445b82740eeb1a37539215
MIGRATIONS_IMAGE=ghcr.io/thekim1/llmgateway/migrations:0.6.3@sha256:ee5e2b101c6e9075a42634f3b4bffd527258f0e164ab11f9c9eea039ac13a2a3
DATAAPI_IMAGE=ghcr.io/thekim1/llmgateway/dataapi:0.6.3@sha256:485323eb56e65ad1f52620e1851398708ff0125071a660c9649f9e03025c5bf2

Deploy with Docker Compose

curl -fsSLO https://github.com/thekim1/llmgateway/releases/download/v0.6.3/ume-llm-gateway-0.6.3.tar.gz
tar -xzf ume-llm-gateway-0.6.3.tar.gz && cd ume-llm-gateway-0.6.3
./deploy/init-deployment.sh
docker compose -f deploy/compose.prod.yaml --env-file deploy/.env up -d --wait

Portainer: paste compose.portainer.yaml from this release into a stack. See docs/runbook.md, Releases.