v0.9.0 — Thread-per-core proxy, semantic caching, and agent observability

🚀 Get started

New to AISIX? Get the gateway running and route your first LLM request in minutes:

• 📖 Documentation — https://docs.api7.ai/ai-gateway/
• ⚡ Gateway quickstart (one container, declarative resources.yaml) — https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart

📦 Download

Pull the container image from the GitHub Container Registry:

docker pull ghcr.io/api7/aisix:0.9.0

Rolling tags (0.9, latest) and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.

---

A release about what the gateway can tell you — and how fast it gets out of the way.

The proxy now serves from thread-per-core workers, roughly doubling throughput and halving p99. Caching learned to match requests that mean the same thing rather than only those spelled the same way. Agent calls stopped being opaque: an A2A call now records which task it advanced, how its stream ran, and what it cost. And a JWT can now stand in for an API key, so an identity provider's claims decide which caller a request runs as.

Highlights

• Thread-per-core proxy — 54–88% more throughput and about half the p99 latency, on by default on Linux (#891).
• Semantic caching — serve a cached answer to a differently-worded question, in-process or shared across replicas on Redis vector search (#918, #921).
• JWT claim mappings — resolve verified OIDC claims to an existing caller API key, inheriting its access, limits and budget (#936).
• A2A protocol observability — operation, task, context and task state on every call, streaming figures, per-agent metrics, and usage metered from the words that passed through (#930, #933, #937).
• The provider's own response id in the access log and in a per-attempt log line, so a provider-side problem can be traced without joining systems by hand (#935).
• Day-long rate-limit windows (#949) and quota conditions that match a model group by its own id (#957).

⚡ Performance

The proxy hands a request between threads about twice per request; each handoff costs a wakeup and a context switch. It now serves from N independent workers, each with its own runtime, SO_REUSEPORT listener and upstream connection pool, so a request is accepted, dispatched and answered on one thread. Syscalls per request drop from 11.9 to 5.0 and context switches from 232k/s to 222/s. On four pinned vCPUs: 45,205 rps versus 24,064 at c=128 (+88%), with p99 roughly halved (#891).

Two bootstrap knobs, applied at startup: proxy.thread_per_core (on for Linux, off elsewhere) and proxy.workers (defaults to the parallelism available to the process, following a cgroup limit or affinity mask). Below about four client connections per worker the kernel's spreading leaves workers uneven, so very low concurrency is slower than the shared runtime — set proxy.thread_per_core: false there.

Alongside it: a jemalloc global allocator and fat-LTO release profile (#939), zero-config fast paths on the per-request hot path (#944), a per-worker metrics handle cache (#925), TCP_NODELAY on accepted downstream connections (#948), cached upstream endpoint URLs (#945, #955), and one config-snapshot load per request (#953).

🗄️ Caching

A cache policy carrying a semantic block embeds a request that misses exact matching and serves the nearest stored entry at or above its cosine threshold. Only fully textual requests take that path — anything with images, audio or tool calls always matches exactly (#918).

With backend: redis, entries and their vectors live in the operator's existing cache.redis and are shared across gateway replicas (#921). This needs a Redis with vector search — Redis 8 or later, or the search module. The gateway probes support once at boot; without it, backend: redis policies keep serving exact matches and say so in the log rather than failing traffic.

Cache entries are also now scoped, and a policy can be emptied without deleting it: bumping purge_generation retires every entry at once.

🔐 Identity

A new environment-scoped claim_mappings resource generalises JWT identity beyond a one-to-one subject binding. After full OIDC verification, when no key binds the token's subject directly, the provider's enabled mappings are evaluated in (priority, name, id) order and the first whose claim conditions all hold selects the API key the request runs as — inheriting its model and tool access, rate limits and budget unchanged. Operators get exact (string) and contains (array) over dotted claim paths. A direct subject binding stays authoritative. Anything unmatched, ambiguous or dangling is denied (#936, #938).

Usage events gain jwt_subject, jwt_provider and jwt_claim_mapping attribution. Note that jwt_subject is an end-user identifier and now reaches configured observability exporters.

🔭 Observability

A2A calls are no longer a single "someone reached agent X" fact. Every call records its operation, task id, context id and final task state, with the two wire vocabularies normalised so one figure is not split in two (#930). Streamed calls record time to first event, event counts and whether the caller was still listening at the end — a caller that hangs up mid-task is no longer recorded as a success (#933). Because the protocol carries no usage block, the gateway meters a call from the message text that passed through it, marked usage_estimated; artifact continuations are reassembled using the protocol's own append flag rather than concatenated or collapsed (#937). Metrics are sliced by agent and operation.

The provider's own response id (chat.completion.id, an Anthropic message id, a Responses resp_…) now appears in the access log and in a provider call completed line emitted once per upstream attempt — covering streamed responses and failed-over attempts that the one-line-per-request access log structurally cannot (#935).

Also: post-authentication refusals on the realtime endpoint are attributed to the resolved caller (#947), and TTFT/latency comments that misdescribed streaming were corrected (#946).

🚦 Traffic controls

Rate-limit policies accept a day window, counted against the UTC day (#949).

A quota condition on model now matches the requested entry as well as the dispatched one, so a Model Group's own id selects every request addressed to that group — previously such a condition could never match, because only the dispatched target was compared (#957). A semantic router honours its members' own gates when selecting a route, falling through instead of dispatching to a member the caller cannot use (#958).

🧩 Model kinds

Wildcard model rows now key their gates, health tracking and telemetry on the row itself rather than on the alias a caller happened to send, so one row is one identity instead of a new series per spelling (#959).

Knobs a kind's runtime never reads are no longer accepted in silence. retries, auto_prompt_caching and cost on a routing parent, the generic set on an ensemble parent, and auto_prompt_caching/cost on a semantic parent are rejected by the strict schema at write time; a row already stored with one loads with the field stripped and reported through the partially-compatible channel, so an upgrade never takes a working group out of service (#963).

⚠️ Breaking changes

The Admin API no longer writes resources (#915). The admin listener keeps its whole read surface — lists and gets for all eight resource kinds, /admin/v1/models/status, health, OpenAPI and the playground — but resource writes are gone. This completes the deprecation announced in v0.4.0.

Before	After
POST /admin/v1/<kind>, PUT/DELETE /admin/v1/<kind>/{id}	405 with Allow: GET
POST /admin/v1/api_keys/{id}/rotate (and the apikeys spelling)	404 — the route is gone
Rotate returned a fresh plaintext key	Rotate declaratively: write the same resource id with a new key_hash; the old plaintext stops authenticating as soon as the write propagates

Manage resources through a resources_file (validate with aisix validate --resources <file>, reload with SIGHUP) or by writing entity-value JSON to etcd. The published OpenAPI no longer documents write operations, and the write-only component schemas are gone.

Cached entries are isolated per caller API key by default. A cache policy's scope defaults to api_key, so one caller's answers are never replayed to another. Deployments that relied on environment-wide sharing must set scope: env explicitly; either way the cache-key shape changes, so expect a one-time full miss after upgrading.

Time to first token is stamped on the first streamed frame of any type. It previously waited for a frame carrying generated output, which on a model that thinks silently recorded the end of the thinking phase — a figure that could exceed the request's own reported latency. The new figure is comparable with what a fronting proxy measures; silent thinking time remains visible in upstream latency. Monitoring and alerting calibrated against the old figure will shift.

A caller-supplied request id is now adopted as the gateway's own. proxy.request_id.accept_headers defaults to ["x-aisix-request-id"], so a client sending that header sees it echoed back and recorded in logs, usage events and the id forwarded upstream. Ids must be 1–256 visible-ASCII bytes; anything else is ignored and the gateway generates its own. Add x-request-id to the list to honour that convention too. Because the id is now caller-controlled, it is not unique and must never be used as a metric label. Set accept_headers: [] to restore the previous behaviour.

Configuration that a model kind never reads is now rejected rather than stored and ignored — see Model kinds above. A resources file carrying one fails at boot with the offending row named.

Wildcard traffic reports the row's name. Successful requests through a wildcard row now carry the row's own name in the model metric label instead of the alias the caller sent. Dashboards filtering on a caller-minted alias will show a series ending where this release begins.

What's Changed

• feat(proxy): serve from thread-per-core workers with per-worker upstream pools by @membphis in #891
• fix(config): make every sequence field reachable from the environment by @jarvis9443 in #912
• fix(core): enforce the a2a/mcp name and credential rules on every config path by @moonming in #848
• fix(a2a): announce the pinned wire version and find path-hosted agent cards by @jarvis9443 in #914
• feat(a2a): relay message/stream as live SSE instead of buffering it by @jarvis9443 in #916
• bench: onthebench-style rig harness for same-rig baselines by @membphis in #917
• feat(cache): semantic (embedding-similarity) matching layer on cache policies by @jarvis9443 in #918
• feat(cache): shared semantic store on redis vector search by @jarvis9443 in #921
• bench(onthebench): shared measurement lib, generic entrant runner, method overrides by @membphis in #919
• feat(admin)!: remove the Admin API resource write path by @moonming in #915
• test(cache): bounded-retry post-store lookups against RediSearch index lag by @moonming in #923
• fix(obs): carry the full error source chain in sink failure details by @jarvis9443 in #927
• test: freeze the unit-test rate-limit clock; widen the audio-timeout e2e margin by @jarvis9443 in #926
• build(deps): bump undici from 6.27.0 to 6.28.0 in /tests/e2e by @dependabot[bot] in #873
• test(auth): self-diagnose an empty log capture by @jarvis9443 in #928
• perf(metrics): per-worker handle cache for every per-request emit path by @membphis in #925
• fix(obs): stamp upstream TTFT on the first streamed frame of any type by @jarvis9443 in #929
• feat(a2a): record the operation, task, context and task state of every call by @jarvis9443 in #930
• feat(a2a): observe how a streamed call ran, and meter calls per agent by @jarvis9443 in #933
• feat(a2a): meter a call by the words that passed through it by @jarvis9443 in #937
• feat(auth): claim_mappings — resolve verified JWT claims to an existing API key by @jarvis9443 in #936
• feat(proxy): reuse a caller-supplied request id as the gateway's own by @jarvis9443 in #934
• perf(server): jemalloc global allocator + fat LTO release profile by @membphis in #939
• refactor(auth): route claim-mapping rejections through deny; decouple the e2e readiness gate by @jarvis9443 in #938
• feat(obs): record provider_request_id in the access log and the plain log by @jarvis9443 in #935
• perf: zero-config fast paths for the per-request hot path by @membphis in #944
• perf(gateway): cache parsed upstream endpoint URLs per ProviderKey by @membphis in #945
• docs(obs): correct three comments that mislead about streaming by @jarvis9443 in #946
• fix(realtime): attribute post-auth refusals to the resolved caller by @jarvis9443 in #947
• feat(ratelimit): add a Day variant to PolicyWindow by @jarvis9443 in #949
• perf(proxy): load the config snapshot once per request and resolve the provider key once per emit by @membphis in #953
• perf(server): set TCP_NODELAY on accepted downstream connections by @membphis in #948
• perf(gateway): cache endpoint URLs on the A2A, Vertex and Azure bridges by @membphis in #955
• fix(quota): match model conditions against the requested parent as well as the dispatched target by @jarvis9443 in #957
• fix(semantic): honor member gates at route selection by @jarvis9443 in #958
• fix(wildcard): key gates and telemetry on the wildcard row, not caller-minted aliases by @jarvis9443 in #959
• build(deps): clear all 8 open dependabot alerts by @membphis in #960
• fix(schema): reject per-kind dead knobs at write time and strip them on lenient loads by @jarvis9443 in #963
• docs(agents): Model Kinds Stay in Lockstep rule by @jarvis9443 in #964
• bench(onthebench): pin the dwarf stack dump and flag flamegraphs that did not unwind by @membphis in #965
• bench(onthebench): verify that the process being measured is the one this run started by @membphis in #966
• chore(review): land the open review-thread fixes from the model-kind batch by @jarvis9443 in #969

Full Changelog: v0.8.2...v0.9.0