Repository navigation
🚀 Get started
New to AISIX? Get the gateway running and route your first LLM request in minutes:
- 📖 Documentation — https://docs.api7.ai/ai-gateway/
- ⚡ Gateway quickstart (one container, declarative
resources.yaml) — https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart
📦 Download
Pull the container image from the GitHub Container Registry:
docker pull ghcr.io/api7/aisix:1.10.0The rolling latest tag and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.
AISIX 1.10.0 brings semantic routing to /v1/messages, /v1/messages/count_tokens and /v1/responses, so Claude Code and Codex can use a semantic router. A router can now send traffic to a model group and keep a session on the route its first request matched. Rate limits can count over a sliding window, and a request through a model group now counts each rate-limit layer the same way on every endpoint. The gateway's default log level is now warn.
- Semantic routers on every LLM endpoint (#1427, #1429):
/v1/messages,/v1/messages/count_tokensand/v1/responseshonour a semantic router, as/v1/chat/completionsalready did. - Model groups as semantic router targets (#1427): a route,
defaultor failure target can be a model group, which dispatches as it does when addressed directly. - Session affinity for semantic routers (#1427, #1429): the new
semantic.session_affinitykeeps a session on the route its first matched request took. - Sliding-window rate limits (#1417):
window_type: slidingcounts over a rolling window, so a burst across a window boundary can no longer admit up to twice the limit. - Codex tools reach chat upstreams (#1419, #1420, #1432): tools declared in
additional_tools, namespacedcustomtools and Codex's client-sidetool_searchnow go through the Responses → Chat Completions bridge, and Lite-mode requests on non-OpenAI native Responses keys take the bridge. - Cost and budget names no longer include a currency (#1425). This changes the budget
429body, the budget metrics and exported usage events; see Breaking changes.
Control-plane, console and AISIX Enterprise / AISIX Cloud changes in this release (cost breakdown, cache-write pricing, off-peak date ranges, Podman support, control-plane API changes) are described in the AISIX release notes.
⚠️ Breaking changes
resources.yaml has no format break in this release: existing files load as they are. The changes below are in what the gateway sends, and each says what to update.
Cost and budget names no longer include a currency (#1425)
Prices can be in a currency other than USD, so the gateway's output stops naming one. The old names are not sent alongside the new ones.
- Exported usage events no longer carry a cost.
cost_usdis removed from SLS columns, object-store NDJSON, Datadog (aisix.cost_usd), OTLP, and the telemetry sent to a control plane. The gateway sent0there except for realtime sessions and completed batch jobs, which it priced locally. That pricing is gone, so a gateway running without a control plane exports no cost for any request. Model prices (cost,pricing_key) still driveleast_costrouting as before. Dropcost_usdfrom queries and dashboards built on these exporters. - Budget gauges are renamed.
aisix_budget_limit_usd,aisix_budget_spent_usdandaisix_budget_remaining_usdare nowaisix_budget_limit,aisix_budget_spentandaisix_budget_remaining, with the same labels. Rename dashboards and alerts that use the old names. Ifobservability.metrics.labelsstill lists an old name, the gateway starts, logs a warning naming the replacement, and ignores that entry until it is renamed. - Budget
429body. In the OpenAI-format error body,error.limit_usdis nowerror.limitanderror.spent_usdis nowerror.spent. Values and format do not change. The Anthropic-format body never carried these fields.
Budgets are pushed to the gateway by a control plane, so the last two items apply only to a gateway connected to one.
🧭 Routing
-
Semantic routers on
/v1/messages,/v1/messages/count_tokensand/v1/responses(#1427). These endpoints used to answer a semantic model with400. All four endpoints now share the same fallback rules, the access-log fieldssemantic_route,semantic_scoreandsemantic_fallback, and thex-aisix-routedebug header (withproxy.expose_debug_headers).count_tokenscounts against the model the real request would go to.- The router classifies the newest user turn that has human text. Tool results, images and
<system-reminder>blocks are skipped, and the system prompt andinstructionsare never classified. With no text anywhere, the request goes todefaultwithsemantic_fallback=empty_prompt. - For a Codex caller, Codex's injected context blocks (
<environment_context>,<user_instructions>,# AGENTS.md instructions for …and similar) are removed before classifying (#1429).
- The router classifies the newest user turn that has human text. Tool results, images and
-
Model groups as semantic router targets (#1427). A route's
target,default,on_embedding_failure.targetandclassifier.on_failure.target(and their*_idforms) can name a model group. These loaded before but failed every request. The group's strategy, retries, fallbacks, cooldown, health andwhen_all_unavailableapply. A group that cannot serve this caller counts as an unavailable target, so the router servesdefaultwithsemantic_fallback=target_unavailable. Guardrails, model access, the router'sallowed_cidrs, the response cache andrequested_modelstay on the router. -
Session affinity for semantic routers (#1427, #1429). The new optional block keeps a session on the route its first matched request took, without classifying again:
semantic: session_affinity: enabled: true # default false; an absent block is off ttl_seconds: 3600 # default 3600, 60–604800
- The session id comes from
x-aisix-session-id, a vendorx-<vendor>-session-idheader such as Claude Code's, Codex's session headers,x-session-id, bodymetadata.user_idormetadata.session_id, or thesession.identry of W3Cbaggage. A request without one is classified as usual. - A pin is per router, API key and session. The TTL slides on each use. Only a request that matched a route and was sent to its target creates a pin. A pin whose target is no longer one of the router's route targets is dropped.
count_tokensand client side requests (Claude Code'sx-claude-code-request-class: auxiliary, and known title-generation and suggestion prompts from Claude Code and Codex) follow an existing pin but never create one.- Pinned requests carry
semantic_pinned=truein the access log. - A gateway without a data-plane manager keeps pins in memory: one process, lost on restart, not shared between replicas.
- The session id comes from
🚦 Rate limits
-
Sliding windows (#1417).
rate_limiton API keys and models,mcp_rate_limitsvalues and thelimitsof conditionalrate_limit_policiesgainwindow_type:fixed(default) orsliding. A sliding window sums sub-buckets of 10 seconds forrpm/tpm, 5 minutes forrphand 1 hour forrpd/tpd, so usage is released between one window and one window plus one step after it was recorded.rpsandconcurrencyare not affected, and the classic policy shape (window+max_requests/max_tokens) does not take the field.rate_limit: rpm: 60 window_type: sliding
- On a
429,Retry-AfterandX-RateLimit-Resetgive the exact seconds until the request would fit. - The shared Redis backend stores each sliding dimension in one hash under the bucket's existing hash tag, so Redis Cluster works as before.
fixedkeeps the existing keys, headers and metrics unchanged. - Switching a limit between
fixedandslidingstarts its count from zero. A gateway older than 1.10.0 ignoreswindow_typeand enforces a fixed window.
- On a
-
Model group attempts are counted per layer (#1426, #1428). See Upgrade notes for what moves.
🔌 Providers and API surface
- Codex over the Responses → Chat Completions bridge (#1419, #1420). Tools that Codex declares in
additional_toolsinput items (Responses Lite mode), andcustomtools inside a namespace, now reach chat upstreams, and calls to them come back ascustom_tool_callorfunction_callitems with their name and namespace instead of as text. Codex's client-executedtool_searchand the tools it loads now reach chat upstreams too, and calls come back astool_search_callitems. A tool declared more than once is offered once, with the latest definition. Prompt-token estimates and OTLP GenAI tool definitions count these tools. - Lite-mode requests on non-OpenAI native Responses keys take the bridge (#1432). See Upgrade notes.
🛡️ Guardrails
- Grouped card numbers and numbers next to CJK text (#1422). The built-in
bank_carddetector now matches cards written in 4-4-4-4, 4-4-4-4-3, 4-6-5 and 4-6-4 groups separated by spaces or hyphens, when the number starts with a card prefix and passes Luhn. Every built-in detector now also matches a value written directly next to Chinese, Japanese or Korean text, such as卡號4111111111111111or手机号13812345678. See Upgrade notes.
📊 Observability
- MCP and A2A failures inside an HTTP 200 are recorded as failures (#1422). A tool call answering
isError: true, an upstream HTTP error, timeout, connection failure, JSON-RPC error, undecodable answer, credential failure, ACL refusal or unknown OpenAPI tool on/mcpnow records anerror_classand a fixederror_messageon its usage event and access-log line. The caller still receives HTTP 200. What the tool said never goes intoerror_message. A2A JSON-RPC errors inside a 200, onmessage/sendand on streams, recordupstream_in_band.tool_errorrequests carryerror_source="upstream"on the request counters whilestatusstays200. Credential failures, ACL refusals and unknown tools readerror_source="aisix".- An OTLP CLIENT span is now ERROR whenever an
error_classis set (except 499). This also applies to a streamed response an output guardrail refuses after its 200 head.
- Video and realtime failures carry the model (#1414). Failed
GET /v1/videos/{id}andGET /v1/videos/{id}/contentrequests, and/v1/realtimehandshakes refused after the model was found, are labelled with the configured model instead ofmodel="unresolved". Realtime refusals after a target was selected also gainprovider,upstream_modeland the provider-key labels, and the access log gainsrequested_model. - Cache-write retention on usage events (#1432). Usage events gain
cache_write_30m_tokensandcache_write_12h_tokens, each present only when an OpenAI-shape upstream reported it. OpenAI returns them on Responses requests that select a cache retention (prompt_cache_options.ttlor a block'sprompt_cache_breakpoint.ttl), and not on Chat Completions. Serialized exporters carry them by name, and Datadog and OTLP asaisix.cache_write_30m_tokensandaisix.cache_write_12h_tokens. - Exporter pipelines and listeners (#1424). With
proxy.thread_per_coreon, exporter pipelines run on the control runtime instead of staying on one request worker, which evens out worker load. Every listener (proxy, admin, metrics, debug) requests a backlog of 4096 instead of 128, capped bynet.core.somaxconn. - Quieter, more useful logs (#1421).
usage event droppedis written at most once per reason per 10 seconds, withdropped=<n>counting the drops since the previous line;aisix_usage_event_drops_totalstill counts every drop.serving last-known-good values for rejected keysmoves from INFO to WARN, so the new default level shows it.
🐛 Fixes
- On Linux kernels older than 6.4, the gateway image no longer panics when it reads the page size (#1421). Automatic heap dumps work again, and
/metricsagain includes theprocess_*andaisix_allocator_bytesfamilies. Affected since 1.5.0. Proxy traffic was not affected. config.example.yamlno longer says that_RJEM_MALLOC_CONF=prof_active:falseturns heap sampling off (#1423). The official image ignores that variable for its default user, so it never did there. Heap sampling stays on in the Linux build.
⬆️ Upgrade notes
These notes apply when you upgrade from 1.9.x or earlier, in addition to Breaking changes above.
The default log level is warn, and the access log no longer depends on it (#1421)
observability.log_level now defaults to warn instead of info, and the bundled config.example.yaml and config.managed.yaml change with it, so a gateway started from the image's own config file moves too. A gateway that sets neither log_level nor RUST_LOG writes only WARN and above, plus the access log. The INFO lines that go quiet include provider call completed, the lifecycle lines (aisix listening, etcd connected, drain progress) and the guardrail block and monitor-mode diagnostics. To keep the old output, set log_level: info. A script that waits for one of those lines must set info or poll /readyz. The aisix Helm chart's default changes to match.
While observability.access_log is on, the access log is now written at any log_level or RUST_LOG. A gateway that already ran at warn or error and had no access log will now write one. access_log: false is the only way to turn it off: a RUST_LOG that sets aisix_obs::access_log=off no longer does.
Rate limits count model group attempts per layer (#1426, #1428)
A request through a model group now counts each rate-limit layer the same way on every routed endpoint. Direct (non-group) dispatch does not change.
- Limits on the caller and on what the caller addressed count once per request, as before: the API key's own limit, the addressed model's or group's own limit, classic
api_key,team,memberandteam_memberpolicies, and MCP limits. - Limits on the target model the group dispatches to — its inline
rate_limit, classicmodel-scope policies on it, and conditional policies its ownmodel/model_namesatisfy — count once per upstream attempt, including retries on the same target. On/v1/messages/count_tokens, completions, embeddings, rerank, images, audio, videos and decisions they used to count once per target model. They now count per attempt, as chat completions, messages and responses already did, so a target model's limits fill faster when retries fire. - A conditional policy that matches through the addressed group itself, whichever target model is dispatched, and whose
group_bydoes not split onmodelorprovider, counts once per request — for example a policy withmember in [...] AND model in [G]split bymember. It used to count again on retries or failover and could run out early. A policy whose match depends on the target model (for examplemodel == G AND NOT provider == X, ormodel ~= M), or that splits onmodelorprovider, counts per attempt. - When the gateway's own limiter refuses a request, the counts it had already taken are returned, so the key's
rpmis no longer spent when the model's limit refuses the request. Once an attempt reaches the upstream, nothing is returned, whatever the upstream answers.
Codex Lite-mode requests take the chat bridge on non-OpenAI keys (#1432)
A /v1/responses request that declares tools in additional_tools input items now goes through the Responses → Chat Completions bridge when the dispatched key's provider is anything other than openai, azure or byo, or is unset, even when the key declares apis.responses. This includes a resources.yaml provider key written with any other provider id and apis.responses. Before, such a request went to the native Responses endpoint, where the upstream ignored those items, so the model received no tools and wrote its tool calls as text.
The decision is made per upstream attempt from the key actually dispatched, so a model group can send its first attempt natively to an OpenAI member and its fallback through the bridge. The usage event, access log, metrics and OTLP span report such an attempt as bridged. A rerouted request does not resolve previous_response_id, which matters only for a client that relies on stored responses (Codex sends store: false with the full input every turn). Requests without additional_tools, and every request on OpenAI, Azure OpenAI and bring-your-own keys, do not change.
Semantic routers classify chat requests differently (#1427, #1429)
For an existing semantic router on /v1/chat/completions, an image-only or reminder-only newest user message used to send the request to default (empty_prompt). The router now classifies the newest earlier user turn that has text. <system-reminder> blocks are no longer part of the classified text, several text parts of one message are joined with a space rather than run together, and Codex's injected context blocks are removed for Codex callers.
PII detection masks more text (#1422)
Grouped card numbers and values written next to CJK text now match the built-in detectors, so more text is masked or blocked after upgrading, including an unrelated grouped number with a card prefix that happens to pass Luhn. In the other direction, us_ssn and ip_address written entirely in full-width or other non-ASCII digits no longer match.
Per-model failure metrics on video and realtime routes (#1414)
Because those failures now carry the configured model, per-model error rates on the video follow-up and realtime routes can rise after upgrading, and model="unresolved" drops by the same amount. On a wildcard row (model_name: "*"), the realtime request families now report upstream_model as the row's template rather than the caller's suffix, so per-suffix series stop growing.
MCP and A2A failures move in telemetry (#1422)
Alerts and dashboards that read error_class, error_source or OTLP span status on /mcp and /a2a will now see the in-band failures described under Observability. Caller responses and status codes do not change.
What's Changed
- fix(telemetry): label video follow-up and realtime refusals with the resolved model by @jarvis9443 in #1414
- feat(ratelimit): add sliding window_type to windowed rate limits by @jarvis9443 in #1417
- fix(responses): carry tools declared in additional_tools input items by @jarvis9443 in #1419
- ci(image): turn off LLVM's MachineOutliner for the instrumented PGO build by @jarvis9443 in #1418
- fix(responses): carry Codex's client-executed tool search over the chat bridge by @jarvis9443 in #1420
- fix(obs): stop the auxv panic on old kernels, default the log level to warn, rate-limit usage drop lines by @jarvis9443 in #1421
- fix: detect grouped bank cards, mark failed MCP and A2A calls in telemetry by @jarvis9443 in #1422
- docs: stop telling operators _RJEM_MALLOC_CONF turns heap sampling off by @jarvis9443 in #1423
- fix(obs): run exporter pipelines on the control runtime, raise listener backlog to 4096 by @jarvis9443 in #1424
- feat!: drop the fixed _usd suffix from user-visible cost names by @jarvis9443 in #1425
- fix(ratelimit): count Model Group attempts per layer and refund the gateway's own refusals by @jarvis9443 in #1426
- fix(ratelimit): count a group policy once per request only when every member satisfies it by @jarvis9443 in #1428
- feat(semantic): route /v1/messages, count_tokens and /v1/responses, allow model-group targets, and pin sessions to their first matched route by @jarvis9443 in #1427
- fix(semantic): keep client side requests from pinning a session, and strip Codex's injected blocks before classifying by @jarvis9443 in #1429
- perf(obs): cut the exporter work done per request by @jarvis9443 in #1430
- revert(obs): back out #1430's exporter optimizations, keep the cancel-spin fix by @jarvis9443 in #1431
- fix(responses): bridge lite-mode tools on non-OpenAI native Responses keys; feat(usage): OpenAI cache-write retention split by @jarvis9443 in #1432
Full Changelog: v1.9.0...v1.10.0