Skip to content

v1.10.0 — Semantic routing on Messages and Responses, sliding-window rate limits

Latest

Choose a tag to compare

@github-actions github-actions released this 12 Nov 08:42
· 6 commits to main since this release
87c05b8

🚀 Get started

New to AISIX? Get the gateway running and route your first LLM request in minutes:

📦 Download

Pull the container image from the GitHub Container Registry:

docker pull ghcr.io/api7/aisix:1.10.0

The rolling latest tag and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.


AISIX 1.10.0 brings semantic routing to /v1/messages, /v1/messages/count_tokens and /v1/responses, so Claude Code and Codex can use a semantic router. A router can now send traffic to a model group and keep a session on the route its first request matched. Rate limits can count over a sliding window, and a request through a model group now counts each rate-limit layer the same way on every endpoint. The gateway's default log level is now warn.

  • Semantic routers on every LLM endpoint (#1427, #1429): /v1/messages, /v1/messages/count_tokens and /v1/responses honour a semantic router, as /v1/chat/completions already did.
  • Model groups as semantic router targets (#1427): a route, default or failure target can be a model group, which dispatches as it does when addressed directly.
  • Session affinity for semantic routers (#1427, #1429): the new semantic.session_affinity keeps a session on the route its first matched request took.
  • Sliding-window rate limits (#1417): window_type: sliding counts over a rolling window, so a burst across a window boundary can no longer admit up to twice the limit.
  • Codex tools reach chat upstreams (#1419, #1420, #1432): tools declared in additional_tools, namespaced custom tools and Codex's client-side tool_search now go through the Responses → Chat Completions bridge, and Lite-mode requests on non-OpenAI native Responses keys take the bridge.
  • Cost and budget names no longer include a currency (#1425). This changes the budget 429 body, the budget metrics and exported usage events; see Breaking changes.

Control-plane, console and AISIX Enterprise / AISIX Cloud changes in this release (cost breakdown, cache-write pricing, off-peak date ranges, Podman support, control-plane API changes) are described in the AISIX release notes.

⚠️ Breaking changes

resources.yaml has no format break in this release: existing files load as they are. The changes below are in what the gateway sends, and each says what to update.

Cost and budget names no longer include a currency (#1425)

Prices can be in a currency other than USD, so the gateway's output stops naming one. The old names are not sent alongside the new ones.

  • Exported usage events no longer carry a cost. cost_usd is removed from SLS columns, object-store NDJSON, Datadog (aisix.cost_usd), OTLP, and the telemetry sent to a control plane. The gateway sent 0 there except for realtime sessions and completed batch jobs, which it priced locally. That pricing is gone, so a gateway running without a control plane exports no cost for any request. Model prices (cost, pricing_key) still drive least_cost routing as before. Drop cost_usd from queries and dashboards built on these exporters.
  • Budget gauges are renamed. aisix_budget_limit_usd, aisix_budget_spent_usd and aisix_budget_remaining_usd are now aisix_budget_limit, aisix_budget_spent and aisix_budget_remaining, with the same labels. Rename dashboards and alerts that use the old names. If observability.metrics.labels still lists an old name, the gateway starts, logs a warning naming the replacement, and ignores that entry until it is renamed.
  • Budget 429 body. In the OpenAI-format error body, error.limit_usd is now error.limit and error.spent_usd is now error.spent. Values and format do not change. The Anthropic-format body never carried these fields.

Budgets are pushed to the gateway by a control plane, so the last two items apply only to a gateway connected to one.

🧭 Routing

  • Semantic routers on /v1/messages, /v1/messages/count_tokens and /v1/responses (#1427). These endpoints used to answer a semantic model with 400. All four endpoints now share the same fallback rules, the access-log fields semantic_route, semantic_score and semantic_fallback, and the x-aisix-route debug header (with proxy.expose_debug_headers). count_tokens counts against the model the real request would go to.

    • The router classifies the newest user turn that has human text. Tool results, images and <system-reminder> blocks are skipped, and the system prompt and instructions are never classified. With no text anywhere, the request goes to default with semantic_fallback=empty_prompt.
    • For a Codex caller, Codex's injected context blocks (<environment_context>, <user_instructions>, # AGENTS.md instructions for … and similar) are removed before classifying (#1429).
  • Model groups as semantic router targets (#1427). A route's target, default, on_embedding_failure.target and classifier.on_failure.target (and their *_id forms) can name a model group. These loaded before but failed every request. The group's strategy, retries, fallbacks, cooldown, health and when_all_unavailable apply. A group that cannot serve this caller counts as an unavailable target, so the router serves default with semantic_fallback=target_unavailable. Guardrails, model access, the router's allowed_cidrs, the response cache and requested_model stay on the router.

  • Session affinity for semantic routers (#1427, #1429). The new optional block keeps a session on the route its first matched request took, without classifying again:

    semantic:
      session_affinity:
        enabled: true      # default false; an absent block is off
        ttl_seconds: 3600  # default 3600, 60–604800
    • The session id comes from x-aisix-session-id, a vendor x-<vendor>-session-id header such as Claude Code's, Codex's session headers, x-session-id, body metadata.user_id or metadata.session_id, or the session.id entry of W3C baggage. A request without one is classified as usual.
    • A pin is per router, API key and session. The TTL slides on each use. Only a request that matched a route and was sent to its target creates a pin. A pin whose target is no longer one of the router's route targets is dropped.
    • count_tokens and client side requests (Claude Code's x-claude-code-request-class: auxiliary, and known title-generation and suggestion prompts from Claude Code and Codex) follow an existing pin but never create one.
    • Pinned requests carry semantic_pinned=true in the access log.
    • A gateway without a data-plane manager keeps pins in memory: one process, lost on restart, not shared between replicas.

🚦 Rate limits

  • Sliding windows (#1417). rate_limit on API keys and models, mcp_rate_limits values and the limits of conditional rate_limit_policies gain window_type: fixed (default) or sliding. A sliding window sums sub-buckets of 10 seconds for rpm/tpm, 5 minutes for rph and 1 hour for rpd/tpd, so usage is released between one window and one window plus one step after it was recorded. rps and concurrency are not affected, and the classic policy shape (window + max_requests/max_tokens) does not take the field.

    rate_limit:
      rpm: 60
      window_type: sliding
    • On a 429, Retry-After and X-RateLimit-Reset give the exact seconds until the request would fit.
    • The shared Redis backend stores each sliding dimension in one hash under the bucket's existing hash tag, so Redis Cluster works as before. fixed keeps the existing keys, headers and metrics unchanged.
    • Switching a limit between fixed and sliding starts its count from zero. A gateway older than 1.10.0 ignores window_type and enforces a fixed window.
  • Model group attempts are counted per layer (#1426, #1428). See Upgrade notes for what moves.

🔌 Providers and API surface

  • Codex over the Responses → Chat Completions bridge (#1419, #1420). Tools that Codex declares in additional_tools input items (Responses Lite mode), and custom tools inside a namespace, now reach chat upstreams, and calls to them come back as custom_tool_call or function_call items with their name and namespace instead of as text. Codex's client-executed tool_search and the tools it loads now reach chat upstreams too, and calls come back as tool_search_call items. A tool declared more than once is offered once, with the latest definition. Prompt-token estimates and OTLP GenAI tool definitions count these tools.
  • Lite-mode requests on non-OpenAI native Responses keys take the bridge (#1432). See Upgrade notes.

🛡️ Guardrails

  • Grouped card numbers and numbers next to CJK text (#1422). The built-in bank_card detector now matches cards written in 4-4-4-4, 4-4-4-4-3, 4-6-5 and 4-6-4 groups separated by spaces or hyphens, when the number starts with a card prefix and passes Luhn. Every built-in detector now also matches a value written directly next to Chinese, Japanese or Korean text, such as 卡號4111111111111111 or 手机号13812345678. See Upgrade notes.

📊 Observability

  • MCP and A2A failures inside an HTTP 200 are recorded as failures (#1422). A tool call answering isError: true, an upstream HTTP error, timeout, connection failure, JSON-RPC error, undecodable answer, credential failure, ACL refusal or unknown OpenAPI tool on /mcp now records an error_class and a fixed error_message on its usage event and access-log line. The caller still receives HTTP 200. What the tool said never goes into error_message. A2A JSON-RPC errors inside a 200, on message/send and on streams, record upstream_in_band.
    • tool_error requests carry error_source="upstream" on the request counters while status stays 200. Credential failures, ACL refusals and unknown tools read error_source="aisix".
    • An OTLP CLIENT span is now ERROR whenever an error_class is set (except 499). This also applies to a streamed response an output guardrail refuses after its 200 head.
  • Video and realtime failures carry the model (#1414). Failed GET /v1/videos/{id} and GET /v1/videos/{id}/content requests, and /v1/realtime handshakes refused after the model was found, are labelled with the configured model instead of model="unresolved". Realtime refusals after a target was selected also gain provider, upstream_model and the provider-key labels, and the access log gains requested_model.
  • Cache-write retention on usage events (#1432). Usage events gain cache_write_30m_tokens and cache_write_12h_tokens, each present only when an OpenAI-shape upstream reported it. OpenAI returns them on Responses requests that select a cache retention (prompt_cache_options.ttl or a block's prompt_cache_breakpoint.ttl), and not on Chat Completions. Serialized exporters carry them by name, and Datadog and OTLP as aisix.cache_write_30m_tokens and aisix.cache_write_12h_tokens.
  • Exporter pipelines and listeners (#1424). With proxy.thread_per_core on, exporter pipelines run on the control runtime instead of staying on one request worker, which evens out worker load. Every listener (proxy, admin, metrics, debug) requests a backlog of 4096 instead of 128, capped by net.core.somaxconn.
  • Quieter, more useful logs (#1421). usage event dropped is written at most once per reason per 10 seconds, with dropped=<n> counting the drops since the previous line; aisix_usage_event_drops_total still counts every drop. serving last-known-good values for rejected keys moves from INFO to WARN, so the new default level shows it.

🐛 Fixes

  • On Linux kernels older than 6.4, the gateway image no longer panics when it reads the page size (#1421). Automatic heap dumps work again, and /metrics again includes the process_* and aisix_allocator_bytes families. Affected since 1.5.0. Proxy traffic was not affected.
  • config.example.yaml no longer says that _RJEM_MALLOC_CONF=prof_active:false turns heap sampling off (#1423). The official image ignores that variable for its default user, so it never did there. Heap sampling stays on in the Linux build.

⬆️ Upgrade notes

These notes apply when you upgrade from 1.9.x or earlier, in addition to Breaking changes above.

The default log level is warn, and the access log no longer depends on it (#1421)

observability.log_level now defaults to warn instead of info, and the bundled config.example.yaml and config.managed.yaml change with it, so a gateway started from the image's own config file moves too. A gateway that sets neither log_level nor RUST_LOG writes only WARN and above, plus the access log. The INFO lines that go quiet include provider call completed, the lifecycle lines (aisix listening, etcd connected, drain progress) and the guardrail block and monitor-mode diagnostics. To keep the old output, set log_level: info. A script that waits for one of those lines must set info or poll /readyz. The aisix Helm chart's default changes to match.

While observability.access_log is on, the access log is now written at any log_level or RUST_LOG. A gateway that already ran at warn or error and had no access log will now write one. access_log: false is the only way to turn it off: a RUST_LOG that sets aisix_obs::access_log=off no longer does.

Rate limits count model group attempts per layer (#1426, #1428)

A request through a model group now counts each rate-limit layer the same way on every routed endpoint. Direct (non-group) dispatch does not change.

  • Limits on the caller and on what the caller addressed count once per request, as before: the API key's own limit, the addressed model's or group's own limit, classic api_key, team, member and team_member policies, and MCP limits.
  • Limits on the target model the group dispatches to — its inline rate_limit, classic model-scope policies on it, and conditional policies its own model/model_name satisfy — count once per upstream attempt, including retries on the same target. On /v1/messages/count_tokens, completions, embeddings, rerank, images, audio, videos and decisions they used to count once per target model. They now count per attempt, as chat completions, messages and responses already did, so a target model's limits fill faster when retries fire.
  • A conditional policy that matches through the addressed group itself, whichever target model is dispatched, and whose group_by does not split on model or provider, counts once per request — for example a policy with member in [...] AND model in [G] split by member. It used to count again on retries or failover and could run out early. A policy whose match depends on the target model (for example model == G AND NOT provider == X, or model ~= M), or that splits on model or provider, counts per attempt.
  • When the gateway's own limiter refuses a request, the counts it had already taken are returned, so the key's rpm is no longer spent when the model's limit refuses the request. Once an attempt reaches the upstream, nothing is returned, whatever the upstream answers.

Codex Lite-mode requests take the chat bridge on non-OpenAI keys (#1432)

A /v1/responses request that declares tools in additional_tools input items now goes through the Responses → Chat Completions bridge when the dispatched key's provider is anything other than openai, azure or byo, or is unset, even when the key declares apis.responses. This includes a resources.yaml provider key written with any other provider id and apis.responses. Before, such a request went to the native Responses endpoint, where the upstream ignored those items, so the model received no tools and wrote its tool calls as text.

The decision is made per upstream attempt from the key actually dispatched, so a model group can send its first attempt natively to an OpenAI member and its fallback through the bridge. The usage event, access log, metrics and OTLP span report such an attempt as bridged. A rerouted request does not resolve previous_response_id, which matters only for a client that relies on stored responses (Codex sends store: false with the full input every turn). Requests without additional_tools, and every request on OpenAI, Azure OpenAI and bring-your-own keys, do not change.

Semantic routers classify chat requests differently (#1427, #1429)

For an existing semantic router on /v1/chat/completions, an image-only or reminder-only newest user message used to send the request to default (empty_prompt). The router now classifies the newest earlier user turn that has text. <system-reminder> blocks are no longer part of the classified text, several text parts of one message are joined with a space rather than run together, and Codex's injected context blocks are removed for Codex callers.

PII detection masks more text (#1422)

Grouped card numbers and values written next to CJK text now match the built-in detectors, so more text is masked or blocked after upgrading, including an unrelated grouped number with a card prefix that happens to pass Luhn. In the other direction, us_ssn and ip_address written entirely in full-width or other non-ASCII digits no longer match.

Per-model failure metrics on video and realtime routes (#1414)

Because those failures now carry the configured model, per-model error rates on the video follow-up and realtime routes can rise after upgrading, and model="unresolved" drops by the same amount. On a wildcard row (model_name: "*"), the realtime request families now report upstream_model as the row's template rather than the caller's suffix, so per-suffix series stop growing.

MCP and A2A failures move in telemetry (#1422)

Alerts and dashboards that read error_class, error_source or OTLP span status on /mcp and /a2a will now see the in-band failures described under Observability. Caller responses and status codes do not change.

What's Changed

  • fix(telemetry): label video follow-up and realtime refusals with the resolved model by @jarvis9443 in #1414
  • feat(ratelimit): add sliding window_type to windowed rate limits by @jarvis9443 in #1417
  • fix(responses): carry tools declared in additional_tools input items by @jarvis9443 in #1419
  • ci(image): turn off LLVM's MachineOutliner for the instrumented PGO build by @jarvis9443 in #1418
  • fix(responses): carry Codex's client-executed tool search over the chat bridge by @jarvis9443 in #1420
  • fix(obs): stop the auxv panic on old kernels, default the log level to warn, rate-limit usage drop lines by @jarvis9443 in #1421
  • fix: detect grouped bank cards, mark failed MCP and A2A calls in telemetry by @jarvis9443 in #1422
  • docs: stop telling operators _RJEM_MALLOC_CONF turns heap sampling off by @jarvis9443 in #1423
  • fix(obs): run exporter pipelines on the control runtime, raise listener backlog to 4096 by @jarvis9443 in #1424
  • feat!: drop the fixed _usd suffix from user-visible cost names by @jarvis9443 in #1425
  • fix(ratelimit): count Model Group attempts per layer and refund the gateway's own refusals by @jarvis9443 in #1426
  • fix(ratelimit): count a group policy once per request only when every member satisfies it by @jarvis9443 in #1428
  • feat(semantic): route /v1/messages, count_tokens and /v1/responses, allow model-group targets, and pin sessions to their first matched route by @jarvis9443 in #1427
  • fix(semantic): keep client side requests from pinning a session, and strip Codex's injected blocks before classifying by @jarvis9443 in #1429
  • perf(obs): cut the exporter work done per request by @jarvis9443 in #1430
  • revert(obs): back out #1430's exporter optimizations, keep the cancel-spin fix by @jarvis9443 in #1431
  • fix(responses): bridge lite-mode tools on non-OpenAI native Responses keys; feat(usage): OpenAI cache-write retention split by @jarvis9443 in #1432

Full Changelog: v1.9.0...v1.10.0