Skip to content

v1.6.0 — Classifier routing and readable metrics

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Oct 18:29
· 124 commits to main since this release
v1.6.0
1a20831

🚀 Get started

New to AISIX? Get the gateway running and route your first LLM request in minutes:

📦 Download

Pull the container image from the GitHub Container Registry:

docker pull ghcr.io/api7/aisix:1.6.0

The rolling latest tag and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.

AISIX 1.6.0 lets a semantic router decide with a classification model, puts readable names on metrics, and makes what the gateway records describe what actually happened. A semantic router can pick its route with TypeSafe's jev decision model instead of embedding similarity. Prometheus series can carry API key, team, provider key and policy names beside their ids, and usage events can record request headers you choose. Streamed responses are counted in the request metrics when they end, under the status they ended on, and every refusal the gateway makes itself is named on both the access log and the usage event. Function calling now works end to end on Vertex AI Gemini, reasoning crosses the Anthropic Messages and chat bridges, and passthrough routes enforce their boundaries, their concurrency and their guardrails for the whole stream.

  • jev classifier for semantic routers (#1257) — a semantic model can carry a classifier block and route by each route's description instead of examples and thresholds; every semantic router's access-log line now names its decision.
  • Readable name labels on metrics (#1268) — opt-in api_key_name, team_name, provider_key_name and policy_name labels beside the id labels.
  • Request headers on usage events (#1269) — observability.usage_event.request_headers records the headers you list on every usage event.
  • Streamed responses are recorded at stream end with their real status (#1273), and gateway refusals carry their reason on the usage event and the access log (#1275, #1279).
  • Tools work on Vertex AI Gemini, and Claude tool calls reach streaming OpenAI-protocol clients (#1277, #1280, #1276); reasoning crosses the Messages/chat bridges (#1267), and Claude models that reject the effort parameter get a thinking budget for reasoning_effort (#1274).
  • Passthrough routes hold their boundaries (#1262) — unsafe paths and conflicting queries are refused, redirects are relayed, timeout_ms bounds event streams, concurrency is held for the whole stream, and output guardrails fail closed on a response they cannot evaluate.

⚠️ Breaking changes

  • Access-log error_kind and usage-event error_class share one vocabulary (#1273). Non-streamed access-log lines, and terminal usage events of failed requests that carry no attempt record, change these values: upstream_error → upstream_status, transport_error → transport, upstream_decode_error → upstream_decode, upstream_in_band_error → upstream_in_band, config_error → config, invalid_request_error → invalid_config (invalid upstream configuration) and authentication_error → invalid_credentials. Per-attempt events and streamed lines already used the new names. Log and event pipelines that match the old values must accept the new ones. The error.type a client receives is unchanged. A streamed passthrough line now carries an error_kind where it used to be empty.
  • A semantic router with on_embedding_failure: fail answers a failed embedding with its own error code (#1274). It still answers 503, but with type: api_error, code: semantic_routing_failed and the message semantic router '<model>' could not pick a route (<reason>), where the reason is embedding_timeout, embedding_upstream or embedding_unavailable, instead of type: provider_unavailable ("no bridge registered for provider"). Clients that matched error.type == "provider_unavailable" for this case must match error.code == "semantic_routing_failed". The access log and usage event carry error_kind / error_class semantic_decision_failed. The jev classifier's on_failure: fail answers the same way, with classifier_* reasons.
  • Function calling on Vertex AI Gemini models refuses what it cannot carry (#1277, #1284). Requests that used to return 200 with tools flattened to text or dropped now answer 400 invalid_request_error: the legacy functions / function_call fields, a tool that is not a function tool, an unsupported tool_choice, a tool result that answers no call, and call arguments that are not JSON. Bedrock also refuses functions / function_call, which Converse used to drop. Move clients to tools / tool_choice. Tool call ids from Vertex Gemini models are longer, because they carry Gemini's thought signature, and must be sent back unchanged; the gateway strips that suffix before sending a conversation to any other upstream.

🧭 Routing and providers

  • jev classifier for semantic routers (#1257). A semantic model can carry a classifier block (type: jev, a typesafe provider key, model, min_confidence, timeout_ms, on_failure). Each route then needs a description and takes no examples or thresholds. A pick below min_confidence goes to the default model. Access-log lines of every semantic router now carry semantic_route, semantic_score and semantic_fallback.
  • Tools work on Vertex AI Gemini models (#1277). A vertex model with a Gemini publisher used to drop tools and tool_choice, send tool results as user text, and lose assistant tool calls from history, so a tool request got a prose answer. Function calls now go both ways, streaming or not, on /v1/chat/completions, /v1/messages and /v1/responses.
  • Claude tool calls reach streaming OpenAI-protocol clients (#1280). A streamed answer from Claude, through Anthropic or Vertex AI, used to end with finish_reason: tool_calls and no tool calls. It now carries delta.tool_calls. /v1/responses streams emit function_call items, and /v1/messages streams to Claude on Vertex AI keep their tool_use blocks.
  • Reasoning crosses the Anthropic Messages and chat bridges (#1267). An Anthropic client on /v1/messages served by an OpenAI-compatible reasoning upstream now sees a thinking block, or thinking events when streaming, before the answer, with signature: "". Its replayed thinking reaches the upstream as reasoning_content, so DeepSeek-style upstreams no longer reject tool-using follow-up turns. Unsigned thinking blocks are removed before a request goes to Anthropic's own API, so a session that fails over or switches to Claude no longer gets an invalid-signature 400. An OpenAI client served by Anthropic with extended thinking now receives reasoning_content, and /v1/responses served by Anthropic returns a reasoning item. Claude on Bedrock or Vertex AI reached through /v1/messages also surfaces its thinking, unsigned.
  • reasoning_effort works on Claude models that reject the effort parameter (#1274). Haiku 4.5, Sonnet 4.5 and earlier Claude models answered reasoning_effort on /v1/chat/completions and reasoning.effort on /v1/responses with a 400. On those models the gateway now sends an extended-thinking budget instead: 1024 tokens for minimal and low, 2048 for medium, 4096 for high, 8192 for xhigh and 16384 for max. A request without max_tokens gets the budget plus 4096; a caller's max_tokens is kept and caps the budget below it, and at 1024 or less thinking stays off. This covers Anthropic, Vertex AI and Bedrock, and an operator effort mapping on /v1/messages and /v1/messages/count_tokens. A caller's own thinking or output_config.effort still wins, and none still disables thinking.

🔀 Passthrough routes

  • Boundaries are enforced (#1262). A path-traversal segment, or a caller query key that collides with a key in target_url's own query, is rejected with 400 at the gateway, including encoded variants and an encoded & or ;. A target_url with its own query composes correctly: the operator's parameters are kept and the caller's are appended. Upstream 3xx responses are relayed to the caller instead of followed, and credentials are never carried to the Location.
  • Event streams are bounded and counted for their whole length (#1262). On an event stream, timeout_ms bounds the wait for the first byte and every later gap with no bytes, falling back to the deployment's stream_timeout_ms, then request_timeout_ms; 1.5.0 applied no timeout to passthrough event streams. A stream holds its concurrency slot until it ends or the client cancels, so a key's or team's concurrency now really limits concurrent passthrough streams. With the Redis rate-limit backend, a stream whose concurrency lease disappears from Redis is cut with 429 and error_class: rate_limit_lease_lost.
  • Guardrails scan what they can prove (#1262). Output guardrails fail closed with 422 unscannable_body unless output_fail_open is set, for a success response that is still compressed and for a stream or body they cannot evaluate. Responses are relayed undecoded; when output guardrails apply, the gateway asks the upstream for an identity encoding, and upstream non-2xx responses bypass output guardrails. Streamed output guardrails judge each source channel separately, scan refusals and legacy function_call.arguments, and skip media and reasoning. Input guardrails scan only the decoded JSON string values of a parseable raw JSON body; malformed JSON, more than 256 KiB of strings, or a body nested too deep fails closed unless fail_open is set.

🛡️ Guardrails

  • A guardrail's timeout_ms bounds the whole call (#1271) on Aliyun text moderation, Aliyun AI Guardrails, Lakera, OpenAI moderation, Presidio, and Azure Prompt Shield and text moderation: sending, reading the body (error bodies included) and decoding it. A provider that sends headers and then stalls its body now fails as a timeout and follows fail_open / output_fail_open, where the request used to hang.
  • Moderation-style guardrails no longer receive tool-call ids (#1278). Aliyun, Lakera, Bedrock, OpenAI moderation and custom guardrails used to receive each tool call's id with its name and arguments. Tool call id, call_id, tool_use_id and tool_call_id values are now left out, on every endpoint and on passthrough routes, so such a guardrail no longer blocks on text found only inside an id. Function names and arguments are scanned as before; keyword and pii guardrails already skipped ids.
  • Output guardrails on typed /v1/responses also scan refusals and terminal and aggregate events (#1262), and PII masking covers them. max_buffer_bytes now also counts events that carry content without deltas.
  • Guardrail failure logs say why (#1271, #1272). A failed remote guardrail call logs http_status, the full error chain, error_kind and elapsed_ms. Custom guardrail aisix.embed, semantic guardrail, semantic router and semantic cache embedding failures log their cause too. None of these logs repeat an upstream response's text, which can quote the prompt.

📊 Metrics, logs and usage events

  • Streamed responses are recorded in the request metrics when the stream ends, with the status it ended on (#1273). This covers /v1/chat/completions, /v1/messages and /v1/responses streams, the transcription and translation event streams, /v1/audio/speech and streamed /a2a calls. A stream delivered whole is 200, one the caller leaves mid-way is 499, an upstream break or in-stream error is 502, a read timeout is 504, and an in-stream 4xx keeps its status; such streams used to count as 200 at response start. Affected families are aisix_requests_total, aisix_request_duration_seconds, aisix_proxy_requests_total, aisix_proxy_failed_requests_total, aisix_proxy_request_duration_seconds, aisix_llm_requests_total and aisix_llm_request_duration_seconds. aisix_request_e2e_latency_seconds and aisix_request_ttft_seconds now carry the stream's real status_class instead of a fixed 200.
  • Readable name labels (#1268). observability.metrics.labels can select api_key_name and team_name on every family that carries api_key_id or team_id, provider_key_name on the deployment counters, and policy_name on aisix_ratelimit_rejections_total. user_name now follows a member's rename. None is on by default, so existing scrapes are unchanged. Renaming a key, team or member retires the gauge series under the old name. The names come from the new teams and users configuration kinds.
  • Request headers on usage events (#1269). observability.usage_event.request_headers lists request headers that every usage event records in request_headers, for example a sub-user or department header on a shared key. SLS, Datadog and object storage carry the map, and OTLP carries one http.request.header.<name> attribute per header. Credential headers are refused at startup.
  • Gateway refusals carry their reason (#1275, #1279). A request the gateway refuses itself (an unknown or forbidden model, an IP restriction, a rate-limit or budget refusal, a guardrail block, a failed semantic decision, a request that is too large) now records the access log's text in the terminal usage event's error_message and in the OTLP aisix.error_message. Errors whose text could quote the request or the upstream's answer leave it empty. Output- and session-ending guardrail refusals on /v1/chat/completions, /v1/responses, /v1/completions, the audio routes and /v1/realtime now carry error_class: content_filter; a streamed answer refused after its 200 head keeps status 200 with that class. /a2a and /mcp guardrail blocks and quota rejections carry error_class and the message too, and for every gateway refusal the access-log error_kind and error now equal the usage event's error_class and error_message.
  • Gateway-initiated embedding calls on usage events (#1265). Embedding calls the gateway makes for semantic routing, semantic guardrails and semantic-cache misses are listed on the request's usage event as gateway_embedding_calls, up to 64, with how many were dropped and whether the upstream reported usage. The request's own model, tokens and cost are unchanged. Embedding responses with missing or partial token counts are kept and marked "usage unavailable" instead of failing.
  • Every request the proxy listener answers writes one access-log line (#1260), including authentication denials (401, and the JWT-scope 403), unrouted paths (404), a wrong method on a known path (405), GET /v1/models, A2A agent-card fetches and OAuth protected-resource documents. The lines never contain a presented credential. /livez and /readyz still write none.
  • observability.access_log: false turns the access log off (#1259). The key was parsed but ignored. The default remains true.

⚙️ Configuration

  • A config file without an observability: block logs at info (#1256), as documented, instead of falling back to errors only with no startup lines and no access log. RUST_LOG still wins when set.
  • Sequence settings can be set from AISIX_* environment variables as comma-separated lists (#1251): observability.heap_profiling.auto_dump.thresholds, cache.redis.nodes, cache.redis.sentinels, ratelimit.redis.nodes and ratelimit.redis.sentinels. Setting any of them used to stop the gateway at startup.

🐛 Fixes

  • Google AI Studio embedding models answered every /v1/embeddings call with 502 upstream_decode_error, because Gemini omits the first item's index (#1276).
  • Gemini and other OpenAI-compatible upstreams that omit the tool call index had streamed parallel tool calls merged into one on /v1/chat/completions, /v1/messages and /v1/responses; streamed chat-completions deltas now always carry index (#1276).
  • Unknown- or forbidden-model refusals on /v1/realtime and files/batches wrapped their sentence twice; they now read model "x" not found / API key is not allowed to use model "x", and transcription, translation and files/batches refusals record the requested model on the usage event (#1275).
  • Redis concurrency leases are renewed during long requests on every endpoint with concurrency, so a request that outlives the lease TTL no longer lets extra requests in (#1262).
  • API-version detection on passthrough routes works when the destination URL carries a query (#1262).

⬆️ Upgrade notes

  • Review dashboards and alerts built on the request-metric families for streaming traffic. Success rates fall to what actually happened, duration percentiles rise to whole-stream length (time to first token stays on the TTFT metrics), and rate() signals lag a burst of long streams by about one stream's length, including the chart's KEDA example on aisix_llm_requests_total.
  • Log and event pipelines that match error_kind or error_class must accept the new names under Breaking changes, and content_filter on output-side and session-ending guardrail refusals, which 1.5.0 left empty.
  • An internet-facing gateway logs noticeably more, since authentication denials and unrouted paths now write access-log lines. Filter in the log pipeline or turn the access log off.
  • Remove a leftover observability.access_log: false to keep the access log. It used to be ignored and now turns the access log off.
  • A gateway whose config has no observability: block now logs at info. Set observability.log_level: error to keep the old silence.
  • A guardrail timeout_ms tuned to header latency alone may need raising, since it now covers reading and decoding the body.
  • Re-check moderation-style guardrail rules that matched text inside a tool-call id; they stop firing.
  • Passthrough routes: callers sending traversal segments or query keys that collide with target_url's query get 400; 3xx responses reach the caller instead of being followed; keyword rules that matched JSON keys or structure in a raw JSON body stop matching; and output guardrails that cannot evaluate a response refuse it with 422 unscannable_body unless output_fail_open is set.
  • Gateways from 0.12.0 through 1.3.x report the new teams and users configuration kinds as rejected in aisix_config_rejected_resources and hold aisix_config_last_reload_successful at 0 until upgraded; configuration loading is unaffected. Gateways from 1.4.0 count them in aisix_config_unknown_kind_resources without failing the reload.

What's Changed

  • fix(config): make every sequence field settable from AISIX_* env by @jarvis9443 in #1251
  • feat(config): publish config.reference.json for the Helm chart's config drift check by @jarvis9443 in #1253
  • feat(config): spell out optional blocks in config.reference.json by @jarvis9443 in #1255
  • fix(config): an omitted startup-config block takes the same defaults as an empty one by @jarvis9443 in #1256
  • feat(semantic): jev classifier backend for semantic routers, with the decision on the access log by @jarvis9443 in #1257
  • feat(obs): make observability.access_log turn the access log off by @jarvis9443 in #1259
  • fix(proxy): write an access-log line for every request the proxy surface answers by @jarvis9443 in #1260
  • docs(semantic): drop stale references to a gateway-added classifier option by @jarvis9443 in #1261
  • docs(schema): describe lazy semantic route embeddings by @moonming in #1263
  • fix(telemetry): audit gateway embedding child usage by @moonming in #1265
  • fix(passthrough): preserve route and stream boundaries by @moonming in #1262
  • fix(anthropic): carry reasoning across the Messages/Chat protocol bridge by @jarvis9443 in #1267
  • feat(metrics): selectable name labels beside every id label; teams and users kinds by @jarvis9443 in #1268
  • feat(telemetry): record operator-selected request headers on every usage event by @jarvis9443 in #1269
  • docs(claude-md): label held charts/aisix PRs with aisix-hold-for-release by @jarvis9443 in #1270
  • fix(guardrails): bound guardrail calls by timeout_ms end to end and log why they fail by @jarvis9443 in #1271
  • fix(guardrails): log custom embed failure causes and keep upstream response text out of failure logs by @jarvis9443 in #1272
  • fix(metrics): record streamed responses at stream end with their terminal status; one error class vocabulary by @jarvis9443 in #1273
  • fix: thinking budget for Claude families that reject effort; name failed semantic routing decisions by @jarvis9443 in #1274
  • fix(usage): gateway-side failures carry their message on the terminal usage event by @jarvis9443 in #1275
  • fix(access-log): name a gateway refusal on the access log wherever the usage event does by @jarvis9443 in #1279
  • fix: handle Gemini's omitted index on embeddings and streamed parallel tool calls by @jarvis9443 in #1276
  • fix(vertex): function calling on the native Gemini path; portable tool call ids by @jarvis9443 in #1277
  • fix(guardrails): never scan tool-call ids by @jarvis9443 in #1278
  • fix(anthropic): stream Claude tool calls to OpenAI-protocol clients by @jarvis9443 in #1280
  • fix(bridge): word caller-shape refusals as request errors, not upstream misconfiguration by @jarvis9443 in #1284

Full Changelog: v1.5.0...v1.6.0

Contributors

moonming and jarvis9443
Assets 2
Loading