v1.5.0 — Model Groups on every endpoint

🚀 Get started

New to AISIX? Get the gateway running and route your first LLM request in minutes:

• 📖 Documentation — https://docs.api7.ai/ai-gateway/
• ⚡ Gateway quickstart (one container, declarative resources.yaml) — https://docs.api7.ai/ai-gateway/getting-started/gateway-quickstart

📦 Download

Pull the container image from the GitHub Container Registry:

docker pull ghcr.io/api7/aisix:1.5.0

The rolling latest tag and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.

---

AISIX 1.5.0 makes a Model Group work on every single-request endpoint, and makes what the gateway records match what actually happened. Completions, embeddings, rerank, images, audio and video submission now walk a group's targets with the same failover, retries and balancing chat already had. A stream that fails after its headers is recorded with the failure's status instead of 200, every model-serving request leaves a usage row, and token usage is forwarded exactly as the upstream reported it. Streamed output guardrails measure their buffer by generated content and honor each row's own buffer settings, and the gateway can now explain its own memory.

• Model Groups on every single-request endpoint (#1236) — /v1/completions, /v1/embeddings, /v1/rerank, image generation and edits, the three audio endpoints and video submission used to answer 400 for a Model Group; they now fail over, retry, write one usage row per attempt and balance least_latency / least_busy groups.
• Shared-secret (HMAC) JWT verification (#1223, #1225) — an OIDC provider can verify HS256 / HS384 / HS512 tokens against hmac_secret instead of a JWKS.
• The Responses-to-chat bridge behaves the way agent clients expect (#1226, #1227) — full Response objects, a response.failed event, replayed reasoning; and /v1/chat/completions accepts developer messages (#1047).
• Usage rows tell the truth (#1227, #1234, #1236, #1238, #1247) — post-header stream failures carry their real status, every request leaves a row, failed attempts get their own rows, and upstream token counts are forwarded as reported, including the upstream's own total.
• Memory diagnostics (#1244) — allocator, cgroup-limit, process, runtime and per-store gauges, a loopback heap-profile endpoint, and automatic heap dumps as memory nears its limit; plus a request body now held in about a third less memory while its upstream works (#1246).

⚠️ Breaking changes

• A 1.5.0 gateway will not start from a resources.yaml that contains a type: openapi MCP server whose document yields no tool (#1236). Such a server — an OpenAPI document with no paths, or only operations the gateway cannot express, such as a multipart-only upload — used to load and serve nothing. It is now a load error for the whole file: the gateway refuses to start, and a SIGHUP reload is refused, keeping the previous snapshot. Remove that server, or give it a document with at least one usable operation; aisix validate --resources <file> finds affected files before you upgrade. Control-plane deployments are unaffected, since the control plane has refused such specs at save time since 1.4.0. Operation ids that collide after name sanitization still load (served as name, name_2, …), and aisix validate now warns about them.

🧭 Routing and endpoints

• Model Groups on the single-request endpoints (#1236). They walk the group's targets with the same strategy order, health filter, retry budget and failover as chat. The usage row is attributed to the target that served the request, the deployment and fallback metrics and the access-log routing summary are filled in, and every target must support the endpoint it is addressed on. /v1/realtime and the jobs surface still refuse Model Groups by design.
• developer messages on /v1/chat/completions (#1047, #1230). Kept for OpenAI, sent as system to OpenAI-compatible providers that do not accept the role, and mapped to the native instruction slot on Anthropic, Gemini and Bedrock. Guardrails treat them like system messages: input_messages: latest_turn excludes them and all includes them.
• The Responses-to-chat bridge (#1226, #1227). Every Response it emits carries the full field set the Responses API defines. A stream that fails part-way ends with response.failed after the existing error event, with invalid_prompt for a guardrail stop and the upstream's own code for context length, quota and rate limits, so Codex CLI no longer retries such a turn as a dropped connection. A stream whose first event is a failure opens with response.created and response.in_progress. An upstream stream that carries nothing fails as a retryable upstream_error instead of completing empty, and a connection that drops after the finish completes normally. Replayed reasoning items reach the upstream as reasoning_content, an assistant message and the tool calls after it go upstream as one message, and namespace tools are offered to the model as <namespace>__<tool>.
• max_tokens to OpenAI and Azure OpenAI reasoning models is sent as max_completion_tokens (#1236) on chat, the /v1/messages bridge and the Responses bridge. A provider key's request.param_renames still has the last word.
• Every upstream event stream is read by one spec-compliant parser (#1155, #1233). Streams framed with bare carriage returns are handled on every route, a stream that opens with a byte-order mark keeps its first event, and A2A agents that split one JSON-RPC event across several data: lines are relayed instead of ending the stream. A single frame that grows past 16 MiB unterminated now fails instead of growing without bound.
• Userinfo in a configured URL (user:pass@host) is accepted and used as written (#1248). It is neither validated nor masked. The gateway now accepts it on Azure OpenAI and Vertex AI api_base, the Azure AD authority_host, OIDC issuer and jwks_uri, the MCP OAuth resource_url, managed.cp_base_url / managed.cp_etcd_endpoint, and loopback http:// OTLP and object-storage exporter endpoints (OTLP sends it as Basic authentication; object-storage providers ignore it and sign requests themselves). Logs, errors and debug output show configured URLs as written, userinfo included; credential query parameters are still redacted.

🔐 Authentication

• Shared-secret (HMAC) JWT verification (#1223, #1225). An OIDC provider can carry hmac_secret instead of a JWKS. Its presence selects HMAC verification: HS256, HS384 and HS512 tokens are verified against the secret's UTF-8 bytes as-is, nothing is fetched, and issuer and audiences become optional. A token whose iss matches a provider's issuer is verified by that provider alone; otherwise it is tried against every enabled issuer-less shared-secret provider in name order. JWKS providers are unchanged, and a kid-less token is now tried against every key in the JWKS.

🛡️ Guardrails

• Streamed output guardrails measure max_buffer_bytes by generated content (#1233, #1238). The cap counts assistant text, reasoning and tool-call arguments on every route, never SSE or JSON framing. A text-only chat stream trips exactly where it did; a chat stream heavy in reasoning or tool-call arguments now trips earlier, and on native /v1/messages and /v1/responses, the Responses bridge and passthrough routes, which used to count wire bytes, it trips later. Each held stream is also bounded at 128 × max_buffer_bytes in raw bytes; crossing it follows on_buffer_exceeded.
• A streamed /v1/messages response under a block-capable output guardrail is delivered after the scan (#1236). window-mode rows — Azure and Aliyun text moderation, custom, aliyun_ai_guardrail — used to let the stream through token by token, so a block only appended an error after the content had gone out. The response is now held and released whole once it scans clean. Monitor-only chains still stream live.
• window-mode rows honor their own buffer settings where the stream is held whole (#1236). On /v1/messages and /v1/responses, native and bridged, a window chain now uses the row's max_buffer_bytes and on_buffer_exceeded instead of a fixed 256 KiB fail-closed cap; with several window rows the smallest cap wins and the stream fails closed unless every row fails open. The same folded cap bounds streamed tool-call arguments on /v1/chat/completions under an all-window chain. on_buffer_exceeded: fail_open is now honored on both endpoints (#1233).
• keyword and pii judge each value on its own (#1233). Each message, content block, tool-call argument, tool result the caller sends, and MCP tools/call result and structuredContent value is judged separately, and object keys and JSON numbers in tool payloads are values of their own. A rule anchored to a whole value (^…$) now blocks; a rule that only matched across two values no longer matches. On /v1/responses, replayed mcp_call arguments and output, mcp_approval_response.reason and array-shaped function_call_output.output are now masked as well as scanned.
• Guardrails on passthrough routes read what the typed endpoints read (#1233): Anthropic Messages streams, tool-call arguments in both directions, system prompts and the tool results the caller sends.
• What a guardrail did is recorded precisely. A pii mask on a route that cannot rewrite content (passthrough, /a2a, jobs, /v1/realtime, /v1/videos) records mask_unsupported / would_mask_unsupported (#1233). A stream refused by the hold-back cap records blocked_buffer_exceeded, and a fail-open release records guardrail_bypassed_reason: output_buffer_exceeded (#1233, #1236). Per-guardrail phase series count only phases where the guardrail ran (#1237). Aliyun Code 588 / 581 are classified as aliyun_throttled / aliyun_timeout instead of aliyun_5xx (#1235).
• Streamed guardrail refusals carry error.code (#1236). On /v1/messages and count_tokens a refusal keeps error.type: invalid_request_error and gains error.code (content_filter or guardrail_unavailable); the streamed refusal frame on /v1/chat/completions carries the same code as the buffered 422. On a passthrough route relaying an Anthropic stream the refusal frame now has the Anthropic shape, so its error.type changes from content_filter to invalid_request_error.
• Custom guardrail scripts see ctx.model on chat, Responses and Messages (#1231), where it was always empty.

📊 Usage and telemetry

• A stream that fails after its 200 headers is recorded with the failure's status (#1227, #1236). Mid-stream upstream, transport, decode and timeout errors, in-band upstream error events and the Responses bridge's empty-stream failure are recorded as 502, 504 or the upstream's own 4xx, with error_class and error_message filled in; a caller that leaves mid-stream is 499 on every streaming endpoint. The access-log line and the status label on aisix_usage_events_emitted_total move accordingly. /v1/audio/speech writes its usage event when the audio ends.
• Every model-serving request leaves a usage row (#1238). Cohere rerank, the gateway's own 501 for a missing provider capability, and an MCP tool call whose result could not be read back now write zero-token rows. Polling a video job and downloading its content remain the only calls that write none.
• Failed attempts get their own rows (#1236). On completions, embeddings, rerank, image generation and edits, audio, video submission and /v1/messages/count_tokens, every failed attempt — failover or same-target retry — writes a zero-token row with its attempt_kind and error, as chat already did.
• Upstream usage is forwarded as reported (#1234, #1247). Each usage event carries the upstream's own total_tokens. Gemini thinking calls, streamed or not, record completion_tokens without the thoughts and the thoughts as reasoning_tokens, and SLS and object-storage exports carry the same raw values; a streamed Vertex AI Gemini chat no longer sends an all-zero usage on its first chunk. /v1/messages and /v1/completions record reasoning tokens as /v1/chat/completions does.
• occurred_at has millisecond precision (2026-09-24T12:42:25.123Z) in every sink (#1236).
• Rejected usage events are counted (#1234). When the control plane accepts a batch but rejects some of its events, the gateway logs one warning and counts them in aisix_usage_events_rejected_total.
• Passthrough routes meter rerank and DashScope native calls (#1232), recording the upstream's token counts and the caller's model.
• Exporters. An object-storage exporter drops a batch the store rejects with a non-retryable 4xx at once, counted as aisix_otlp_fanout_drops_total{reason="permanent_error"}, instead of retrying it for five minutes while later records were dropped as queue_full; a Datadog exporter now retries a 408 (#1239). An exporter retrying a failing receiver reports its latest error from the first failed attempt (#1242). Delivery errors no longer expose secrets configured on the exporter (#1243). Deleting or disabling an exporter stops its pipeline, and a Google Cloud Storage exporter honors endpoint, which must serve the Cloud Storage XML API (#1233, #1239).

🔭 Observability

• Memory diagnostics (#1244). The Prometheus listener publishes aisix_allocator_bytes, aisix_memory_limit_bytes, the standard process_* metrics, per-runtime task gauges, and aisix_component_entries / aisix_component_bytes for in-process stores. A loopback-only listener, observability.debug (default 127.0.0.1:9091, on), serves GET /debug/pprof/heap, and observability.heap_profiling.auto_dump (on by default; 80% and 90% of the memory limit; /var/lib/aisix/heap; 5 files kept) writes a profile as memory crosses each threshold. Process metrics and heap profiles are Linux-only.
• The time-to-first-token metrics gain a side label (#1235). aisix_request_ttft_seconds and aisix_llm_time_to_first_token_seconds keep their existing observation as side="upstream" and add side="downstream", measured from the gateway receiving the request to the first frame the client gets.
• aisix_request_e2e_latency_seconds gains a side label with the opposite default (#1245, #1250). Its existing observation becomes side="downstream"; the new side="upstream" is how long the upstream took on the attempt that produced the response. side is always emitted on all three metrics and is not removed by observability.metrics.labels.
• Body sizes on the access log (#1245). The proxy request completed line gains request_body_bytes and response_body_bytes, each omitted when it cannot be final. The line is now written when the response body is finished, after the request's provider call completed lines.
• The routing target attempt failed warning carries request_id at every log level (#1236).
• Non-release builds report dev+sha-<sha> (#1229) in --version, the Server header and the heartbeat, instead of an empty version.

🐛 Fixes

• A gateway held about three copies of a request body, and more on some routes, while its upstream worked, which could run it out of memory under concurrent large base64-image requests; it now holds at most two, plus one per additional ensemble member in flight (#1246).
• A gateway whose configuration watch was cancelled for a reason other than compaction silently stopped applying changes while /status/config still reported it connected; it now re-reads its configuration and watches again, and a reconnect after a healthy period waits 1 second instead of an inherited backoff of up to 60 seconds (#1249).
• A resource deleted just before a restart came back when the on-disk snapshot cache was enabled (#1236).
• Cohere provider keys reached the wrong URL for chat and embeddings with the bare host as api_base, and for rerank with the /compatibility/v1 base (#1236).
• An upstream error with an empty body or blank message reached the caller as an empty error.message; it now reads, for example, upstream returned 404 Not Found (#1233).
• OpenAI-compatible upstreams that omit model from a response or stream chunk failed with upstream_decode_error (#1233).
• Provider keys with no api_base on the openai or anthropic adapter answered 400 from a resources file on routes that build their own upstream URL (#1233).
• Usage rows on many non-chat routes were missing applied_guardrails, including on a guardrail refusal's row (#1233).
• Monitor-mode output guardrails missed a match that arrived after the first 256 KiB of a streamed response (#1238).
• /v1/messages/count_tokens recorded only the successful attempt of a request that retried or failed over (#1236).
• Output keyword and pii guardrails masked or blocked on the raw reasoning of a streamed native /v1/responses reply from upstreams that close reasoning with response.content_part.done (#1241).
• redacted_entity_counts counted every entity twice in a chat message whose content is an array of text blocks (#1233).
• A streaming ensemble repeated the panel's usage on every judge chunk that carried usage; it now reports it once (#1233).

⬆️ Upgrade notes

• Add side!="downstream" to time-to-first-token queries and alerts, and side!="upstream" to end-to-end latency queries and alerts. Without them, a query blends the new observation into the existing one (and a TTFT _count doubles). Both filters also match 1.4.0 gateways during a rolling upgrade.
• Success rates, latency percentiles and status breakdowns shift for streaming traffic that failed after its headers, now recorded as 502, 504, the upstream's 4xx or 499 instead of 200.
• Parsers of occurred_at in SLS, Datadog or object-storage exports must accept a fractional second.
• Re-check keyword and pii rules written to match across two values — two messages, two text blocks of one chat message, a key and its value, a tool name and its arguments. They no longer match, while rules anchored to a whole value, and rules matching object keys or numbers in tool payloads, now block.
• Guardrails on passthrough routes may block traffic that used to pass unscanned.
• Streams heavy in reasoning or tool-call arguments trip max_buffer_bytes earlier on /v1/chat/completions. Raise the cap on a guardrail that holds such streams if they now hit on_buffer_exceeded.
• Consumers that switch on guardrail hit actions see new values: blocked_buffer_exceeded, mask_unsupported and would_mask_unsupported.
• Clients reading a streamed guardrail refusal relayed through a passthrough route from an Anthropic upstream should branch on error.code, not error.type.
• Reported token counts and cost can change. Gemini thinking rows record completion_tokens without the thoughts, which are in reasoning_tokens, including in SLS and object-storage exports. Reasoning on /v1/messages and /v1/completions is now recorded, so where a model has a separate reasoning price it bills at it.
• While 1.4.0 and 1.5.0 gateways share a Redis response cache, the 1.4.0 gateways treat entries written by 1.5.0 as misses until those entries expire. Responses and billing are unaffected.
• A password written in a Redis URL (redis://user:pass@host) now appears in the gateway's Redis connection and degradation warnings, as does userinfo in any configured URL. For single-node and cluster Redis, move the credential into the separate username and password fields; Sentinel authentication still lives in the sentinel URLs.
• Each gateway now listens on 127.0.0.1:9091, samples heap allocations, and writes heap profiles to /var/lib/aisix/heap as its memory nears its limit. Nothing has to be edited. To opt out, set observability.debug.enabled: false, observability.heap_profiling.auto_dump.enabled: false, or _RJEM_MALLOC_CONF=prof_active:false in the gateway's environment. A port already in use is logged and does not stop the gateway.
• If the shutdown grace period runs out while a slow client is still reading a non-streamed response, that request's access-log line is lost, as a streamed request's already was.

Known issues

• Thought tokens of thinking Gemini models are under-counted on google (AI Studio) provider keys. The gateway calls Gemini's OpenAI-compatible endpoint for these keys, which reports thought tokens only inside total_tokens, so reported cost, token-based rate limits and budgets under-count them.

What's Changed

• test(telemetry): deliver an attempt before letting the paused clock kill it by @jarvis9443 in #1219
• fix(redis): a refused credential is told apart from an outage, and password reaches the handshake by @jarvis9443 in #1220
• fix(obs): keep the log-drop summary alive across shutdown by @jarvis9443 in #1221
• test(e2e): wait for what the spec asserts, not for something written before it by @jarvis9443 in #1222
• feat(auth): verify inbound JWTs against a shared HMAC secret by @jarvis9443 in #1223
• docs(releasing): the offline package is dispatched, not triggered by a tag by @jarvis9443 in #1224
• fix(jwt): try every issuer-less HMAC provider and every kid-less JWKS key by @jarvis9443 in #1225
• fix(responses): align the chat bridge with what agent clients expect by @jarvis9443 in #1226
• fix(streaming): open failed Responses streams properly and record post-header stream failures in usage by @jarvis9443 in #1227
• docs(agents): re-baseline the agent instruction files by @jarvis9443 in #1228
• fix(version): report dev+sha- for non-release builds by @jarvis9443 in #1229
• fix(chat): accept developer messages in /v1/chat/completions by @ormandj in #1047
• docs(schema): name developer messages in the guardrail input_messages description by @jarvis9443 in #1230
• fix(guardrails): give custom scripts ctx.model on the segment pass by @jarvis9443 in #1231
• fix(passthrough): meter rerank and DashScope native bodies on passthrough routes by @jarvis9443 in #1232
• fix(a2a): parse multi-line SSE events by @mat973252 in #1155
• fix(guardrails): classify Aliyun Code 588/581; feat(obs): side label on TTFT metrics by @jarvis9443 in #1235
• fix(usage): record upstream usage as reported; count control-plane rejections by @jarvis9443 in #1234
• fix: batch of open data-plane bug fixes (guardrails telemetry and scanning, stream buffering, SSE framing, exporters, upstream errors) by @jarvis9443 in #1233
• fix(guardrails): record a guardrail only on the hooks it is attached to by @jarvis9443 in #1237
• fix: data-plane bugs filed in AISIX-Cloud (single-shot Model Groups, per-attempt telemetry, streamed guardrail holds, error codes) by @jarvis9443 in #1236
• fix: raw hold-cap 128x, full monitor-mode output scans, and a usage event for every request by @jarvis9443 in #1238
• fix(obs): drop object-store batches on non-retryable 4xx; state the GCS endpoint's XML API requirement by @jarvis9443 in #1239
• docs(redact): state what the response-side guardrail walk scans and masks by @jarvis9443 in #1240
• fix(guardrails): leave a streamed reasoning_text content part out of the Responses output scope by @jarvis9443 in #1241
• fix(obs): report an exporter stuck retrying as failing in exporter health by @jarvis9443 in #1242
• fix(obs): keep configured URL userinfo and secrets out of exporter delivery errors by @jarvis9443 in #1243
• feat(obs): body sizes on the access log, side label on e2e latency by @jarvis9443 in #1245
• fix(vertex): record a streamed Gemini call's raw usage row by @jarvis9443 in #1247
• fix: treat userinfo in configured URLs as part of an opaque URL by @jarvis9443 in #1248
• fix(obs): stop a finished stream's request and attempt clocks at one instant by @jarvis9443 in #1250
• fix(proxy): hold a request body at most twice while its upstream is pending by @jarvis9443 in #1246
• feat(obs): memory observability — allocator, process, runtime and component gauges, heap profiles on demand and near the limit by @jarvis9443 in #1244
• fix(etcd): recover from a server-cancelled watch, restart backoff after a healthy cycle; cover compaction resync end to end by @jarvis9443 in #1249

New Contributors

• @mat973252 made their first contribution in #1155

Full Changelog: v1.4.0...v1.5.0