Skip to content

v1.7.0 — Native Claude on Bedrock and Vertex AI

Choose a tag to compare

View all tags
@github-actions github-actions released this 22 Oct 03:49
· 17 commits to main since this release
v1.7.0
25f032f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🚀 Get started

New to AISIX? Get the gateway running and route your first LLM request in minutes:

📦 Download

Pull the container image from the GitHub Container Registry:

docker pull ghcr.io/api7/aisix:1.7.0

The rolling latest tag and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.

AISIX 1.7.0 serves Claude on Amazon Bedrock and Vertex AI natively, stops dropping media on native-protocol upstreams, and brings exported telemetry onto the released OpenTelemetry GenAI conventions. /v1/messages and /v1/messages/count_tokens go to each platform's own Anthropic Messages route, and prompt caching, including auto_prompt_caching, now works there. Images, audio and files in chat requests reach Vertex Gemini, Anthropic and Bedrock instead of being dropped. Bedrock gains API-key, default-credential-chain and assumed-role authentication. API keys, teams and members can carry cost-attribution tags that flow to every usage event and can drive rate limits. Two decision endpoints, /v1/systemone and /v1/decisions, are new.

  • Claude on Bedrock and Vertex AI, natively (#1318, #1316, #1323, #1325): signed thinking, native stop reasons and cache counters on /v1/messages, a working count_tokens, and prompt caching on every entry point.
  • Media reaches native-protocol upstreams (#1314, #1327): image, audio and file parts are translated for Vertex Gemini, Anthropic, Claude on Vertex AI and Bedrock, and URLs the upstream cannot read are downloaded by the gateway behind one SSRF setting.
  • Bedrock, broadened (#1319, #1322, #1328): API-key, default-chain and AssumeRole authentication, GPT 5.6+ and Grok on Bedrock's Chat Completions endpoint, Converse requests that follow what the caller sent, and the proxy environment honoured.
  • Telemetry on the GenAI semantic conventions (#1288, #1299, #1300): OTLP spans follow v1.41.1, and the OTLP, Datadog and SLS exporters carry structured messages with reasoning and tool calls. Each exporter chooses its own headers to record.
  • Cost-attribution tags (#1301) on API keys, teams and members, stamped on every usage event and usable as tag:<key> rate-limit dimensions.
  • Decision endpoints (#1293, #1294): /v1/systemone and /v1/decisions, with conversion between the two formats.

⚠️ Breaking changes

  • OTLP spans follow the OpenTelemetry GenAI semantic conventions v1.41.1 (#1288, #1299). Saved queries and dashboards keyed on the old names must be updated:
    • gen_ai.system is removed; use gen_ai.provider.name (a hosting platform wins, so Claude on Bedrock is aws.bedrock).
    • gen_ai.request.model is now the model name sent upstream. The model or Model Group name the caller sent moves to aisix.requested_model.
    • gen_ai.usage.input_tokens now includes cached tokens for Anthropic-shape usage (Anthropic, Bedrock Converse, and passthrough routes reporting cache_*_input_tokens). OpenAI-shape usage is unchanged. The Datadog exporter's gen_ai.usage.input_tokens changes the same way; its aisix.* counters do not.
    • Span names change: the operation span is {operation} {upstream model} (for example chat gpt-4o) instead of chat.completions, the SERVER span is POST /v1/chat/completions and the like, and MCP spans are tools/call {tool}.
    • Successful spans leave status unset instead of OK. A 4xx no longer marks the SERVER span as an error, and a caller that hung up (499) is not an error on any span.
    • gen_ai.prompt / gen_ai.completion are removed. With content_mode: full, content is exported as gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.tool.definitions; content that is not a message list, or was cut by the capture cap, stays under aisix.content.prompt / aisix.content.response.
    • aisix.client_source_ip / aisix.client_user_agent become client.address / user_agent.original, and they and http.request.header.* move to the SERVER span.
    • Only the attempt span carries gen_ai.operation.name / gen_ai.request.model, so a backend that counts GenAI spans counts each attempt once instead of three times.
    • The resource service.name is observability.service_name (default aisix) instead of the fixed aisix-dp.
  • Datadog and Aliyun SLS exporters carry structured content under content_mode: full (#1300). On SLS, prompt is replaced by input_messages (plus system_instructions and tool_definitions when present) and response by output_messages, each a JSON string. On Datadog, gen_ai.prompt is replaced by gen_ai.input.messages (plus gen_ai.system_instructions and gen_ai.tool.definitions) and gen_ai.completion by gen_ai.output.messages. Output messages hold ordered text, reasoning and tool_call parts, and finish_reason uses the conventions' values (stop, length, tool_call, content_filter, error). A side that was truncated or is not a message list (embeddings, images, audio, jobs, A2A, MCP) stays in the old flat field with content_truncated, so one record can mix both. The object-store exporter is unchanged.
  • observability.usage_event.request_headers no longer does anything (#1299). Request-header capture moved to each observability exporter (see Observability). A gateway that still sets the key starts, logs one warning, and records none of those headers until its exporters list them.
  • Guardrail log lines name the guardrail as guardrail_name (#1313), which replaces row (and member on the mask-mismatch warning, name on the invalid-config warning) with no alias. On Amazon Bedrock guardrail lines the AWS guardrail identifier is logged as bedrock_guardrail_id instead of guardrail_id. Usage events, metrics and spans are unchanged.
  • Renamed failure tags and error classes.
    • A semantic guardrail whose embedding model answers with vectors that do not fit the request (wrong length, base64 instead of floats, fewer vectors than inputs) now reports semantic_embed_upstream instead of semantic_embed_unresolved in the fail-closed 422 message, guardrail hit records, guardrail_bypassed_reason and aisix_guardrail_bypasses_total{reason}; aisix.embed in a custom guardrail script throws Upstream instead of Unresolved (#1309).
    • The Amazon Bedrock guardrail tags a non-429 4xx from ApplyGuardrail (AccessDeniedException, ValidationException, ResourceNotFoundException, ...) and credentials that are missing or refused before the call as bedrock_config_error instead of bedrock_5xx; every 429 is bedrock_throttled. The tag appears in guardrail_bypassed_reason, aisix_guardrail_bypasses_total{reason}, the error_type label of aisix_guardrail_latency_seconds, and the fail-closed refusal message. 5xx and transport failures stay bedrock_5xx (#1328).
    • When the gateway cannot translate a request for the target provider, usage events, the access log and exporters (OTLP aisix.error_class, Datadog, SLS) record error_class / error_kind invalid_request_error instead of invalid_config. The error the caller receives is unchanged (#1328).

No resources.yaml field was removed, renamed or retyped in this release; a configuration file that loads on 1.6.0 loads unchanged.

🧭 Claude on Amazon Bedrock and Vertex AI

  • /v1/messages is served natively (#1318). /v1/messages addressed to a Claude model on Bedrock (an anthropic. model id, with or without a region or inference-profile prefix, or an ARN naming one) or Vertex AI (claude-*) is sent to the platform's own Anthropic Messages route (Bedrock InvokeModel / InvokeModelWithResponseStream, Vertex AI :rawPredict / :streamRawPredict) instead of being translated, and the caller receives the platform's answer: signed thinking and signature_delta, redacted_thinking, the upstream message id, native stop reasons, streamed message_start usage with cache counters, and on Bedrock amazon-bedrock-invocationMetrics on message_stop. metadata.user_id, context_management, images inside tool_result and supported anthropic-beta values now reach the model. An exception Bedrock raises mid-stream becomes an Anthropic error event with an Anthropic error type (throttlingException → rate_limit_error, and so on). Streamed requests to Bedrock now use InvokeModelWithResponseStream instead of ConverseStream; both need the bedrock:InvokeModelWithResponseStream permission. /v1/chat/completions and /v1/responses are unchanged, as is a Bedrock application inference profile ARN, and a structured-output request to a Claude family before 4.5 keeps the translated path.
  • /v1/messages/count_tokens works on both platforms (#1318). It used to answer 400. On Bedrock it uses CountTokens (permission bedrock:CountTokens), falling back to Bedrock Mantle for models CountTokens refuses, such as cross-region-only models; on Vertex AI it uses the platform's count-tokens route. The new startup key upstream.bedrock.mantle_endpoint (default: the public regional Mantle host) sets the Mantle URL for VPC-endpoint or GovCloud deployments.
  • Prompt caching works on every entry point (#1316). cache_control markers are kept through the chat translation for these targets and become cachePoint blocks on Converse (Claude in system, messages and tools; Amazon Nova in system and messages, without ttl). Toward Bedrock a marker keeps only type and ttl; toward Vertex AI scope is dropped. auto_prompt_caching can now be set on Claude models on Bedrock (an anthropic. model id, including region-prefixed ids and ARNs) and Vertex AI (claude-*) and applies there as on the Anthropic API; other models still refuse it with 400. Some Claude models on Bedrock accept only the 5-minute cache, and Bedrock refuses a request that asks for 1h.
  • anthropic-beta is filtered to what the platform runs (#1323, #1325). On Bedrock, betas are always sent in the request body's anthropic_beta, never as a header, including values from a provider key's default_headers or a header forward_client_headers admits; on Vertex AI they stay in the header. Values the platform does not run are dropped, and advanced-tool-use-2025-11-20 is renamed to tool-search-tool-2025-10-19. On /v1/chat/completions and /v1/responses to Claude on Bedrock, a body anthropic_beta value outside that set is now removed instead of forwarded. On Bedrock, interleaved-thinking-2025-05-14, claude-code-20250219, output-128k-2025-02-19, structured-outputs-2025-11-13, token-efficient-tools-2025-02-19, tool-examples-2025-10-29 and thinking-token-count-2026-05-13 are forwarded, so Claude 4.x on Bedrock now thinks between tool calls for Claude Code. On /v1/messages, a provider key's default_headers betas are merged with the caller's and with the betas the request body implies before filtering; on /v1/chat/completions and /v1/responses, default_headers wins over a forwarded value as for any other header.

🔌 Providers and endpoints

  • Images, audio and files reach native-protocol upstreams (#1314, #1327, #1333). image_url, input_audio, file and other media parts of /v1/chat/completions (and of /v1/messages / /v1/responses converted to chat) are now translated into the upstream's own media blocks for Vertex Gemini, Anthropic, Claude on Vertex AI and Bedrock (Claude InvokeModel and Converse), streamed or not; they used to be dropped, and the caller got 200 from a model that never saw them. Media an upstream cannot take (for example audio to Claude) is now a 400 naming messages[i].content[j]. Anthropic (https://) and Vertex Gemini (https:// / gs:// with a known media type) still receive the URL itself. Any other http(s) media URL is downloaded by the gateway and sent inline; another scheme, or a URL that cannot be downloaded, is a 400 naming the part, with the URL's query string and fragment removed from the message. Downloads:

    • refuse any host that resolves to a non-public address (loopback, private and CGNAT ranges, link-local including 169.254.169.254, IPv6 unique-local and similar), checked on every redirect hop, unless the host is listed in outbound.ssrf_protection.allowed_hosts or the check is turned off with outbound.ssrf_protection.enabled: false. With the check on, the host must resolve from the gateway even when a proxy carries the download;
    • go through HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY like every other upstream call, verify certificates under upstream.tls (verify, ca_file), and never forward the caller's headers;
    • are capped at upstream.media_fetch.max_bytes per URL (default 50 MiB), share the request's upstream timeout, and count together with the request body against proxy.request_body_limit_bytes (exceeding it is the same 413 request_too_large).

    Two new startup blocks govern this. upstream.media_fetch (enabled, default true; max_bytes) turns downloading on or off; with enabled: false, URL parts toward an upstream that cannot read the URL are a 400. outbound.ssrf_protection (enabled, default true; allowed_hosts, default empty, each host, host:port or [v6]:port; env AISIX_OUTBOUND__SSRF_PROTECTION__ENABLED / AISIX_OUTBOUND__SSRF_PROTECTION__ALLOWED_HOSTS) governs every URL a caller supplies that the gateway fetches, today only media downloads; URLs operators configure are unaffected. Both are declared in the 1.7.0 aisix Helm chart values. Providers on OpenAI-compatible endpoints forward media parts unchanged, as before.

  • Upstream error messages reach the caller (#1319). Errors from every Amazon Bedrock operation, Azure OpenAI and Vertex AI, and 5xx errors from every provider, now carry the upstream's own message instead of a fixed status phrase or upstream returned <status>. That text can include account ids, ARNs, deployment or project names. An error without a message is still named by its status.

  • Bedrock authentication modes (#1322). A Bedrock provider key's secret takes auth_type: access_key (default, as before, with optional session_token), api_key (a Bedrock API key, sent as a bearer token) or default_chain (the gateway's own AWS environment: environment variables, web identity / IRSA, the ECS task role or the instance profile). With access_key or default_chain, role_arn (optional external_id, role_session_name) assumes an IAM role. The Bedrock guardrail's aws_credentials gains kind: api_key and kind: default_chain, and static gains session_token and the role fields. Every Bedrock call uses the chosen mode. Missing or refused credentials are a non-retryable 401 authentication_error naming the mode; a failing STS or metadata endpoint is retryable.

  • GPT 5.6 and newer, and Grok, on Amazon Bedrock use Bedrock's Chat Completions endpoint (#1319). A openai.gpt-5.6-or-newer (gpt-oss excluded) or xai.grok-* model addressed through a cross-region inference profile (global., us., …) is sent to Bedrock's OpenAI-compatible /openai/v1/chat/completions instead of Converse, signed with the same provider key. Parameters AWS refuses there are dropped instead of failing (n, stop; penalties and Grok's reasoning_effort: "none" on Grok; temperature, top_p, penalties and logprobs on GPT while it reasons). A request needing a Converse-only feature (Bedrock guardrail, performance or service tier config, top_k, a response_format other than text or json_schema with a schema, function tools on a reasoning GPT model) stays on Converse. Remote image URLs are inlined and positional tool-call ids are replaced with unique call_… ids.

  • Bedrock Converse requests follow what the caller sent (#1319). On every Bedrock publisher: consecutive turns of one role are merged (tool results followed by a user message used to be refused by Bedrock); stop becomes stopSequences; max_completion_tokens wins over max_tokens; a response_format with a schema becomes a tool for every publisher, including Titan, AI21, gpt-oss and unclassified ones, which used to ignore it; reasoning_effort is mapped per publisher (gpt-oss and DeepSeek V3, GPT 5.x / 6, Nova 2 — where a value other than low, medium or high is a 400 — and dropped for others); top_k reaches Claude and Nova; parallel_tool_calls reaches Claude 4.5+; guardrailConfig, performanceConfig, serviceTier and requestMetadata reach the request, and an OpenAI service_tier maps to serviceTier. Top-level fields that are not Chat Completions parameters now reach non-Claude Converse models as additionalModelRequestFields instead of being dropped. Tool results can carry images, documents and video. With no tools to send, tool-call history is sent as text instead of being refused.

  • Bedrock errors are classified like other upstreams' (#1319). A Bedrock model id the gateway cannot recognise is a non-retryable configuration error, so a Model Group stops on it instead of failing over to its next target. ModelErrorException is a retryable upstream error (502 to the caller), ModelTimeoutException a timeout, and a context-window overflow carries code: context_length_exceeded.

  • Bedrock chat returns reasoning in every mode (#1285). Streamed Bedrock chat now returns reasoning (delta.reasoning_content, a reasoning item on /v1/responses, and an unsigned thinking block on /v1/messages to non-Claude publishers; Claude on /v1/messages returns Bedrock's own signed thinking, see above), and so does non-streamed chat to non-Anthropic Bedrock publishers.

  • System-only requests to Bedrock are refused up front (#1285, #1311). A request to any Bedrock model that holds only system or developer messages is now refused by the gateway with 400 invalid_request_error on /v1/chat/completions, /v1/messages and /v1/responses, streamed or not, and nothing is sent to Bedrock. It used to fail with 500 config_error, or, for a non-streamed request to Claude on Bedrock, reach Bedrock and come back as 400 with the message upstream returned 400 (upstream_error on chat and responses). The old 500 was retryable, so a routing group failed over to its next target; the new 400 ends the fallback chain, even when the group's fallback_on_statuses lists 400, and the request no longer counts as a health failure. A Vertex Gemini system-only request now succeeds.

  • A tool message without tool_call_id (#1285). On the Anthropic, Bedrock /invoke and Vertex Claude bridges the 400 message loses its invalid upstream configuration: prefix. Bedrock Converse forwards such a message as a tool result instead of silently dropping it.

  • Cohere embeddings on Amazon Bedrock (#1319) take the caller's input_type, embedding_types (or encoding_format) and, on Embed v4, dimensions, and report usage from Cohere's billed units.

  • Tool-calling turns that the upstream ends with stop report tool_calls (#1285). On every OpenAI-wire upstream (OpenAI, Azure OpenAI, OpenAI-compatible providers, the Vertex OpenAI endpoint), a turn that emitted at least one tool call and ended with stop is now reported as tool_calls on /v1/chat/completions and tool_use on /v1/messages, streamed or not. This includes OpenAI's own replies when tool_choice names a function. Other finish reasons pass through unchanged.

  • Decision endpoints /v1/systemone and /v1/decisions (#1293, #1294). /v1/systemone takes TypeSafe's System One format and /v1/decisions OpenAI's Decisions format. Either can be served by a typesafe, azure (Microsoft Foundry), openrouter or openai provider key, forwarding verbatim when formats match and converting questions and answers when they differ; a converted answer that lacks a required answer field fails with 502, and a routing group fails over to its next target. Direct and routing models and wildcard aliases are served, with failover, rate limits, model access control, usage events (operation systemone / decisions) and metrics. Guardrails do not run on these endpoints.

  • Responses conversations stay on the Model Group target that produced them (#1332). On /v1/responses through a routing model group, a follow-up turn that carries encrypted reasoning (encrypted_content) or a previous_response_id is sent first to the target that produced it, then to other targets on the same provider key, then to the rest. Before, it could fail with 400 invalid_encrypted_content or 404 when the group's strategy picked another deployment or key. On any other target, the reasoning item is sent without its encrypted content, and its readable summary is kept. Responses served through a routing model group carry a different response id from the upstream's, in non-streamed bodies and in streamed events; clients must echo values back exactly as received. Usage events, the access log and OTLP still record the upstream's own response id. Direct models are not affected. It is always on.

  • service_tier in responses (#1305). /v1/chat/completions served by an OpenAI-shape upstream now passes the upstream's service_tier through to the client (it used to be dropped). /v1/responses bridged onto a non-Responses upstream now reports the tier the upstream served on, or "default", instead of echoing the tier the caller requested.

  • Realtime sessions close cleanly (#1333, #1339, #1340). /v1/realtime now completes the WebSocket close handshake. A client that closes normally gets its own code and reason back instead of 1006, and the upstream receives the client's code and reason. When the upstream closes, the client gets the upstream's code and reason, with the reason relayed unredacted and cut to 123 bytes. A close code that cannot be sent in a Close frame becomes 1011, in either direction. A close frame of exactly 1002 "Protocol violation" also becomes 1011, because the gateway cannot tell it apart from an invalid code; 1002 with any other reason is relayed. Any end other than a normal close (1000) is preceded by a Realtime error event of type server_error that says what happened: the upstream's close code and reason, a close frame the gateway could not relay, a close without a status code, a failed connection and its cause, or a connection that ended without a close frame. Usage that arrives while the close completes is still counted. The access-log status is unchanged.

💬 Chat completions

  • Chat usage carries the upstream's token breakdown as reported (#1341). In /v1/chat/completions responses, streamed and not, usage.prompt_tokens_details and usage.completion_tokens_details now carry every counter the upstream reported, with its value. A reported 0 is included. These include cached_tokens, reasoning_tokens, audio_tokens, image_tokens, text_tokens, accepted_prediction_tokens and rejected_prediction_tokens. A counter the upstream did not report is omitted, and so is a details object with nothing reported. Before, a counter of 0 was dropped, an object whose counters were all 0 was absent, and text_tokens, image_tokens and the prediction counters were never relayed. A response served from the response cache still reports only non-zero cached_tokens and reasoning_tokens. Usage events and metrics are unchanged.

  • Streamed chat usage arrives on its own chunk (#1343). With stream_options.include_usage: true, a streamed /v1/chat/completions now ends with one usage-only chunk after the finish chunk. Its choices is [] and it is the only chunk that carries usage, as OpenAI sends it. Before:

    • from OpenAI-protocol upstreams and Bedrock Converse, that chunk carried [{"index":0,"delta":{}}];
    • from Anthropic, Claude on Vertex AI and Gemini on Vertex AI, there was no such chunk. usage rode on the finish chunk, and Gemini could also report running counts on earlier chunks.

    A client that reads choices[0] on every chunk, or that reads usage from the finish chunk, must change. Without include_usage, nothing changes. Usage events and rate-limit accounting are unchanged, including for streams that end early.

🏷️ Cost-attribution tags

  • Tags on API keys, teams and members (#1301). api_keys, teams and users entries accept tags, a map of tag key to value (keys match ^[a-z][a-z0-9_]{0,62}$, values 1 to 128 characters, at most 32 entries). The gateway resolves a request's tags per key — the API key's own value, else its team's, else its member's — and stamps them, with the key's team_id, on every usage event. Exporters ship them: OTLP aisix.tags.<key> and aisix.team_id, Datadog aisix.tags, SLS one tags.<key> field per tag, object store tags. Tags are never Prometheus labels and are never read from the request.
  • Tag rate limits (#1301). Rate-limit policies accept tag:<key> as a condition dimension (==, ~=, in, ~~, ~*) and in group_by, for example one counter per department. A request without the tag does not match the condition or its negation, and is not subject to a group_by on it.

🛡️ Guardrails

  • Guardrail log lines are easier to trace (#1313). Every gateway log line about one guardrail carries its configured name under guardrail_name (see Breaking changes). "guardrail blocked …" warnings on every endpoint now carry guardrail_name too, a semantic guardrail's embedding failure names the guardrail and the embedding model's display name instead of guardrail="semantic" and the model id, and a streamed output released unscanned under on_buffer_exceeded: fail_open is now logged.
  • Passthrough replies with two envelopes are refused (#1289). A buffered passthrough reply carrying both a Chat and an Anthropic envelope is now refused by a fail-closed output guardrail.

📊 Observability and usage events

  • Per-exporter header capture (#1299). Every observability exporter accepts request_headers and upstream_response_headers (lowercase header names; credential headers and set-cookie are refused). Each exporter records only its own lists: OTLP as http.request.header.<name> and, per attempt, http.response.header.<name>; Datadog, SLS and object store as request_headers / upstream_response_headers maps.
  • More request telemetry (#1299, #1303, #1305, #1321). Usage events and exporters gain requested_model_id (the id of the model entry the caller addressed, beside the serving model_id), api_key_name, gateway_overhead_ms (time spent in the gateway, excluding upstream waits), upstream_tpot_ms (time per output token on streams), upstream-reported audio and image token counts (gen_ai.usage.{audio,image}.{input,output}_tokens on OTLP), and service_tier, the tier the upstream actually served the request on (OTLP / Datadog aisix.service_tier, SLS service_tier): OpenAI and compatible service_tier, Anthropic usage.speed / service_tier, Bedrock serviceTier, Gemini usageMetadata.serviceTier. Prompt-cache writes are split by cache lifetime as cache_creation_5m_tokens and cache_creation_1h_tokens (OTLP aisix.cache_creation_5m_tokens / aisix.cache_creation_1h_tokens) when the upstream reports the split; cache_creation_tokens stays the total. gen_ai.tool.definitions now carries each tool's description and parameters.
  • A usage event's model_id names only a model that dispatches itself (#1304). For a Model Group (routing or semantic) or an ensemble that fails before a target is selected — guardrail block, entry-level rate limit, model forbidden, semantic resolve failure — and for a routing or semantic group's response-cache hit, model_id is now empty on /v1/chat/completions and /v1/messages instead of the group's id; requested_model_id still names the group. A direct model's pre-dispatch failure on /v1/responses, /v1/embeddings and the other single-shot endpoints now carries the model's own id instead of empty. Queries that select a group's events by model_id should use requested_model_id.
  • total_tokens is recorded on every endpoint family that reports it (#1335). Usage events for embeddings, rerank, audio transcription and translation, image generation and edits, Realtime, batch-completion attribution and passthrough routes metered as these families now carry the upstream's reported total_tokens. Before, it was null: gateways 1.5.0 through 1.6.x recorded it only on chat completions, completions, /v1/messages, /v1/responses and passthrough-route events, and gateways before 1.5.0 never recorded it. Realtime-session and batch_completion events record the sum of the reported totals. The field is null if any call that counted tokens reported no total. Client responses, rate limits and Prometheus counters are unchanged.
  • Passthrough routes meter usage by the protocol they carry (#1289). A passthrough exchange identified as chat, Anthropic Messages, Responses, Completions, embeddings, images or rerank is read with that endpoint's own usage parser, and is now counted once in the LLM request tier (aisix_llm_requests_total, aisix_llm_request_duration_seconds, the aisix_llm_* token families and TTFT / end-to-end latency) with endpoint="/passthrough_route" and model = the body's model (unregistered names fold to unresolved). Usage objects on an unidentified stream are no longer read. An agent backend's token_usage frame no longer counts its cache read twice. The passthrough access-log total_tokens now includes cache tokens, like the typed endpoints.
  • /v1/responses usage events carry a finish_reason (#1299) (stop, tool_calls, length, content_filter, or empty for failed / cancelled responses), derived from the result the caller received.
  • provider_request_id is recorded wherever the provider sent a request id (#1324). Failed upstream attempts, /v1/messages/count_tokens calls and successful calls whose response has no id of its own (Bedrock Converse and InvokeModel, embeddings, audio, images, rerank) now record the provider's request-id header (x-amzn-requestid, x-request-id, request-id, x-ms-request-id, apim-request-id, x-goog-request-id or cf-ray) in usage events and the access log; a response object's own id still wins. On a failed attempt it can therefore appear as OTLP / Datadog gen_ai.response.id. Each retried attempt carries its own id. A Bedrock Converse response's id, which used to be empty, is the request id. Access-log lines of failed requests name the provider the request was sent to instead of unknown (or no provider on /v1/chat/completions), and the multipart image-edit and audio routes name the requested model.
  • Ensemble usage is recorded whole (#1298, #1308). When a caller abandons an ensemble request, the panel members and judge that already returned are recorded as usage events ahead of the 499 event, and the tokens of those that answered are charged to the key's, team's and member's token limits. A panel member or judge call that fails now records a zero-token usage event (attempt_kind panel or judge, with its model, status, error_class and error_message); it used to leave none. Events are numbered by panel slot, so when a member fails, every event after it moves up one attempt_index per failed member.
  • Prompt estimates are realistic for images and opaque payloads (#1287). When an upstream reports no usage, images are estimated with OpenAI's vision tile formula, and encrypted reasoning, signatures, files and audio are no longer tokenized as text. Abandoned requests carrying screenshots or reasoning items no longer record hundreds of thousands of phantom prompt tokens.

⚙️ Deployment and configuration

  • Amazon Bedrock calls honour the proxy environment (#1328). Bedrock runtime calls, the Bedrock guardrail and STS calls now go through HTTP_PROXY / HTTPS_PROXY / ALL_PROXY unless NO_PROXY covers the host, the same way as every other upstream; they used to connect directly. Requests to the AWS credential endpoints (instance metadata, ECS and EKS Pod Identity, a loopback credential agent, a configured metadata endpoint) always stay direct.
  • Realtime sessions honour the proxy environment (#1330). /v1/realtime now reaches a wss:// provider through HTTPS_PROXY (a ws:// one through HTTP_PROXY), falling back to ALL_PROXY, unless NO_PROXY covers the host; it used to connect directly. A SOCKS proxy is not supported: the session fails with an error frame and close code 1011 (502 in the access log).
  • An open Realtime session holds a draining gateway (#1317). On shutdown the gateway now waits for open /v1/realtime sessions to close, as it does for streams, so their usage is recorded.
  • Requests arriving late in a drain are served once (#1317). A request that arrives after the drain ends, on a connection accepted before it, is now answered with Connection: close and sent upstream exactly once; it could be sent two or three times (thread-per-core mode) or closed without a response. A TLS connection that completed its handshake but has not sent a request now holds the drain until it sends one, downstream.idle_timeout_secs expires, or the client closes it.
  • Configuration errors name the key (#1333). A configuration value the gateway cannot read now stops startup with the full dotted key, for example upstream.media_fetch.max_bytes. If an AISIX_* environment variable set the key, the error names that variable too. Before, the error was the bare parser message.

🐛 Fixes

  • Chat requests to Claude accept stream_options, max_completion_tokens and stop (#1343, #1344, #1345). These used to fail with 400 ("… Extra inputs are not permitted") on Anthropic and Anthropic-compatible providers, Claude on Vertex AI, and non-streamed Claude on Amazon Bedrock. stream_options is no longer sent to the upstream. max_completion_tokens is the output cap and wins over max_tokens when both are sent, and a reasoning_effort thinking budget is sized against it. stop becomes stop_sequences, with a string sent as a one-element array. A stop_sequences the caller sends itself wins over stop on every Claude path, streamed Claude on Bedrock (Converse) included. Non-streamed Claude on Bedrock sends exactly what the Anthropic path sends, and invalid values reach the upstream, which rejects them.
  • Streamed chat completions from audio-output models (for example modalities: ["text","audio"]) now relay delta.audio, end with the usage chunk and [DONE], and report audio tokens (#1333, #1341). Before, the stream dropped the audio and ended in upstream_decode_error. This affects OpenAI and OpenAI-compatible upstreams, Azure OpenAI, and the OpenAI-compatible routes of Bedrock and Vertex AI. Chat output guardrails do not inspect the generated audio or its transcript.
  • When an upstream reports "cached_tokens": null or "reasoning_tokens": null, the request now succeeds and the counter is treated as unreported (#1341). Before, the chat completion failed with 502, and on a passthrough route no tokens were recorded.
  • On every Claude upstream (the Anthropic API, Claude on Vertex AI, Claude on Bedrock), a cache_control beside a chat message's content, on a role: "tool" message or on a tool_calls[] entry is now applied to that message's last block, the tool result or the tool call; it used to be dropped (#1316).
  • Claude on Bedrock no longer refuses thinking.display: "updates" (sent by Claude Code, whose first request in a session used to fail with 400); a display other than summarized or omitted is dropped. Requests carrying dangerous-tool-use-2026-09-03, thinking-binding-controls-2026-08-01 or thinking-display-updates-2026-08-18, which Bedrock refused with 400, now succeed without them (#1323).
  • Chat to Claude on Vertex AI with a provider key whose anthropic-beta header carries a value Vertex AI does not run no longer fails with 400; the value is dropped (#1325).
  • Structured-output requests with thinking on Claude models without native structured output (Anthropic, Bedrock /invoke, Vertex Claude) now stream the reasoning they return when not streamed (#1286).
  • A Responses stream that fails with a nested error event now records the upstream's message in the usage event and access log instead of an all-null envelope (#1297).
  • Every request yields exactly one usage event with attempt_index 0: a cancel during retry backoff, a /v1/videos failure after a winning attempt, and ensemble failure exits no longer write a second one, and the duplicate span that went with it is gone (#1296).

⬆️ Upgrade notes

  • Update trace queries and dashboards for the new OTLP span shape, and Datadog queries on gen_ai.usage.input_tokens for Anthropic-shape traffic (see Breaking changes).
  • Move SLS indexes and Datadog facets to the structured content fields. On SLS, create field indexes for input_messages, system_instructions, tool_definitions and output_messages and move saved queries off prompt / response; on Datadog, move facets and monitors from gen_ai.prompt / gen_ai.completion to gen_ai.*.messages.
  • Reconfigure request-header capture on your exporters. If your gateway sets observability.usage_event.request_headers, list the same headers in each observability exporter's request_headers after upgrading, and remove the key. The 1.7.0 aisix chart no longer carries the value.
  • Update log and metric queries on renamed values: row=, member= and name= on guardrail log lines become guardrail_name=, and a Bedrock guardrail line's guardrail_id= becomes bedrock_guardrail_id=; bedrock_5xx for Bedrock guardrail configuration errors becomes bedrock_config_error; invalid_config for request-shape refusals becomes invalid_request_error; semantic_embed_unresolved for unusable embedding responses becomes semantic_embed_upstream.
  • Error messages now carry the upstream's text. If your gateway serves callers who must not see provider account details, note that errors from Amazon Bedrock, Azure OpenAI and Vertex AI, and 5xx errors from every provider, now include the provider's own message, which can name account ids, ARNs, deployment or project names.
  • List internal media hosts in outbound.ssrf_protection.allowed_hosts, or set outbound.ssrf_protection.enabled: false. Chat media URLs toward Bedrock, Claude on Vertex AI, and http:// URLs toward Anthropic or Gemini, are now downloaded by the gateway, which refuses non-public addresses. If callers send media URLs served from internal hosts, list them (Helm: config.outbound.ssrf_protection.allowed_hosts). Public names that a fake-ip or transparent proxy resolves into 198.18.0.0/15 are refused too; list them or turn the check off. Inline media (base64, data: URLs) and URLs the upstream reads itself are unaffected. With proxy.request_body_limit_bytes set, downloaded media counts toward it, so a request that passed because its media was dropped can now get 413.
  • Bedrock and Realtime traffic now goes through the proxy environment. To keep it direct (for example a Bedrock VPC endpoint), add the host to NO_PROXY. Realtime cannot use a SOCKS proxy.
  • Size the termination grace period for the longest Realtime session, as well as the longest stream, since an open session now holds a draining gateway.
  • Responses conversations through a routing model group across gateway versions. During a rolling upgrade or a gateway rollback, a /v1/responses follow-up through a routing model group that continues a conversation started on a 1.7.0 gateway fails if it lands on an older gateway: OpenAI answers 400 ("string too long", because the response id exceeds its 64-character limit), and other upstreams may answer 404. Start a new conversation to recover. Direct models are not affected.

What's Changed

  • fix(bridge): Bedrock streamed reasoning, tool-call finish reason, request-shape errors by @jarvis9443 in #1285
  • fix(structured-output): carry reasoning through the tool-route simulated stream by @jarvis9443 in #1286
  • fix(usage): price images and skip opaque payloads in local prompt estimates by @jarvis9443 in #1287
  • feat(otlp): align exported spans with the released GenAI semantic conventions by @jarvis9443 in #1288
  • fix(passthrough): meter an identified protocol with the typed endpoint's parser by @jarvis9443 in #1289
  • build(deps): consolidate open dependabot bumps and security fixes by @jarvis9443 in #1291
  • docs(releasing): drop the downstream offline-package section by @jarvis9443 in #1292
  • docs: define AISIX OSS and shared gateway terminology by @kayx23 in #1290
  • feat(proxy): serve /v1/systemone and /v1/decisions with cross-format conversion by @jarvis9443 in #1293
  • fix(proxy): fail a converted decision answer that lacks a required field by @jarvis9443 in #1294
  • fix(telemetry): give every request exactly one attempt_index 0 usage event by @jarvis9443 in #1296
  • fix(proxy): read the nested error object of a Responses stream error event by @jarvis9443 in #1297
  • fix(telemetry): keep ensemble usage rows and charges whole when the caller leaves by @jarvis9443 in #1298
  • feat(telemetry): structured output content, per-exporter header capture, latency breakdown and modality tokens by @jarvis9443 in #1299
  • feat(telemetry)!: structured content on the Datadog and SLS exporters by @jarvis9443 in #1300
  • feat(attribution): cost-attribution tags on api keys, teams and members by @jarvis9443 in #1301
  • docs(admin-api): clarify the AISIX OSS scope by @kayx23 in #1302
  • feat(obs): export the addressed model entry's id as requested_model_id by @jarvis9443 in #1303
  • fix(obs): a usage event's model_id names only a dispatchable model by @jarvis9443 in #1304
  • feat(obs): record the service tier the upstream served on by @jarvis9443 in #1305
  • fix: error messages read correctly for vowel-initial kinds by @jarvis9443 in #1306
  • fix(guardrails): report an unusable embedding response as an upstream failure by @jarvis9443 in #1309
  • fix(telemetry): give every failed ensemble sub-call its own usage row by @jarvis9443 in #1308
  • fix(chat): translate image, audio and file parts on native bridges instead of dropping them by @jarvis9443 in #1307
  • Revert "fix(chat): translate image, audio and file parts on native bridges instead of dropping them (#1307)" by @jarvis9443 in #1310
  • fix(bedrock): refuse a system-only request on the Claude /invoke path too by @jarvis9443 in #1311
  • test(e2e): keep every rate-limit burst inside one fixed window by @jarvis9443 in #1312
  • docs(agents.md): define AISIX as the product suite by @kayx23 in #1295
  • fix(guardrails): name the guardrail on every guardrail log line by @jarvis9443 in #1313
  • fix(chat): translate media parts on native bridges and download URLs the upstream cannot read by @jarvis9443 in #1314
  • test: fix four flaky tests at their cause (scrape handover, sentinel TILT, etcd slow-watcher gauge, shared wiremock pool) by @jarvis9443 in #1315
  • fix(bedrock,vertex): make Claude prompt caching work on Bedrock and Vertex AI by @jarvis9443 in #1316
  • fix(bedrock): serve GPT 5.6+/Grok on Chat Completions, fix Converse requests, pass upstream error text by @jarvis9443 in #1319
  • feat(bedrock,vertex): serve /v1/messages to Claude natively on Bedrock and Vertex AI by @jarvis9443 in #1318
  • feat(telemetry): split prompt-cache writes by cache lifetime on the UsageEvent by @jarvis9443 in #1321
  • fix(bedrock): send Claude only the betas Bedrock runs, drop a refused thinking display, count thinking on string bodies by @jarvis9443 in #1323
  • feat(bedrock): Bedrock API key, default credential chain and AssumeRole for provider keys and guardrails by @jarvis9443 in #1322
  • fix(telemetry): record the provider's request id wherever the response has no id of its own; name the provider on failed requests' access-log lines by @jarvis9443 in #1324
  • fix(bedrock,vertex): filter a provider key's anthropic-beta for Claude; on Bedrock send it in the body, never as a header by @jarvis9443 in #1325
  • fix(server): no duplicate upstream calls or dropped late requests at the end of a drain by @jarvis9443 in #1317
  • ci: shard e2e across jobs and cancel superseded PR runs by @jarvis9443 in #1326
  • feat(outbound): one product-wide SSRF setting for caller-supplied URLs by @jarvis9443 in #1327
  • fix: Bedrock proxy env, Bedrock guardrail 4xx tag, caller-refusal error class, chat audio tokens by @jarvis9443 in #1328
  • chore(deps): bump hono to 4.13.13 in tools/mcp-conformance by @jarvis9443 in #1329
  • fix(realtime): dial the upstream WebSocket through the proxy environment by @jarvis9443 in #1330
  • ci: stop running docker-image on PRs, smoke every pushed image, clippy --locked by @jarvis9443 in #1331
  • feat(responses): keep Responses context on the routing target that produced it by @jarvis9443 in #1332
  • ci: pull etcd from the ghcr.io/api7/etcd mirror instead of quay.io by @jarvis9443 in #1334
  • fix: rc.6 QA findings — media-fetch diagnostics, config key paths, streamed audio, realtime close by @jarvis9443 in #1333
  • fix(usage): record the upstream's reported total_tokens on every endpoint family by @jarvis9443 in #1335
  • test(admin): drop the dead testcontainers etcd test and fold its checks into CI by @jarvis9443 in #1337
  • fix(media-fetch): log redirected_to the way url is logged by @jarvis9443 in #1338
  • fix(realtime): stop reporting the gateway's own 1002 as the upstream's close code by @jarvis9443 in #1339
  • fix(realtime): stop passing the library's 1002 upstream as the client's close code by @jarvis9443 in #1340
  • fix(chat): relay the upstream's usage details as reported, empty choices on the usage frame by @jarvis9443 in #1341
  • fix(chat): Claude bridge drops stream_options, maps max_completion_tokens and stop; include_usage usage on its own chunk by @jarvis9443 in #1343
  • fix(bedrock): forward stop and max_completion_tokens as sent on Claude InvokeModel by @jarvis9443 in #1344
  • fix(bedrock): let a Claude caller's native stop_sequences win over stop on Converse by @jarvis9443 in #1345

Full Changelog: v1.6.0...v1.7.0

Contributors

kayx23 and jarvis9443
Assets 2
Loading