Skip to content

v1.7.1

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Oct 11:38
· 6 commits to main since this release
v1.7.1
9263881
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🚀 Get started

New to AISIX? Get the gateway running and route your first LLM request in minutes:

📦 Download

Pull the container image from the GitHub Container Registry:

docker pull ghcr.io/api7/aisix:1.7.1

The rolling latest tag and the docker.io/api7/aisix mirror for private/offline deployments are listed on the package page.

AISIX 1.7.1 is a patch release. The gateway now handles every request field by what the target platform documents: a parameter the target can carry is sent as written and the upstream's own error comes back, a parameter it cannot carry is dropped, and a value that is illegal in the caller's own protocol is a 400 naming the field. Answers come back in full, including every choice, logprobs, refusal and annotations.

  • Request parameters follow the target platform's contract (#1352, #1353): Bedrock, Vertex Gemini and Claude receive the fields they document, in the slot they document, and the upstream's rejection reaches the caller.
  • Answers come back in full (#1353, #1354, #1356): every choice, logprobs, refusal, annotations and legacy function_call, and answers whose content is an array of parts work end to end.
  • Legacy function calling works on every provider (#1353), including Claude, Vertex Gemini and Amazon Bedrock.
  • Streamed chat usage follows stream_options.include_usage on every upstream (#1351).
  • Exporters accept plaintext http:// endpoints on any host (#1346).
  • Vertex AI global, us and eu locations reach Google's endpoints, and transcription usage and latency are recorded more completely (#1357, #1359, #1360).

⚠️ Behavior changes

See Upgrade notes for what to update.

  • Streamed /v1/chat/completions usage follows stream_options.include_usage on every upstream (#1351). With include_usage: true, usage is sent exactly once, on its own chunk (choices: []) after every other chunk. Without it, no chunk carries usage, as with OpenAI. Before, Claude (any platform), Vertex Gemini and OpenAI-compatible upstreams that report usage on the finish chunk showed it there even without include_usage. Usage events and TPM limits are unchanged. Clients that read usage from a stream must request include_usage.
  • Parameters the gateway used to drop now reach the upstream (#1352, #1353). Fields the target defines go out as the caller wrote them, and the upstream's rejection reaches the caller with its status and message; the gateway no longer drops, rewrites or pre-validates them. On Amazon Bedrock this means stop to Grok or GPT 5.6 and later, temperature / top_p / logprobs to a reasoning GPT, the penalties and n, and a tool's strict for a model that does not take it can now get AWS's 400; on Vertex Gemini 2.5, so can a /v1/messages thinking.budget_tokens above the model's ceiling. Details are under Providers and endpoints.
  • Values that are illegal in the caller's own protocol answer 400 naming the field (#1351, #1353). This covers a body that is valid JSON but does not fit the request (for example max_tokens: invalid value: integer -1, expected u32 or missing field messages, instead of invalid JSON request body), a message content that is a number, boolean or object, and, on the Claude, Converse and Vertex Gemini bridges, a wrong-typed stop, max_tokens, max_completion_tokens, temperature, top_p, tool_choice, reasoning_effort or service_tier, a malformed tool, or a replayed tool call without id or function.name. These used to be dropped silently, forwarded raw or read as empty. On Bedrock Converse, a stop that is not a non-empty string or string array, and a max_completion_tokens that is not a non-negative integer, are refused the same way. A lone empty-string stop is still skipped and null is still treated as absent.
  • anthropic-beta is forwarded to Anthropic by default (#1353). On /v1/messages and /v1/messages/count_tokens to Anthropic or an Anthropic-compatible endpoint, the caller's anthropic-beta header now reaches the upstream without being listed in the provider key's forward_client_headers. A value the provider key sets through default_headers or forward_client_headers still wins. anthropic-version stays pinned, and Claude on Bedrock and Vertex AI keeps its platform's beta filtering.
  • An answer the upstream already served is not run again (#1353). A 2xx the gateway cannot decode now returns 502 directly. It used to be retried, fail over to the next Model Group target and count toward cooldown, so the request was billed twice. This includes /v1/images/edits. A Decisions Model Group no longer fails over an incomplete converted answer either; the caller gets 502 (unexpected decision response) and the target is not cooled down. Credential token exchanges (Azure AAD, Vertex service account) keep their retry behavior.
  • Claude stop reasons map to OpenAI values on chat (#1353). refusal is now content_filter, model_context_window_exceeded is length and pause_turn is stop; they used to appear as raw finish reasons. On /v1/messages from a chat upstream, a content filter or an OpenAI refusal ends as refusal instead of stop_sequence, with the refusal text as a text block.
  • Anthropic server tools are not offered to chat upstreams (#1353). On /v1/messages converted to chat, web_search_*, web_fetch_*, code_execution_* and tool_search_tool_* are no longer sent as parameterless functions. Client tools (bash_*, text_editor_*, computer_*) are unchanged.
  • Error messages from Anthropic carry Anthropic's own text (#1353). On /v1/messages and count_tokens to Anthropic, an upstream error's error.message is Anthropic's message instead of the whole raw body as a string. An OpenAI-shape upstream error with a numeric code or param keeps its message as well.

🔌 Providers and endpoints

  • Amazon Bedrock Chat Completions route (Grok, GPT 5.6 and later) (#1352). n, stop, frequency_penalty, presence_penalty, Grok's reasoning_effort: "none", and GPT's temperature / top_p / logprobs / top_logprobs while it reasons are now sent as given. They used to be dropped. On Converse, stop is sent as stopSequences to every model, including Grok and GPT 5.6 and later. Where AWS refuses one, the caller now gets AWS's 400.
  • Amazon Bedrock Converse (#1353). OpenAI models (gpt-oss, GPT, Grok) receive seed, the penalties, user, safety_identifier, logit_bias, parallel_tool_calls, metadata and a json_object response_format through additionalModelRequestFields, as AWS documents. Cohere Command R receives seed and the penalties, Jamba the penalties, and Z.AI GLM and Kimi K3 reasoning_effort. Nova 2 maps reasoning_effort: "none" to disabled reasoning. Titan Multimodal embeddings map dimensions. A function's strict reaches Converse as toolSpec.strict; a model that does not take strict (for example Nova Lite) now answers with Bedrock's 400 where the request used to succeed without it. The output cap for GPT, Grok and Kimi K3 is no longer raised to 16. A request with no user or assistant turn goes to Bedrock and returns Bedrock's error instead of a gateway 400. Converse temperature / top_p keep their exact decimal value.
  • Vertex Gemini (#1353). generationConfig now carries max_completion_tokens, stop, seed, the penalties, n (as candidateCount), logprobs / top_logprobs and a /v1/messages top_k. reasoning_effort maps to thinkingLevel on Gemini 3 and later and to a thinking budget on Gemini 2.5. A /v1/messages thinking.budget_tokens goes to Gemini 2.5 as sent, so a budget above the model's ceiling now gets Vertex's 400 where the request used to succeed without thinking. service_tier flex / priority is sent as Vertex's Flex / Priority PayGo header. Function parameters go out unchanged as parametersJsonSchema on Gemini 2.5 and later, so additionalProperties is kept. Every candidate comes back as a choice with its logprobs, and a prompt Gemini blocks finishes as content_filter.
  • Claude on chat (Anthropic, Vertex AI, Bedrock InvokeModel) (#1353). OpenAI-only fields such as seed, n, logprobs, logit_bias, the penalties, store and prompt_cache_key are dropped instead of reaching Anthropic and causing a 400. user and safety_identifier become metadata.user_id, and a function tool's strict is copied to the Claude tool.
  • Fields reach the upstream as written (#1353). A caller's own strict json_schema goes out unchanged; the gateway closes a schema only when it set strict itself while translating /v1/messages output_format. An assistant content: null is sent as null, and temperature / top_p keep the caller's decimal (0.7, not 0.699999988). The max_tokens → max_completion_tokens rename for OpenAI reasoning models applies only on OpenAI's own API; an OpenRouter key serving openai/o3-mini, for example, now receives max_tokens, the only field OpenRouter documents.
  • Legacy function calling works on every provider (#1353). functions / function_call and role: "function" turns are accepted on every target, and null values are treated as absent. OpenAI-wire targets receive them as sent. Claude (Anthropic, Vertex AI), Vertex Gemini and Bedrock InvokeModel and Converse used to answer 400; they now receive them as tools, and the answer comes back as message.function_call / delta.function_call with finish_reason: "function_call". Only one call per turn fits the legacy shape.
  • Answers come back in full (#1353). Responses from OpenAI-compatible providers, Azure OpenAI, the Vertex AI OpenAI-compatible endpoint and Bedrock Chat Completions now keep every choice (streamed deltas keep their index), choice logprobs, refusal, annotations and legacy function_call. Only the first choice was kept before. /v1/messages and /v1/responses read the first choice. The response cache stores the whole response.
  • Answers whose content is an array of parts work end to end (#1353, #1354, #1356). OpenAI-compatible upstreams such as Mistral and Databricks that answer with content as an array of parts now work, streamed or not. Their text reaches /v1/messages, /v1/responses and content capture, and their reasoning parts are relayed as reasoning_content on chat, a thinking block on /v1/messages and a reasoning item on /v1/responses, counted in estimated completion tokens and captured for exporters. On a /v1/responses stream, reasoning that follows text now opens a new item in order instead of being appended to the earlier one.
  • /v1/embeddings forwards user and other fields to OpenAI-wire targets and accepts token-id input (#1353) ([1, 2], [[1, 2]]), which used to be a 400. Token ids carry no text, so input guardrails let such a request through. Bedrock and Vertex AI embeddings refuse token ids with a 400.
  • Claude below 4.5 on Vertex AI or Bedrock with structured output stays on the native Messages route (#1353), so Anthropic-only fields are no longer lost. Reasoning that streams while a tool call is open is now kept as its own thinking block. /v1/messages top_k and the exact thinking.budget_tokens reach targets that have a slot for them.
  • Vertex AI location global and the multi-region locations us / eu reach Google's endpoints (#1357). With no API base on the provider key, global now goes to https://aiplatform.googleapis.com and us / eu to https://aiplatform.{us,eu}.rep.googleapis.com, on every Vertex AI route (Gemini, embeddings, Claude, partner models, the OpenAI-compatible endpoint). They used to go to global-aiplatform.googleapis.com and us-aiplatform.googleapis.com, which do not exist. Regional locations such as us-central1 are unchanged.
  • Bedrock Converse request bodies are byte-identical for identical requests (#1348). JSON keys, including those inside tool schemas, are now sent in a stable order.
  • A Vertex AI provider key with a mistyped secret field gets an error naming the field (#1351), for example service_account_json must be a JSON object (the contents of the service-account key file), got a string, instead of "must be valid JSON".

🛡️ Guardrails

  • Input guardrails scan reasoning replayed as a content part (#1355). Mistral thinking, Databricks reasoning and Anthropic-shaped thinking parts in chat content are now judged like reasoning_content, so a block rule that matches them returns 422 where the request used to pass. Mask rules rewrite unsigned parts. A part with a non-empty signature is scan-only, like a signed thinking block on /v1/messages: block rules still fire, but a mask hit is forwarded unchanged. The gateway does not verify signatures, so use a block action where such text must not reach the upstream. On passthrough routes, a Mistral thinking part no longer makes the body unscannable.
  • Output guardrails scan every choice (#1353). Output guardrails and masking now cover every choice, the refusal text and legacy function_call arguments. Masking does not rewrite logprobs, so a caller that requests logprobs receives the upstream's token text unmasked.

📊 Observability and usage events

  • Exporters accept plaintext http:// endpoints on any host (#1346). The otlp_http and object_store exporter endpoint now accepts http:// with any host, for example an intranet collector at http://otel-collector-dev:4318/v1/traces. It used to be limited to a fixed set of test hosts. Aliyun SLS endpoint and Datadog site keep their restrictions.
  • Response-cache hits export the same output as the answer they replay (#1357). A cache hit's usage event now carries the stored response's finish_reason instead of an empty one, and its OTLP / Datadog span gains gen_ai.response.finish_reasons and the structured gen_ai.output.messages (reasoning included) that a fresh answer exports.
  • Latencies under 1 ms are reported as 1 ms (#1359). upstream_latency_ms, downstream_latency_ms, upstream_ttft_ms, per-attempt latency and embedding-call latency used to be reported as not measured when they were under a millisecond, as the caller latency of an exact-match cache hit usually is. A latency that was never measured is still 0 or absent.

🐛 Fixes

  • A streamed ensemble request with stream_options no longer fails with 502 (#1351). Panel members no longer receive stream_options; the caller still gets its usage chunk.
  • Claude chat streams that stop early are recorded with the real input and cache tokens (#1351). A stream to Anthropic or Claude on Vertex AI that ends before its final event (client disconnect or an in-stream upstream error) now records the input, cache-read and cache-write counts Claude reported, instead of an estimated prompt and no cache tokens. A stream that stops before its first content is still estimated.
  • Passthrough streams are metered from server-labelled usage frames (#1351) when the protocol parser read no usage, so a server that reports tokens only on event: token_usage frames is no longer recorded with zero tokens.
  • Transcription and translation usage reported as prompt_tokens / completion_tokens is recorded (#1357). An upstream that reports chat-style counts instead of input_tokens / output_tokens (Z.AI glm-asr-2512, for example) now gets token counts and token-priced cost on its usage events, for /v1/audio/transcriptions (streamed or not) and /v1/audio/translations. Upstreams that report the transcription names or a duration are unaffected.
  • Duration-billed transcriptions record the right length (#1359). A streamed /v1/audio/transcriptions now records the duration the upstream reports on its final event, and falls back to the uploaded file's length only when there is none, as non-streamed requests already did; it used to price on the file length alone. WAV and FLAC files written as a stream (as OpenAI's /v1/audio/speech and ffmpeg to a pipe produce them) are now measured instead of recorded with no duration. A duration too large to store is recorded as unmeasured with a warning in the gateway log. After the upgrade, these requests record a different duration and cost, and streamed WAV and FLAC uploads that used to be recorded with no duration start to carry a cost.
  • Requests that reached no upstream no longer report an upstream latency (#1360). On a response-cache hit, or a request refused before its first upstream attempt (an input guardrail block, an unknown model, a rate-limit or budget refusal, and similar), upstream_latency_ms and upstream_ttft_ms are now 0 (not measured) instead of the gateway's own handling time, on usage events, exporters and per-attempt rows, so statistics computed from upstream_latency_ms change for these rows. downstream_latency_ms still reports what the caller waited.

⬆️ Upgrade notes

These notes apply when you upgrade from 1.7.0 or earlier.

  • Request include_usage to read usage from a chat stream. Clients that read usage from the finish chunk of a /v1/chat/completions stream without stream_options.include_usage: true receive none after the upgrade.
  • Requests that relied on the gateway dropping parameters can now get the upstream's 400. On Bedrock, stop to Grok or GPT 5.6 and later, temperature / top_p / logprobs to a reasoning GPT, the penalties and n, and strict on a tool for a model that does not take it. On Vertex Gemini 2.5, a thinking.budget_tokens above the model's ceiling. Stop sending them for those models.
  • anthropic-beta now reaches Anthropic. Provider keys that did not list it in forward_client_headers start forwarding the caller's value. Set the header in default_headers if callers must not choose it.
  • Gateways of different versions sharing a response cache. During a rolling upgrade or a rollback, a response-cache entry with more than one choice or with logprobs, written by a 1.7.1 gateway, is a cache error for a 1.7.0 or earlier gateway sharing the cache. Other entries are unaffected.

Known issues

  • An ensemble whose judge is a reasoning GPT 5.6 or later on Amazon Bedrock gets a 400. The judge always sends temperature: 0.2, which AWS refuses while the model reasons. Use a different judge model.

What's Changed

  • feat(obs): accept plain http:// exporter endpoints on any host by @jarvis9443 in #1346
  • ci(image): build the gateway binary on the runner, package it with a thin Dockerfile by @jarvis9443 in #1347
  • ci: delete expired ci-test-* branch images from ghcr daily by @jarvis9443 in #1349
  • refactor(bedrock): send Converse request bodies with a stable key order by @tc3oliver in #1348
  • ci: ghcr retention for main builds, orphans and their cosign signatures by @jarvis9443 in #1350
  • fix: chat stream usage contract, Claude early-stop accounting, Bedrock Converse stop/max validation, and other 1.7.0 QA fixes by @jarvis9443 in #1351
  • fix(bedrock): forward parameters AWS refuses instead of silently dropping them by @jarvis9443 in #1352
  • fix: handle request parameters by the target platform's contract by @jarvis9443 in #1353
  • fix: render the text and reasoning of an array-content chat answer on /v1/messages, /v1/responses and content capture by @jarvis9443 in #1354
  • fix: treat replayed reasoning content parts like reasoning_content in input guardrails by @jarvis9443 in #1355
  • fix: relay the reasoning parts of a streamed array-content delta on every endpoint by @jarvis9443 in #1356
  • fix: 1.7.1 release-surfaced data-plane fixes by @jarvis9443 in #1357
  • fix: 1.7.1 release-surfaced data-plane fixes (QA round 2) by @jarvis9443 in #1359
  • fix(obs): report no upstream latency on rows that reached no upstream by @jarvis9443 in #1360

New Contributors

Full Changelog: v1.7.0...v1.7.1

Contributors

tc3oliver and jarvis9443
Assets 2
Loading