Repository navigation
Agent Router v1.3.0
Plain-markdown copy of the v1.3.0 release notes, suitable for pasting into the GitHub release body. The full rendered version lives at https://theagentrouter.ai/release-notes/v1.3.
Agent Router v1.3.0 makes QuotaPolicy a complete token-budget system. Per-model quotas are now charged accurately and applied without restarts. Limits can come from request metadata, windows can be weekly, monthly, or yearly, policies can target backends in other namespaces, and routes can fail over away from backends whose quota is exhausted. The release also adds the OpenAI Files API and the Decisions API, OpenAI workload identity federation for BackendSecurityPolicy, Claude Sonnet 5.5 between-tools thinking, and moves to Envoy Gateway v1.10.0 with Envoy Proxy v1.40.2. A few provider translations and the Go API changed, so read the breaking changes before you upgrade.
✨ New Features
QuotaPolicy, End to End
- Accurate token charging — A request used to be charged one extra unit when it was admitted, on top of its real token cost when the response finished, so a 28-token request deducted 29. The admission check now charges zero, and only the token cost reported at the end of the response or stream is deducted. Budgets now match the usage your providers bill for.
- Policy changes apply live — Creating, updating, or deleting a
QuotaPolicyused to take effect only after the controller and the Envoy pods restarted. The controller now stamps a hash of the applicable policies on the generated HTTPRoute (aigateway.envoyproxy.io/quota-policy-hash), so Envoy Gateway re-translates and the new limits, models, and bucket rules apply within seconds. Deleting a policy also updates routes in other namespaces. - Per-tenant limits from request metadata with
fromMetadata— Serve many tenants from one policy by letting an earlier filter, usuallyext_authz, decide each tenant's budget. SetfromMetadata.namespaceandfromMetadata.keyon anyQuotaValue, and have the filter writerequests_per_unitandunit(for exampleHOUR) to dynamic metadata. The staticlimitanddurationstay required and apply when the metadata is missing or malformed. A value of0suspends the tenant. Dynamic bucket limits - Weekly, monthly, and yearly windows —
durationnow accepts1w,1mo, and1yin addition to1s,1m,1h, and1d, so you can express weekly team budgets or monthly spend caps directly. These are Envoy rate limit windows, not calendar-aligned billing periods. Your rate limit service image must support theWEEK,MONTH, andYEARunits (see Upgrade Guidance). - Cross-namespace quota targets — A platform team can keep budgets in its own namespace while backends live with the teams that own them. Set
targetRefs[].namespaceto point aQuotaPolicyat anAIServiceBackendin another namespace, and authorize it with a ReferenceGrant in the backend's namespace (fromQuotaPolicy,toAIServiceBackend). Omitting the namespace keeps the previous same-namespace behavior and needs no grant. - Quota-aware backend failover (preview) — When a route has several prioritized backends and one runs out of quota, Agent Router can now route the request to the next backend that still has budget instead of returning 429. The quota rate limit service reports which (backend, model) groups still have quota, and Agent Router wraps quota-backed clusters in Envoy's
host_allowlistload balancing policy so exhausted backends are skipped. A 429 is returned only when every quota group on the route is exhausted. This needs an Envoy build that includeshost_allowlist, which Envoy v1.40.2 (shipped by Envoy Gateway v1.10.0) does not, and a rate limit service withRESPONSE_DYNAMIC_METADATA=true. On other data planes the route falls back to its normal load balancing and quotas are still enforced.
OpenAI APIs & Providers
- OpenAI Files API — Upload, list, retrieve, download, and delete files through the gateway with
/v1/files, so batch and fine-tuning workflows no longer need direct provider access. Uploads route on the multipartmodelfield. The gateway never returns the upstream file ID. It returns an encrypted ID that records the owning backend and model, so later requests route back to the right place. Files work with OpenAI-schema backends and need exactly onebackendRefon the matching rule. Files API - Decisions API on OpenAI and Azure OpenAI —
POST /v1/decisionsis passed through to OpenAI, and to Azure OpenAI at/openai/v1/decisions, with usage tracking, metrics, and tracing under the newdecisionsoperation. Questioninstructionscan be a string or a list of text and image parts. Streaming andfile_idinputs are not supported. - Claude Sonnet 5.5
between_toolsthinking — Send"thinking": {"type": "between_tools"}to skip up-front thinking while keeping short progress updates between tool calls. It works on/v1/messagesand Chat Completions for Anthropic, GCP Anthropic, AWS Anthropic, and AWS Bedrock Converse backends. Messages requests to Bedrock Converse now also forwardadaptivethinking. - Structured output and effort for Sonnet 5.5 and Haiku 5.5 — Chat Completions requests to
sonnet-5-5andhaiku-5-5now forwardreasoning_effortas Anthropicoutput_config.efforton AWS and GCP Anthropic, andresponse_formatJSON schemas asoutput_config.formaton GCP Anthropic. These were silently dropped before. reasoning_effort: highon Gemini 3 Pro —reasoning_effort: highnow maps tothinking_level: highon every Gemini 3 model instead of returning 400 on non-Flash models. See Breaking Changes for the related change tomedium.
Credentials & Security
- OpenAI workload identity federation — Stop storing long-lived OpenAI API keys. A
BackendSecurityPolicyoftype: OpenAICredentialsexchanges a SPIFFE JWT-SVID or an OIDC token for a short-lived OpenAI access token, rotates it before it expires, and sets the optionalOpenAI-OrganizationandOpenAI-Projectheaders. SetAI_GATEWAY_OPENAI_PROXY_URLon the controller to send the exchange through a proxy. Every SPIFFE-based policy authenticates as the controller's identity, so only trusted administrators should be allowed to create them. - Configurable credential pre-rotation window —
controller.credentialPreRotationWindow(default5m) sets how long before expiry the controller refreshes AWS, Azure, GCP, and OpenAI tokens. Raise it when your identity provider is slow or rate-limits token requests. - MCP OAuth resource identifier derived from the request —
protectedResourceMetadata.resourceon anMCPRouteis now optional. Leave it out and the gateway builds the identifier from each request's scheme, host, and MCP path, so one route can serve several hostnames or environments. Restrict the accepted hosts with route or listenerhostnames. Routes that set it keep exactly the same behavior. - Bring-your-own Secret for the MCP fallback seed —
controller.mcp.sessionEncryption.fallback.existingSecretreads the fallback session encryption seed from a Secret you manage, so GitOps users can rotate seeds without the seed appearing in Helm values.
Inference & Operations
- Weighted routing across multiple InferencePools — An
HTTPRouterule can now list severalInferencePoolbackends with Gateway APIweights, for example to shift traffic between model server fleets. Each pool keeps its own endpoint picker. This needs Envoy Gateway v1.11 or later. On Envoy Gateway v1.10, a rule with more than one InferencePool returns 500.AIGatewayRouterules still accept one InferencePool. - One endpoint picker call per request — A pool reached by several route matches used to get one endpoint picker filter per match, so each request ran several scheduling passes. That broke stateful scorers such as prefix-cache affinity. The extension server now adds one filter per pool.
- More reliable reconciliation — Finalizer writes use merge patches and are retried, so resources no longer end up without a finalizer or stuck terminating after a conflict. InferencePool status updates are retried too, so
status.parentslists every Gateway.
🔗 API Updates
QuotaPolicy.spec.targetRefs[].namespace—targetRefschanged fromLocalPolicyTargetReferencetoNamespacedPolicyTargetReference. The optionalnamespacedefaults to the policy's namespace. A cross-namespace target needs a ReferenceGrant, and an ungranted target is reported with the newRefNotPermittedreason. v1alpha1 only.QuotaValue.fromMetadata— Optional{namespace, key}(each 1-256 characters,^[A-Za-z0-9._:-]+$) naming the dynamic metadata struct that overrideslimitanddurationper request.limitanddurationare now explicitly required.fromMetadataunderserviceQuotais accepted but has no effect.QuotaValue.duration:1w,1mo,1y— Three new enum values.BackendSecurityPolicytype: OpenAICredentials— NewopenAICredentialsfield with optionalorganizationandprojectand a requiredworkloadIdentityFederationConfig(identityProviderID,serviceAccountID, and asubjectTokenthat is exactly one ofspiffeJWTSVIDoroidcExchangeToken). An OIDC client Secret in another namespace needs a ReferenceGrant. v1beta1 only.MCPRouteprotectedResourceMetadata.resourceis optional — No longer required in v1alpha1 or v1beta1. When it is unset, the identifier is derived from the request.- Embedded Envoy Gateway v1.10 types — The CRDs pick up new optional fields from Envoy Gateway v1.10, such as OIDC
cookieNamesandcodeVerifierTTLunderoidcExchangeToken, andhttp3,enableTrailers, andfailTrafficOnPanicin backend settings. CRD descriptions are now truncated to 200 characters to keep the CRD Helm release under the 1 MiB Secret limit. No schema types or validation changed. - New Helm values —
controller.credentialPreRotationWindow,controller.fileIDEncryption.{seed, existingSecret, iterations, fallback.seed, fallback.iterations}, andcontroller.mcp.sessionEncryption.fallback.existingSecret.
Deprecations
cache_creation_input_tokensin translated OpenAI-format responses — Still returned alongsidecache_write_tokensin v1.3. Readcache_write_tokensinstead. The legacy field was scheduled for removal in v1.3.0, and removal has moved to v1.4.0.
⚠️ Breaking Changes
- Legacy Gemini schema conversion removed — Chat Completions and tokenize requests to GCP Vertex AI now always send
response_formatschemas asresponseJsonSchemaand tool parameters asparametersJsonSchema, unchanged. Before, models whose name (ormodelNameOverride) didn't containgeminiplus2.5or3got a converted OpenAPI-subset schema. If you route to Gemini through an alias without amodelNameOverride, or still use a pre-2.5 model, check that your schemas work as standard JSON Schema. - Gemini
reasoning_effort: mediummapping —mediumnow maps tothinking_level: mediumon every Gemini 3 model exceptgemini-3-pro*and Flash-Lite Image, which gethigh. Before, every non-Flash model gothigh, so models such as Gemini 3.1 Pro now think less atmedium. Sendhighto keep the previous behavior. gen_ai_backendmetric label uses the backend's namespace — The backend metric attribute is now theAIServiceBackend's own<namespace>/<name>. Before, it used the route's namespace, so the value changes for routes that reference a backend in another namespace. Update dashboards and alerts that filter on the old value.- MCP OAuth protected resource metadata is served by the external processor —
/.well-known/oauth-protected-resource/...is now answered by the MCP proxy in the external processor sidecar instead of an Envoy direct response. It returns 503 while the route isn't loaded, for example during an external processor restart. The controller deletes the oldHTTPRouteFilteron upgrade. If a later release drops that cleanup, users upgrading directly from v1.2 or earlier will need to delete the stale filter by hand. GCPWorkloadIdentityProviderGo type removed — The unused type was removed from theapi/v1alpha1andapi/v1beta1Go packages. No CRD field referenced it, so manifests are unaffected. Only Go code that imports the type needs to change.
🐛 Bug Fixes
- GCP workload identity federation survives config reloads — With ADC credentials of type
external_account(for example from EKS or AKS), every GCP request failed withcontext canceledafter the first config reload until the external processor restarted. The token source now outlives the reload. - Anthropic overload errors arrive in the stream — When Anthropic sends
overloaded_errormid-stream on a Chat Completions request to AWS or GCP Anthropic, the client now receives a final SSE error event, so it can retry. Before, it got a generic 500 or a truncated stream. - Bedrock ping events are forwarded —
/v1/messagesstreams to AWS Anthropic backends now forwardpingkeepalive events, so clients such as Claude Code no longer hit idle timeouts during long tool calls. - Bedrock guardrail stops report
content_filter— A Bedrockguardrail_intervenedstop reason now maps tofinish_reason: content_filterinstead ofstop, so clients can tell a guardrail block from a normal completion. - Anthropic passthrough keeps the query string —
/v1/messagesrequests to anAnthropic-schema backend now forward the client's query string, for example?beta=true. - More request bodies redacted in debug logs — With redaction enabled, debug logs now redact
/v1/responses/input_tokens,/v1/messages/count_tokens, and tokenizechat_template_kwargs.
📖 Upgrade Guidance
Upgrading from v1.2 needs no CRD storage migration, and every new field is optional. Upgrade from v1.2.x, not directly from v1.1.x.
1. Upgrade Envoy Gateway first
Gateway API stays at v1.6.1. Upgrade the Envoy Gateway CRDs and controller to v1.10.0:
helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm \
--version v1.10.0 \
--set crds.gatewayAPI.enabled=true \
--set crds.envoyGateway.enabled=true \
| kubectl apply --force-conflicts --server-side -f -
helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
--version v1.10.0 \
--namespace envoy-gateway-system \
-f https://raw.githubusercontent.com/theagentrouter/agent-router/v1.3.0/manifests/envoy-gateway-values.yamlEnvoy Gateway v1.10 has its own breaking changes. Read its v1.10.0 release notes. The ones most likely to affect AI traffic:
- Rate-limited 429 responses now include
Retry-Afterby default. SetClientTrafficPolicy.spec.headers.disableRetryAfterHeader: trueto turn it off. - JWT requirement names change once on upgrade. On MCPRoutes with OAuth, requests on connections opened before the upgrade can get 403
Wrong requirement_nameuntil those connections drain. - OIDC session cookies move to AES-256-GCM, so users with an active OIDC session sign in again once. HTTP OIDC issuer URLs are rejected.
StrictLua validation now denies filesystem and environment access by default.
2. Keep Files API IDs stable
The chart now creates a random file ID encryption seed in the Secret <controller fullname>-file-id-encryption and reuses it on later helm upgrades. If you deploy with helm template or GitOps, the chart can't read back the seed it generated, so it generates a new one on every render and invalidates file IDs it has already issued. Supply your own Secret instead:
kubectl create secret generic file-id-seed -n envoy-ai-gateway-system \
--from-literal=seed="$(openssl rand -hex 32)"Then set controller.fileIDEncryption.existingSecret: file-id-seed. To rotate the seed later without breaking existing IDs, move the old seed to controller.fileIDEncryption.fallback.seed.
3. Update the quota rate limit service
If you use QuotaPolicy:
1w,1mo, and1ywindows need a rate limit service image that supports theWEEK,MONTH, andYEARunits. Agent Router's CI usesdocker.io/envoyproxy/ratelimit:63c7e9cc. Older images, including60d8e81bfrom thetoken_ratelimitexample, don't support these units.- To try quota-aware failover, run that image with
RESPONSE_DYNAMIC_METADATA=true, and use an Envoy build that includes theenvoy.load_balancing_policies.host_allowlistload balancing policy. Envoy v1.40.2 doesn't include it. Without it, routes keep their normal load balancing and quotas are still enforced. - Quota counters are now charged exactly once per request. Clients that sat just under a limit get slightly more headroom.
serviceQuotais still not enforced. Use per-model quotas.
4. Check Gemini requests and dashboards
- If you route to Gemini through an alias without
modelNameOverride, or still use a pre-2.5 model, check that yourresponse_formatand tool schemas are valid standard JSON Schema. - If you send
reasoning_effort: mediumto Gemini models other than 3 Pro and Flash, sendhighto keep the previous behavior. - If a route references an
AIServiceBackendin another namespace, update dashboards and alerts that filter ongen_ai_backend. The label now uses the backend's namespace.
5. Upgrade Agent Router
helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \
--version v1.3.0 \
--namespace envoy-ai-gateway-system
helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \
--version v1.3.0 \
--namespace envoy-ai-gateway-systemThe controller pod restarts because its arguments and environment changed. If you're upgrading from v1.2.0 rather than v1.2.1, Gateway pods that serve MCPRoutes also roll once.
📦 Dependency Versions
| Dependency | Version |
|---|---|
| Go | 1.27.3 |
| Envoy Gateway | v1.10.0 |
| Envoy Proxy | v1.40.2 |
| Gateway API | v1.6.1 |
| Gateway API Inference Extension | v1.6.2 |
| MCP Go SDK | v1.7.0 |
🙏 Acknowledgements
v1.3 came from 34 contributors. Special thanks to:
- Hrushikesh Patil, Ayush Sawant, Fernando Escolar, and Gavrish Prabhu, who took QuotaPolicy end to end with dynamic limits, cross-namespace targets, long windows, live updates, accurate charging, and quota-aware failover.
- Sivanantham, who built the OpenAI Files API.
- hustxiayang and Mike Toldov, who added the Decisions API, Claude Sonnet 5.5 thinking, and a run of translator fixes.
- junfeiliu531, who added OpenAI workload identity federation.
- zirain and Gerrit van Doorn, who improved InferencePool routing.
- Ignasi Barrera and Aishwarya Raimule, who continued hardening MCP and its OAuth support.
- Everyone who reported bugs, reviewed PRs, and joined the community meetings.
🔮 What's Next
Work already in flight for upcoming releases:
- Service-wide quotas, enforcing
serviceQuotaacross all models on a backend. - Quota-aware failover on released Envoy versions, once
host_allowlistships in an Envoy Gateway release. - Weighted InferencePool routing with Envoy Gateway v1.11.
- The stateless 2026-07-28 MCP specification, switched on end to end.
- Removing
cache_creation_input_tokensfrom translated OpenAI-format responses.
The roadmap is community-driven. Join us and help shape it.