fix(router): strip Anthropic beta fields from Vertex rawPredict body#2465
fix(router): strip Anthropic beta fields from Vertex rawPredict body#2465zanetworker wants to merge 1 commit into
Conversation
…redict body Claude Code v2.1.156+ sends `context_management` on every request. The direct Anthropic API accepts it, but Vertex AI rawPredict rejects it with HTTP 400 "Extra inputs are not permitted". The inference proxy was only stripping the `model` field for Vertex routes; extend it to also strip Anthropic-SDK-only beta fields listed in VERTEX_UNSUPPORTED_BODY_FIELDS. Fixes NVIDIA#2444 Signed-off-by: Adel Zaalouk <azaalouk@redhat.com>
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Validation: This is a concentrated router bug fix linked to confirmed issue #2444, with a clear reproduction and focused tests. I also checked @matthewgrossman’s maintainer recommendation on #2444 to strip only explicitly known incompatible fields; the implementation follows that direction.
Head SHA: a4d8409d635a7e0cad2ad8797e530bf8080c4670
Review findings:
- No blocking code correctness, regression, or security defects found.
- Update
docs/providers/google-vertex-ai.mdxto explain that Vertex Claude rawPredict requests stripcontext_management, while direct Anthropic requests preserve it. Users otherwise cannot tell that this requested beta behavior will not reach Vertex. Nodocs/index.ymlchange is needed. - Update
architecture/google-vertex-ai-provider.mdso its enumeratedprepare_backend_requesttransformations include unsupported-field filtering and nameVERTEX_UNSUPPORTED_BODY_FIELDSas the maintenance point.
Docs: Missing for this direct user-visible provider behavior change.
Next state: gator:in-review
|
I think we can skip the docs updates @gator. That missing behavior is a function of the hosted service, not the client omitting it. |
|
/ok to test a4d8409 |
|
Label |
Summary
context_management(and future Anthropic-SDK-only beta fields) from request bodies forwarded to Vertex AI rawPredict endpoints, fixing HTTP 400 "Extra inputs are not permitted" errors that break all Claude Code sessions throughinference.localwith a Vertex provider.Fixes #2444
Changes
crates/openshell-router/src/backend.rsVERTEX_UNSUPPORTED_BODY_FIELDSconstant listing Anthropic-SDK-only body fields that Vertex rawPredict rejects. New beta fields can be added to this single list.prepare_backend_request()to iterate over the strip list after removingmodelfor Vertex routes.crates/openshell-router/tests/backend_integration.rsproxy_vertex_strips_beta_fields_e2eintegration test with aVertexStrictBodyValidatormock that simulates Vertex's pydantic validation (rejects unknown fields).crates/openshell-router/src/backend.rs(unit tests)vertex_ai_body_strips_unsupported_beta_fields— verifies stripping for Vertex routes.direct_anthropic_preserves_beta_fields— verifies beta fields are preserved for direct Anthropic API routes.Testing
context_managementcauses 400, stripping it returns 200.context_managementthroughinference.localto real Vertex AI and got 200 OK.Long-term fix
This PR uses a static denylist to unblock users now. A more robust approach (provider-aware body sanitization using the
anthropic-betaheader or an allowlist per backend contract) is a bigger change that needs design discussion and is tracked separately in #2459.Checklist