Skip to content

fix(router): strip Anthropic beta fields from Vertex rawPredict body#2465

Open
zanetworker wants to merge 1 commit into
NVIDIA:mainfrom
zanetworker:fix/vertex-strip-beta-fields-2444
Open

fix(router): strip Anthropic beta fields from Vertex rawPredict body#2465
zanetworker wants to merge 1 commit into
NVIDIA:mainfrom
zanetworker:fix/vertex-strip-beta-fields-2444

Conversation

@zanetworker

@zanetworker zanetworker commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Strip context_management (and future Anthropic-SDK-only beta fields) from request bodies forwarded to Vertex AI rawPredict endpoints, fixing HTTP 400 "Extra inputs are not permitted" errors that break all Claude Code sessions through inference.local with a Vertex provider.

Fixes #2444

Changes

crates/openshell-router/src/backend.rs

  • Added VERTEX_UNSUPPORTED_BODY_FIELDS constant listing Anthropic-SDK-only body fields that Vertex rawPredict rejects. New beta fields can be added to this single list.
  • Extended prepare_backend_request() to iterate over the strip list after removing model for Vertex routes.

crates/openshell-router/tests/backend_integration.rs

  • Added proxy_vertex_strips_beta_fields_e2e integration test with a VertexStrictBodyValidator mock that simulates Vertex's pydantic validation (rejects unknown fields).

crates/openshell-router/src/backend.rs (unit tests)

  • Added vertex_ai_body_strips_unsupported_beta_fields — verifies stripping for Vertex routes.
  • Added direct_anthropic_preserves_beta_fields — verifies beta fields are preserved for direct Anthropic API routes.

Testing

  • 84 tests pass (66 unit + 18 integration), clippy clean, format clean.
  • Verified against real Vertex AI: context_management causes 400, stripping it returns 200.
  • End-to-end on OpenShift: built fixed supervisor, deployed to workload cluster, sent context_management through inference.local to real Vertex AI and got 200 OK.

Long-term fix

This PR uses a static denylist to unblock users now. A more robust approach (provider-aware body sanitization using the anthropic-beta header or an allowlist per backend contract) is a bigger change that needs design discussion and is tracked separately in #2459.

Checklist

  • Tests added
  • Clippy clean
  • Format clean
  • Signed-off-by (DCO)
  • Conventional commit

…redict body

Claude Code v2.1.156+ sends `context_management` on every request. The
direct Anthropic API accepts it, but Vertex AI rawPredict rejects it with
HTTP 400 "Extra inputs are not permitted". The inference proxy was only
stripping the `model` field for Vertex routes; extend it to also strip
Anthropic-SDK-only beta fields listed in VERTEX_UNSUPPORTED_BODY_FIELDS.

Fixes NVIDIA#2444

Signed-off-by: Adel Zaalouk <azaalouk@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Validation: This is a concentrated router bug fix linked to confirmed issue #2444, with a clear reproduction and focused tests. I also checked @matthewgrossman’s maintainer recommendation on #2444 to strip only explicitly known incompatible fields; the implementation follows that direction.
Head SHA: a4d8409d635a7e0cad2ad8797e530bf8080c4670

Review findings:

  • No blocking code correctness, regression, or security defects found.
  • Update docs/providers/google-vertex-ai.mdx to explain that Vertex Claude rawPredict requests strip context_management, while direct Anthropic requests preserve it. Users otherwise cannot tell that this requested beta behavior will not reach Vertex. No docs/index.yml change is needed.
  • Update architecture/google-vertex-ai-provider.md so its enumerated prepare_backend_request transformations include unsupported-field filtering and name VERTEX_UNSUPPORTED_BODY_FIELDS as the maintenance point.

Docs: Missing for this direct user-visible provider behavior change.

Next state: gator:in-review

@johntmyers johntmyers added the gator:in-review Gator is reviewing or awaiting PR review feedback label Jul 24, 2026
@johntmyers

Copy link
Copy Markdown
Collaborator

I think we can skip the docs updates @gator. That missing behavior is a function of the hosted service, not the client omitting it.

@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test a4d8409

@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Jul 24, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for a4d8409. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:approval-needed Gator completed review; maintainer approval needed test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inference proxy: support or strip unsupported beta parameters when forwarding to Vertex rawPredict

2 participants