Using adaptive router with mixed Anthropic+Other providers

[suranjanotti]

Hi!

Has anyone managed to make the adaptive router work with anthropic provider? It seems all responses from Anthropic are cryptographically signed and adding the drop_params or modify_params doesn't fix the multi-turn thinking block errors that Anthropic provider returns.

[krrish-berri-2]

can i see your configuration?

[suranjanotti]

# Minimal repro: adaptive router mixing an Anthropic provider with a
# non-Anthropic (hosted_vllm / OpenAI-compatible) backend.
# modify_params + drop_params are ON but the multi-turn thinking-block
# error from the anthropic/ provider still surfaces.

litellm_settings:
  modify_params: true   # meant to drop `thinking` when thinking_blocks are missing from tool_calls
  drop_params: true     # drop unsupported params (reasoning effort, etc.) before forwarding

model_list:
  # ── Non-Anthropic backend (local vLLM, OpenAI-compatible) ──
  - model_name: qwen-dense
    litellm_params:
      model: hosted_vllm/qwen36-27b-dense-mtp
      api_base: http://LOCAL_VLLM_HOST:8003/v1
      api_key: fake-key
      input_cost_per_token: 0.000000003
    model_info:
      input_cost_per_token: 0.000000003
      output_cost_per_token: 0.00000003
      max_input_tokens: 192000
      max_output_tokens: 32768
      supports_reasoning: true
      adaptive_router_preferences:
        quality_tier: 1
        strengths: [factual_lookup, general, writing, code_generation]

  # ── Anthropic provider models ──
  - model_name: claude-haiku
    litellm_params:
      model: anthropic/claude-haiku-4-5-20251001
      api_key: os.environ/ANTHROPIC_API_KEY
      input_cost_per_token: 0.00000025
    model_info:
      input_cost_per_token: 0.00000025
      output_cost_per_token: 0.0000025
      max_input_tokens: 200000
      max_output_tokens: 8192
      adaptive_router_preferences:
        quality_tier: 1
        strengths: [factual_lookup, general, writing, code_generation]

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY
      input_cost_per_token: 0.000003
    model_info:
      input_cost_per_token: 0.000003
      output_cost_per_token: 0.000015
      max_input_tokens: 1000000
      max_output_tokens: 64000
      adaptive_router_preferences:
        quality_tier: 2
        strengths: [code_generation, code_understanding, analytical_reasoning, writing]

  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
      input_cost_per_token: 0.000005
    model_info:
      input_cost_per_token: 0.000005
      output_cost_per_token: 0.000025
      max_input_tokens: 1000000
      max_output_tokens: 128000
      adaptive_router_preferences:
        quality_tier: 3
        strengths: [technical_design, analytical_reasoning]

  # ── The adaptive router mixing both providers ──
  - model_name: adaptive
    litellm_params:
      model: auto_router/adaptive_router
      adaptive_router_config:
        available_models:
          - qwen-dense
          - claude-haiku
          - claude-sonnet
          - claude-opus
        weights:
          quality: 0.1
          cost: 0.9
    model_info:
      max_input_tokens: 192000
      max_output_tokens: 32768

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

[Ganesh-403]

@suranjanotti
I don't think this is something that drop_params or modify_params can solve.

The error you're referring to ("multi-turn thinking block") comes from Anthropic's conversation protocol, where thinking blocks are cryptographically signed and must be preserved across turns. Those signatures are part of the conversation state, so simply dropping or rewriting request parameters won't make an invalid conversation valid again. :contentReference[oaicite:0]{index=0}

In your setup, the adaptive router is selecting between:

• Anthropic models
• OpenAI-compatible vLLM

These providers have different conversation semantics. If one turn is handled by Anthropic and the next turn is routed to (or from) a different provider, the message history may contain provider-specific content (e.g. Anthropic thinking blocks) that another provider doesn't understand—or vice versa.

modify_params and drop_params primarily normalize request parameters (unsupported OpenAI params, provider-specific options, etc.). They don't rewrite the conversation history or strip provider-specific message content from previous turns. :contentReference[oaicite:1]{index=1}

A few things I'd check

1. Does the issue reproduce on the first turn?

   • If it only happens on turn 2+, that strongly suggests it's related to conversation history rather than request parameters.

2. Are you routing each turn independently?

   • If the adaptive router picks a different provider on subsequent turns, you'll end up mixing provider-specific conversation state.

3. Try pinning a conversation to a single provider

   • For example:
     • Anthropic-only adaptive router
     • Non-Anthropic adaptive router
   • Then compare that with a mixed-provider router.

My suspicion

From what I can tell, the adaptive router currently makes routing decisions per request, but I couldn't find documentation stating that it normalizes or removes provider-specific conversation artifacts (such as Anthropic thinking blocks) when switching providers mid-conversation. :contentReference[oaicite:2]{index=2}

If that's the case, then this is less a configuration issue and more a limitation (or bug) when mixing providers with incompatible conversation formats.

Recommendation

If you need multi-turn conversations today, I'd recommend keeping each conversation pinned to a single provider family (e.g. all Anthropic or all OpenAI-compatible) and using the adaptive router to select the provider at conversation start, rather than allowing it to switch providers between turns.

If you can reproduce this with:

• Turn 1 → claude-sonnet
• Turn 2 → router selects qwen-dense

(or vice versa) and the only difference is the provider switch, then I'd consider that a strong candidate for a LiteLLM bug or unsupported edge case.

One question: are you routing every turn through the adaptive router, or is only the initial request routed and subsequent turns intended to stay on the selected model? That distinction will help determine whether this is expected behavior or a routing regression.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.

[dlanov]

The issue is not drop_params or modify_params. The root cause is that Anthropic's extended thinking blocks must survive the entire round-trip unmodified, and LiteLLM's adaptive router breaks this contract when it retries or falls back across models.

What's actually happening

When you use Claude with thinking enabled, Anthropic returns a response that includes a thinking content block alongside the text block:

  {
    "content": [
      {"type": "thinking", "thinking": "...", "signature": "..."},
      {"type": "text", "text": "..."}
    ]
  }

The signature field is a cryptographic proof generated by Anthropic. On the next turn, you must echo that entire thinking block back verbatim in the assistant message because Anthropic validates the signature server-side and rejects the request if it's missing, truncated, or modified.

The adaptive router breaks this in two ways:

1. On retry: when the router retries a failed request it reconstructs the messages array. Any thinking content blocks from previous turns get dropped or have their structure altered during serialisation, invalidating the signature.
2. On fallback to a non-Anthropic model: if the router falls back to GPT-4o or Gemini, those models have no concept of the thinking block. The router strips it from the messages, which makes the fallback succeed but corrupts the conversation state if you later return to Claude.

Fix is to isolate extended thinking models into their own model group

Do not mix Anthropic extended thinking models with other providers in the same adaptive router group. Create a dedicated group for thinking-enabled Claude calls:

  model_list:
    # Group 1: extended thinking — Anthropic only, no fallback
    - model_name: claude-thinking
      litellm_params:
        model: anthropic/claude-opus-4-5
        thinking:
          type: enabled
          budget_tokens: 8000
      model_info:
        id: claude-thinking-1

    # Group 2: standard models — adaptive routing fine here
    - model_name: standard-llm
      litellm_params:
        model: anthropic/claude-sonnet-4-5
    - model_name: standard-llm
      litellm_params:
        model: openai/gpt-4o
    - model_name: standard-llm
      litellm_params:
        model: gemini/gemini-2.0-flash

  router_settings:
    routing_strategy: usage-based-routing-v2   # adaptive
    # only apply adaptive routing to standard-llm group

Then route thinking-enabled conversations exclusively to claude-thinking and never through the adaptive router.

If you need redundancy for extended thinking calls, then use fallback only within Anthropic models of the same capability tier but not cross-provider:

  router_settings:
    fallbacks:
      - claude-thinking: ["claude-thinking-fallback"]   # another Anthropic thinking model
    # NOT: claude-thinking: ["gpt-4o"]  ← this will corrupt thinking blocks

Why drop_params doesn't help:
drop_params drops unsupported request parameters before sending to a provider. The problem here is on the response side. Anthropic sends back a signed thinking block that your code must preserve in subsequent requests. There's no LiteLLM parameter to drop because the issue is in your conversation history management, not the outbound request shape.

[unusdon]

Building on @dlanov's answer — that diagnosis is right. Two supplements that might help you confirm and mitigate:

The fingerprint you'll see in your logs

When the router mangles a thinking block, the failure surfaces as:

• Anthropic direct: HTTP 400 with a body of the form {"type":"error","error":{"type":"invalid_request_error","message":"messages.N.content.M: input_message... thinking block signature ..."}} — the literal substring "thinking" in the error body is the fingerprint.
• Anthropic on Bedrock: the same underlying error wrapped in a Bedrock ValidationException.

From the LiteLLM proxy client side you'll get a BadRequestError whose response body contains "thinking". That's the grep test to distinguish this failure from every other 400 (rate limit, bad shape, etc.).

Minimal reproduction

Isolates the failure mode without your full router config in the loop:

import litellm
litellm.set_verbose = True

MODEL = "anthropic/claude-opus-4-5"

resp1 = litellm.completion(
    model=MODEL,
    thinking={"type": "enabled", "budget_tokens": 4000},
    messages=[{"role": "user", "content": "Think about 2+2 briefly, then answer."}],
)

# grab the assistant reply verbatim, including the signed thinking block
prior = resp1.choices[0].message.model_dump()

# turn 2, correct path — echo the prior message unchanged
resp2_ok = litellm.completion(
    model=MODEL,
    thinking={"type": "enabled", "budget_tokens": 4000},
    messages=[
        {"role": "user", "content": "Think about 2+2 briefly, then answer."},
        prior,
        {"role": "user", "content": "Now do 3+3."},
    ],
)

# turn 2, the router-mangling scenario — strip content blocks a naive router
# doesn't understand, watch it fail
mangled = dict(prior)
mangled["content"] = [b for b in prior["content"] if b.get("type") != "thinking"]

resp2_broken = litellm.completion(  # raises BadRequestError with "thinking" in the body
    model=MODEL,
    thinking={"type": "enabled", "budget_tokens": 4000},
    messages=[
        {"role": "user", "content": "Think about 2+2 briefly, then answer."},
        mangled,
        {"role": "user", "content": "Now do 3+3."},
    ],
)

If resp2_ok succeeds and resp2_broken raises BadRequestError mentioning thinking, you've confirmed the same failure mode you're seeing in your adaptive router.

Migrating a conversation that's already in the broken state

If a running conversation is already mangled (mixed history, some thinking blocks preserved, some dropped), you can't retroactively repair it — Anthropic validates against the current history the model sees. Cleanest recovery: strip all thinking blocks from history for one turn (the messages API accepts this — the strict signature-required rule applies when the immediately-prior assistant turn contains a thinking block), then resume from the next turn onward with the isolated-group config @dlanov posted above.