Auto-Router: what improvements do you want to see?

[krrish-berri-2]

Auto-Router today supports 3 flows

• Semantic routing (similarity to utterances)
• Complexity routing (heuristic)
• Adaptive routing (improve routing based on user feedback)

Looking for input on how we can simplify this + where this should go next. Some directions we're considering:

• Combine as 1 auto-router
• Add LLM based classification (e.g. use claude haiku/fast model) for model selection
• Cache-aware routing (decide if we should switch, if prompt caching is enabled)

If you're using Auto-Router (or thought about using it and decided against it), we'd like to hear what's missing or what would make it more useful for your setup.

[jeann2013]

From working on the Adaptive Router implementation in the Rust project, I think all three strategies should eventually converge into a single Auto-Router.

Adaptive routing shows that model selection is fundamentally a scoring problem. Semantic similarity, complexity, user feedback, cache probability and optional LLM classification can all be treated as signals contributing to a final routing score.

Rather than exposing multiple routing modes, I'd expose one Auto Router with configurable signals and weights. This keeps the API simple while allowing the routing engine to evolve internally.

I'd also like to see Adaptive Router learn from implicit production signals such as latency, retries, failures, cache hits and token usage, not just explicit user feedback. Finally, making routing decisions explainable would greatly improve debugging and production tuning.

[krrish-berri-2]

Combining the 3 is the right instinct. It's too fragmented. I'm working on it this week. Is this something you'd give feedback on? @jeann2013

[jeann2013]

I'd mainly focus on making the Auto-Router extensible from the start. If routing is signal based, adding new signals shouldn't require changing the core logic.

Key areas I'd look at:

-A clear signal interface.
-Configurable weights.
-Explainable routing decisions.
-Easy integration of future signals, for example: LLM classifiers or (especially plugins).

That should keep the API stable while allowing the routing engine to evolve.

[icsy7867]

Does complexity routing fall under this? I am not sure if auto routing uses the same logic, but it seems that I cannot hit the anthropic messages endpoint with a complexity router.

When using Bedrock on the backend, only Anthropic endpoints work with prompt caching. Chat Completions and Responses seem to always return 0 prompt caching. But if the routers worked with the anthropic endpoints, it would work I think.

[krrish-berri-2]

hi @icsy7867 can you share how you're doing this? any steps to repro would be great.

[krrish-berri-2]

also what version?

[icsy7867]

Of course.

My prod instance is on v1.89.0, but i have also tested on another deployment on versions 1.91.0, 1.92.0-rc.1 and 1.93.0-dev1. I am using litellm in kubernetes. I have tried several versions, but this was my last attempt

      - model_name: smart-router
        litellm_params:
          model: auto_router/complexity_router
          drop_params: true
          cache_control_injection_points:
            - location: message
              role: system
          complexity_router_config:
            tiers:
              SIMPLE: NVIDIA-Nemotron-3-Super-120B-A12B
              MEDIUM: Claude-Sonnet
              COMPLEX: gpt-5.4
              REASONING: claude-opus-4.8-dev
            tier_boundaries:
              simple_medium: 0.24   # Below 0.15 → SIMPLE
              medium_complex: 0.55   # 0.15-0.35 → MEDIUM
              complex_reasoning: 0.70  # 0.35-0.60 → COMPLEX, above → REASONING
            dimension_weights:
              tokenCount: 0.20        # Prompt length
              codePresence: 0.30      # Code-related keywords
              reasoningMarkers: 0.15  # "step by step", "think through", etc.
              technicalTerms: 0.25    # Domain-specific complexity
              simpleIndicators: 0.05  # "what is", "define", greetings
              multiStepPatterns: 0.03 # "first...then", numbered steps
              questionComplexity: 0.02 # Multiple questions
          complexity_router_default_model: Claude-Sonnet

I am hosting nemotron locally using VLLM. Claude-Sonnet, GPT 5.4 and claude-opus-4.8-dev are all being hosted in AWS Bedrock/Mantle GovCloud.

When using opencode, if I hit the model using an OpenAI frontend endpoint like completions or responses, everything works, but no prompt caching.

To use this in opencode, I set it up just like any other model/provider:

...
    "anthropic-smart": {
      "npm": "@ai-sdk/anthropic",
      "name": "Anthropic Smart Routing",
      "options": {
        "baseURL": "https://litellm-dev.company.com/v1",
        "apiKey": "sk-xxxxxxxxxxxxxxxxxxxxx"
      },
      "models": {
        "smart-router": {
          "name": "smart-router"
        }
      }
    },
 ...

Or you can try to force responses:

...
    "anthropic-smart": {
      "npm": "@ai-sdk/openai",
      "name": "Anthropic Smart Routing",
      "options": {
        "baseURL": "https://litellm-dev.company.com/v1",
        "apiKey": "sk-xxxxxxxxxxxxxxxxxxxxx"
      },
      "models": {
        "smart-router": {
          "name": "smart-router"
        }
      }
    },
 ...

Or chat completions:

...
    "anthropic-smart": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Anthropic Smart Routing",
      "options": {
        "baseURL": "https://litellm-dev.company.com/v1",
        "apiKey": "sk-xxxxxxxxxxxxxxxxxxxxx"
      },
      "models": {
        "smart-router": {
          "name": "smart-router"
        }
      }
    },
 ...

However, if I try to use Anthropic Messages, It basically gets angry at everything.

Bad Request: {"error":{"message":"litellm.BadRequestError: OpenAIException - {\"error\":{\"message\":\"13 validation errors:\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', \\\"literal['none','auto','required']\\\"), 'msg': \\\"Input should be 'none', 'auto' or 'required'\\\", 'input': {'type': 'auto'}, 'ctx': {'expected': \\\"'none', 'auto' or 'required'\\\"}}\\n  {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'mode'), 'msg': 'Field required', 'input': {'type': 'auto'}}\\n  {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'tools'), 'msg': 'Field required', 'input': {'type': 'auto'}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'type'), 'msg': \\\"Input should be 'allowed_tools'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'allowed_tools'\\\"}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceTypes', 'type'), 'msg': \\\"Input should be 'file_search', 'web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'file_search', 'web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\\\"}}\\n  {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'type'), 'msg': \\\"Input should be 'function'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'function'\\\"}}\\n  {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'server_label'), 'msg': 'Field required', 'input': {'type': 'auto'}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'type'), 'msg': \\\"Input should be 'mcp'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'mcp'\\\"}}\\n  {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceCustom', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceCustom', 'type'), 'msg': \\\"Input should be 'custom'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'custom'\\\"}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceApplyPatch', 'type'), 'msg': \\\"Input should be 'apply_patch'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'apply_patch'\\\"}}\\n  {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceShell', 'type'), 'msg': \\\"Input should be 'shell'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'shell'\\\"}}\\n\\n  File \\\"/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/serve/utils/api_utils.py\\\", line 48, in create_responses\\n    POST /v1/responses [{'type': 'literal_error', 'loc': ('body', 'tool_choice', \\\"literal['none','auto','required']\\\"), 'msg': \\\"Input should be 'none', 'auto' or 'required'\\\", 'input': {'type': 'auto'}, 'ctx': {'expected': \\\"'none', 'auto' or 'required'\\\"}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'mode'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'tools'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'type'), 'msg': \\\"Input should be 'allowed_tools'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'allowed_tools'\\\"}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceTypes', 'type'), 'msg': \\\"Input should be 'file_search', 'web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'file_search', 'web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\\\"}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'type'), 'msg': \\\"Input should be 'function'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'function'\\\"}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'server_label'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'type'), 'msg': \\\"Input should be 'mcp'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'mcp'\\\"}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceCustom', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceCustom', 'type'), 'msg': \\\"Input should be 'custom'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'custom'\\\"}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceApplyPatch', 'type'), 'msg': \\\"Input should be 'apply_patch'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'apply_patch'\\\"}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceShell', 'type'), 'msg': \\\"Input should be 'shell'\\\", 'input': 'auto', 'ctx': {'expected': \\\"'shell'\\\"}}]\",\"type\":\"Bad Request\",\"param\":null,\"code\":400}}. Received Model Group=smart-router\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"400"}}

More generically, when running in AWS Bedrock, if I use the anthropic compatible API, prompt caching seems to work just fine. I am getting cache creation and cache read tokens.

But if I hit the OpenAI compatible endpoints (Chat Completions or Responses) that use AWS Bedrock backend, I can't ever (And have never) gotten prompt caching to work. Though I have tried several things such as:

          cache_control_injection_points:
            - location: message
              role: system
            

To see if I can force the cache flags when doing this.

I do think this is a pretty big deal for anyone using AWS Bedrock (Being an organization that requires FedRAMP compliance, AWS GovCloud is great...). Our bill was $20,000+ more expensive last month mostly due to prompt caching issues (Not from this though... inference profiles which you guys awesomely have fixed!)

[krrish-berri-2]

Hey @icsy7867 working on a fix for your issues here #32974

[icsy7867]

You guys are awesome!

[akapur99]

Hey @jeann2013 - would love for you to take a look at updated autorouter --> converged to a single auto-router merging complexity and semantic routing: #32859 + added option for LLM based classification + basic logging ( #32943)

would love to know your thoughts on what level of observability you want on routing decisions, also what kind of plugins are you envisioning to be added?

[krrish-berri-2]

#32972 @jeann2013 is this what you wanted?

[jeann2013]

I was actually thinking they should be endpoint-aware rather than always running.

For example, a domain-classifier plugin probably only makes sense for chat/responses endpoints, while a budget-policy or tenant-policy plugin could apply to any request (chat, embeddings, image generation, rerank, etc.).

Maybe each plugin could declare something like:

plugins:
  - name: domain-classifier
    applies_to:
      - chat
      - responses

  - name: budget-policy
    applies_to:
      - "*"

That keeps the pipeline efficient while still allowing generic policies to run across all request types.

I'll also take a look at #32972. At first glance, this is very close to what I had in mind.

[jeann2013]

I took a look at the PR, and I think it's very close to what I had in mind. I really like the introduction of RoutingContext and the signals object—it gives plugins a clean way to collaborate without coupling them to provider-specific request formats.

One thing I wonder about for future iterations is whether candidate_models could eventually evolve beyond simple include/exclude filtering into a scoring mechanism. Some plugins may not want to eliminate models entirely, but instead express preferences ("prefer Claude", "penalize expensive models", etc.) and let the AutoRouter combine those signals.

I also think this opens the door for richer observability. Since plugins now form a routing pipeline, it would be great if each plugin could emit a structured trace (inputs, outputs, candidate changes) that could be surfaced in logs or exported to OpenTelemetry/Langfuse for debugging.

Overall though, this is very close to the mental model I was describing.

[akapur99]

Speaking of observability -

for the decision trace are you expecting this - in the admin logs only? or also surfaced to the end-user?

[jeann2013]

Speaking of observability -

for the decision trace are you expecting this - in the admin logs only? or also surfaced to the end-user?

I see the decision trace primarily as an observability feature for platform operators, so I'd expect it to show up in admin logs, tracing systems (OpenTelemetry, Langfuse, Datadog), and maybe the LiteLLM UI.

For end users, I wouldn't expose it by default. Most users just care about getting the best response, not the routing internals.

That said, I could see value in making it optionally available (e.g. a debug flag or API option) for teams building their own dashboards or debugging routing behavior in development. So my preference would be:

• Default: admin/observability only.
• Optional: exposed via an API/debug mode when explicitly requested.

That keeps the default experience clean while still giving developers full visibility when they need it.

[jeann2013]

One funny side effect of this discussion: I actually started thinking about litellmhub.ai 😄. If third-party plugins become a thing, I could easily imagine a community hub where people publish, rate, and discover routing plugins.

[krrish-berri-2]

Proposal: allow budget-based limits on models within the auto-router.

Two budget mechanisms already exist in the router and both work by filtering a deployment out of the candidate pool once exhausted, not by erroring:

• Global: max_budget / budget_duration on litellm_params, enforced by RouterBudgetLimiting
• Per-key: model_max_budget on the virtual key, enforced by _PROXY_VirtualKeyModelMaxBudgetLimiter (subclasses the above, same filter logic, scoped to key+model)

Neither is visible to auto-router today. complexity_router.py's tier-list picker (_pick_from_tier_value) does a plain random.choice() over a tier's model list before deployment selection runs, so a budget-exhausted model in the list still gets picked and the request fails downstream instead of falling through to another model in the tier.

Proposal: route tier-list selection through the same budget-filtering pool instead of picking blind. No new config surface needed, this reuses max_budget/budget_duration and model_max_budget as they exist today.

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          COMPLEX: [fable, claude-sonnet-4]
  - model_name: fable
    litellm_params:
      model: anthropic/claude-fable-5
      max_budget: 50
      budget_duration: 1d

or per-key, via the existing model_max_budget field on a virtual key, with no auto-router-specific config at all.

Separate question worth its own thread: what happens when every model in a tier is exhausted. Bump to the next tier, drop down, or error. Probably needs to be configurable rather than a fixed default.