Auto-router - Content-Aware Preference-Aligned Routing

[krrish-berri-2]

We're discussing a possible improvement to auto-routers on LiteLLM. The goal here is to help reduce costs, while maintaining quality for users.

The idea is to onboard models with descriptions on what they do well. Then expose 2 auto-router models - 'auto' and 'premium'.

'auto' - would route to the cheapest, competent model for the task
'premium' - would filter for more expensive models and pick from among them (e.g. just opus and gpt-5.4-pro, not sonnet).

The routing would occur, based on the models the user has access to.

1. Example config.yaml

model_list:
  - model_name: claude-sonnet-4-6
    litellm_params:
      model: anthropic/claude-sonnet-4-6
    model_info:
      description: wins for boilerplate, CRUD, tests, docs — ~98% of Opus quality at 1/5 the cost
  - model_name: claude-opus-4-6
    litellm_params:
      model: anthropic/claude-opus-4-6
    model_info:
      description: wins for deep reasoning, architectural decisions, 1M-context codebase analysis, Agent Teams
  - model_name: gpt-5.3-codex
    litellm_params:
      model: openai/gpt-5.3-codex
    model_info:
      description: wins for long multi-step sessions: large refactors, codebase migrations, sustained multi-file work, security audits — strongest on SWE-Bench Pro specifically

2. User Request

curl https://litellm-proxy/v1/chat/completions \
  -H "Authorization: Bearer sk-user-key" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "write unit tests for this UserService class"}]
  }'

Response headers expose the decision:

X-LiteLLM-Router: auto
X-LiteLLM-Routed-Model: claude-sonnet-4-6

3. Preview Endpoint

curl https://litellm-proxy/v1/routing/preview \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "..."}]}'

# → {"router": "auto", "chosen_model": "claude-sonnet-4-6",
#    "candidates": [{"model": "...", "score": 4.2, "price": 3.0}, ...]}

How it might work

Request arrives with model=auto
      │
      ▼
Resolve user's accessible models via existing key/team ACL
  → e.g. [sonnet, opus, codex]
      │
      ▼
For each accessible model, fetch description
  → admin override in model_info.description (if set)
  → else LiteLLM's built-in registry
      │
      ▼
BM25-score the user prompt against each description
  → e.g. sonnet:4.2, opus:2.1, codex:5.8
      │
      ▼
Selection: take top-K by score, pick the cheapest among them
      │
      ▼
Substitute chosen model_name into the request
      │
      ▼
Hand off to existing LiteLLM router
  (load balancing, fallbacks, rate-limit handling — unchanged)
      │
      ▼
Stream response back, add X-LiteLLM-Routed-Model header

[DmitriyAlergant]

Personally I don't think this kind of auto-routing belongs to LiteLLM at all. Not looking to use it at any of our implementations.

The client application (agent) generally has more information for routing, that a single LLM request to LiteLLM may carry; Including, particularly, stickiness of the conversation thread or an agentic session. At least not without some very advanced machinery (e.g. what OpenRouter with their huge R&D resources may have behind the scenes for their auto).

Even for "pick the cheapest" how do you even do that ignoring sticky caching? Caching has 10x impact on costs. Also frontier LLMs are becoming more and more pickier on continuing existing conversation threads, requiring encrypted thoughts etc - switching the horse mid-conversation is going to be non-trivial.

As the context grows longer with multiple conversation turns and tool calls, what good would BM25 do by simply matching all of that against brief model description? T

Besides, with frontier LLMs breathing neck-to-neck into each other how do you even differentiate them by "CRUD, tests, deep reasoning, architectural decisions, 1M-context codebase analysis, large refactors, codebase migrations, sustained multi-file work, security audits" - they are all doing that quite well. There is no one who can meaningfully maintain this kind of differentiation especially as the situation in benchmarks changes week after week. At the very least, I would refocus the demonstrative example (above) to "cheap local gemma vs frontier Sonnet/Opus/gpt LLM".

Long story short, the entire concept of auto-routing at the gateway level feels to me stuck in "2024" then what would be relevant in mid-2026 with all the complexity we are having today, given how much there remains to do for stabilizing and maintaining the core things LiteLLM is supposed to be doing well.

Leave model choices to the client applications/agents.

[krrish-berri-2]

thanks @DmitriyAlergant that's helpful feedback.

How do you feel litellm can increase in value, for your team, over the next several years?

Also reached out over linkedin, for a deeper conversation here!

[krrish-berri-2]

Noting details from our conversation @DmitriyAlergant :

• agent better than gateway
  • agent maintains session stickiness
  • stickiness is important for caching, consistency, etc.
  • a conversation started w/ model A, should stick with it.
  • Don’t swap model mid conversation
  • breaking post thinking + caching would lead to costs and wouldn’t be great
  • “switching something behind the scenes and not being sticky is a bad decision.”

[marty-sullivan]

I generally agree with this, most model providers have their own internal "auto-routing" functionality that I would say LiteLLM should focus on having support for, along with all of the other core features of the gateway itself (LLM gateway, MCP gateway, A2A gateway, etc.), but having a cross-provider "auto-routing" seems like a problem that is hard to provide a solution for.

I don't see our users generally mingling between model providers for a particular task anyway. Usually, the choice in model is around model capabilities (e.g. coding for Claude/Codex, multi-modal input/output for Gemini, open-weight models for reproducible research) so having an auto-routing feature between (for example) gpt-5.x, claude models, gemini models and the dozens of other models our users are given access to would both 1) probably not work that great and 2) be routed for a reason outside why a person would choose a particular model family in the first place (cost savings over specific functionality).

Perhaps it does make sense in specific contexts though, like when someone is using the gateway while using Claude Code / Codex CLI. But even then, it seems like a very hard problem for you to solve with hard-to-measure outcomes. Plus, in general, most of our users just use the three flavors of Claude in CC and aren't choosing to ever use, e.g. gpt-5.x-codex with Claude Code.

So yes, to reiterate @DmitriyAlergant's point, I think there are a lot of core features I'd prefer be focused on improving vs this type of new functionality.

[krrish-berri-2]

around model capabilities (e.g. coding for Claude/Codex

isn't this exactly multi-provider? or are you saying there's not much need to go across claude and gpt?

[krrish-berri-2]

in general, most of our users just use the three flavors of Claude in CC and aren't choosing to ever use, e.g. gpt-5.x-codex with Claude Code.

got it

[krrish-berri-2]

Sharing feedback from another user:

• where would he find this useful?

  • if i had a client who didn’t know or had the capability to know
  • if the client thought that it was using a more advanced model
  • for these elements route to nemotron, for these elements route to qwen
  • start off with a regex/label system
    • that’s when the system gets really powerful, so i can break down the core components of the protocol and i can route to models based on what they’re good at
    • especially in the lab settings
    • the perf i’m getting in nemotron, far exceeds what i can do on qwen
    • but nemotron isn’t the best for reasoning vs. qwen
    • so i have to pick and choose the right model for the right task
    • if i have a routing table that lets me do both with regex + semantic, then that’s really powerful
      • it lets me decouple the request calling chain for opus

• “in my opinion, an auto-router should be able to compute whether an endpoint is down and what it’s latency is.”

  • “is uptime, is latency, is these internal states of the endpoint?”
    • “selecting the best endpoint, based on availability, client-fit, and does loadbalancing”
      • “i have these 3 servers, i have this client request, how do i make it work?”
      • “e.g. i have a request without vision, so i don’t need to go to a model which needs vision.”
      • “i can go to a model for tool calls or programmatic access.”
      • make a tiered system for granular stuff - availability x capability x paradigm for an auto-route
        • litellm has almost all of this, but it’s in different parts of the system
        • e.g. tags, label routing - it’s not in 1 endpoint
          • i can setup label based routing, i can setup health checks, i can setup semantic routing but i can’t package this into 1 endpoint across models, across everything in 1 structure, and make them aware of each other

[csoni-cweave]

The way I see it, there's a huge difference between "auto-routing" (the system magically picks the best model) and what's actually being proposed here - which is closer to "the admin tells the gateway what each model is good at, and the gateway respects that at request time." One of these is an OpenRouter-style black box. The other is a cost governance primitive and I, from the enterprise point of view, am interested in the second one.

The person deploying the agent is almost never the person paying the bill. Developers hardcode Opus for everything because it's the best and they don't care about cost - including me (when acting as a developer and not a platform owner). Meanwhile the platform team is staring at a monthly bill that's 4-5x (this is real data) what it needs to be because - every small request (even a one line change), unit test generation and boilerplate CRUD are going to be the most expensive model available. The answer today is "go talk to every agent developer and ask them to use cheaper models for simple tasks." That doesn't scale since there is no enforcement. Same reason we don't ask every micro-service to implement its own rate limiting - we put it in the gateway.

re: the point about session stickiness and caching is spot on and I don't think anyone should ship this without solving for that. But that's an implementation constraint, not a reason to kill the feature. Route at session initiation, stick for the lifetime of the conversation, done. The X-LiteLLM-Routed-Model header already gives the client what it needs to pin. If someone tries to re-route mid-conversation and breaks prompt caching, that's a bug, not an architecture problem.

re: users not mingling between providers - I'd argue that's mostly true cross-provider (nobody's bouncing between Claude and GPT mid-task), but intra-family routing is already a pattern people are doing by hand. Haiku for classification, Sonnet for code gen, Opus for architectural reasoning. Or mini for extraction, 5.3 for generation, 5.4-pro for complex planning.
Enterprises are literally building this logic into their agent code today. Pulling it into the gateway as admin-configurable policy just means you don't need every team to reinvent the wheel.

The idea of availability × capability × paradigm as a unified routing surface is spot on. LiteLLM already has health checks, tag routing, label routing, semantic routing - they're just scattered across different configs and not aware of each other. Composing them into one declarative routing decision is the actual feature request here.

A few things that would make this actually work for enterprise adoption:

• Structured model cards instead of free-text descriptions for BM25. Nobody's going to trust fuzzy text matching in production. Give me capabilities: [code-gen, unit-tests, boilerplate], tier: standard, and let me write deterministic rules against those. Regex/label matching as the primary routing mechanism, semantic as a fallback - not the other way around.

• The preview endpoint from the proposal (/v1/routing/preview) is a great way to pre-determine/benchmark requests for routing. Platform teams need to audit routing decisions before flipping the switch.

• Cost saving in the response headers(or just logging it on LiteLLM side) would be a great feature for the finance team to review.

I don't think this is a 2024 concept at all. If anything it's the opposite - it's a 2026 cost governance requirement that's emerging precisely because usage-based AI spend is now big enough to show up on the P&L. Opus 4.7 is already using xHigh effort by default and this will keep getting worse over time.

[krrish-berri-2]

i can't, but do you think the point is moot then?

[DmitriyAlergant]

@krrish-berri-2 yes, given where the industry is today, I do think that any simplistic, non-intelligent (not using another LLM to assess task complexity) matching based on verbal description whether hardcoded (regexps) or "similiarity scoring" (BM25 etc) is moot.

[DmitriyAlergant]

@csoni-cweave don't you think that this kind of Task-level complexity-based routing should rather come from Agents side - e.g. Claude Code or OpenCode build in an "auto" model (to downgrade to Haiku/Sonnet). This is something they could do using a light LLM that's grounded in project context, current user task, and, ideally, previously accumulated notes/experience (e.g. fixing flaky CI tests in this project often ended up being a hard task).

[DmitriyAlergant]

Even WITH the LLM classifying task complexity, one can only strive to meaningfully route between model tiers of the same vendor (e.g. Nano vs Mini vs full; or Haiku vs Sonnet vs Opus), not advise between vendors (Gemini vs Anthropic vs OpenAI). The differences across vendors are too subtle and genuinely hard to benchmark, plus evolving very quickly with each release.

[csoni-cweave]

these are subjective thoughts but imo - proprietary AI clients like Claude Code will always optimize for their own token consumption at the end of the day. That's their business model. They're not incentivized to route you to a cheaper model, they're incentivized to keep you on Opus (or whatever gives the best ROI). And honestly the Claude Code (which is amazing) narrative is so strong right now that most developers don't even question it and everyone defaults to Opus max/fast mode because "it's the best" and nobody's doing the math on whether Sonnet+Haiku would've gotten you 98% of the way there for 1/5 the cost.

As an enterprise platform owner, I want to be the one making that call. I want to benchmark different vendors and model tiers over time, look at my own data, and from that classify which agent requests or task patterns can be safely routed to a cheaper tier without meaningful quality loss. The differences across vendors are shrinking like you said - which actually makes the case stronger for routing. If Sonnet and GPT-5.3 are neck and neck for 80% of tasks, then the decision should come down to cost and latency, and I want that decision at my layer, not delegated to the model vendor's client.

The other thing - LiteLLM already has A/B testing as a feature. I can see a really natural evolution where content-based routing becomes an extension of A/B experiment results over time. You run experiments, you collect the data on which model performed how on which task types, and then you encode those learnings into routing rules- kinda data-driven policy router.

Apart from this, I also made a request to @krrish-berri-2 to incorporate something similar to behavioral signals (https://docs.planoai.dev/concepts/signals.html) per request which can be a bedrock for routing decisions or custom model/agent training (RL etc) over time. But that's a different conversation and that's the developer mode talking :)

[atxapple]

Sharing operating experience rather than a feature ask, since I run a tier-based router in production and this proposal is close to what we built:

• Predictability beat cleverness for us. A fixed, versioned tier→model mapping (updated deliberately, server-side) turned out easier for users to trust than per-request content scoring — when routing varies per prompt, debugging "why was this answer worse/slower/more expensive" gets hard. If you ship BM25-per-request, the /routing/preview endpoint and the X-…-Routed-Model response header in this proposal are exactly the right mitigations; I'd call the header mandatory, not optional.
• Two router aliases ("auto"/"premium") may be too few. We ended up with four classes (speed / balance / intelligence / reasoning) because "cheap but competent" hides a real split between latency-sensitive and quality-sensitive light tasks.
• Let admins pin. Whatever the scorer says, an escape hatch that pins a key/team to a concrete model preserved trust when a routing change regressed one team's evals.

Disclosure: I'm the founder of TierUp (https://tierup.ai/?ref=github), an OpenAI-compatible API that exposes this idea as tier-1…tier-4 model names — mentioning it for context on where these lessons come from, not as a substitute for building it into LiteLLM.