Repository navigation
Native LLM routing, fail-over, load balancing, and smart model selection #15666
Replies: 1 comment
|
There are two existing pieces worth reusing. The factory already resolves separate task defaults. For example, Also, For a first native implementation, I would preserve the existing streaming boundary. The retry loop in I would start with explicit, capability-compatible targets and one budget across pre-first-chunk attempts, preserving the per-chunk stall timeout once streaming starts. Trace each attempted provider and the final selected model separately. Weighted routing could follow once failure handling is clear. These are source-based implementation suggestions, not a tested native-routing configuration. |
Uh oh!
There was an error while loading. Please reload this page.
Feature request
Onyx currently supports configuring multiple LLM providers/models, but routing between them is largely explicit: a model/provider is selected and requests are sent there.
For deployments using multiple providers or models, this often means running an additional gateway such as LiteLLM or Portkey purely to provide routing, fail-over, or load balancing.
This adds another service to operate, configure, secure, monitor, and keep compatible with Onyx.
It would be useful if Onyx had a native routing layer between its application logic and configured LLM providers.
What I would like to see
Allow administrators to define a logical model/router backed by multiple models and providers.
The user or Agent would select something like
General Assistant, while the router decides which actual model/provider should receive each request.The router could support several strategies:
Smart routing could start as rule-based configuration and optionally support a classifier/LLM-based router later.
Why implement this in Onyx?
Today, this kind of setup can be achieved by putting an external gateway in front of Onyx, for example:
Those projects are useful, but for relatively common routing requirements it feels unnecessary to run an additional infrastructure component solely for this purpose.
Onyx already knows quite a lot about the request:
That makes Onyx itself potentially a good place to make routing decisions.
It could also avoid situations where routing is split between Onyx and an external gateway, with each having only partial context about why a particular LLM call is being made.
Internal/background LLM calls
The same routing abstraction could also be useful for internal operations.
Classification, query expansion, naming, and other lightweight tasks may not need the same model used for the final user-facing response. Routing could therefore take both the request and the specific Onyx workload into account.
Possible first implementation
I don't think the first version needs a sophisticated AI router.
An incremental implementation could be:
Even the first few steps could remove the need for an external LLM gateway in many deployments while leaving room for more advanced routing later.
Would anyone be interested in expanding upon this/contributing towards getting this into Onyx?
All reactions