Summary
The knn routing backend needs several endpoints alive in one process. That was
built by talking to the endpoints directly, in a provider class of our own that
bypasses LiteLLM -- and with it, everything LiteLLM provides.
Impact
Calls routed to a knn endpoint lost native streaming, retries and tool-calling,
because the bespoke client implemented none of them. The behaviour of a model
therefore depended on whether routing happened to select it, and the difference
was invisible from config. A second execution path also meant every provider
fix had to be applied twice or it silently applied to one of them.
Proposal
Build one LiteLLM provider per endpoint instead. Several endpoints coexist
because the key and base travel per call rather than per process, so the
requirement that motivated the bespoke client is met without a second execution
path. knn becomes a decision layer again; execution is LiteLLM everywhere.
Summary
The knn routing backend needs several endpoints alive in one process. That was
built by talking to the endpoints directly, in a provider class of our own that
bypasses LiteLLM -- and with it, everything LiteLLM provides.
Impact
Calls routed to a knn endpoint lost native streaming, retries and tool-calling,
because the bespoke client implemented none of them. The behaviour of a model
therefore depended on whether routing happened to select it, and the difference
was invisible from config. A second execution path also meant every provider
fix had to be applied twice or it silently applied to one of them.
Proposal
Build one LiteLLM provider per endpoint instead. Several endpoints coexist
because the key and base travel per call rather than per process, so the
requirement that motivated the bespoke client is met without a second execution
path. knn becomes a decision layer again; execution is LiteLLM everywhere.