The Feature
I’d like LiteLLM to support vector store retrieval for all model providers, not only OpenAI models via the file_search tool. Concretely:
Add a provider-agnostic “vector_search” tool that works with Anthropic, Google, Azure OpenAI, local/vLLM/Ollama, etc.
From your docs:
curl http://localhost:4000/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer $LITELLM_API_KEY"
-d '{
"model": "claude-3-5-sonnet",
"messages": [{"role": "user", "content": "What is litellm?"}],
"tools": [
{
"type": "file_search",
"vector_store_ids": ["T37J8R4WTM"]
}
]
Motivation, pitch
LiteLLM is a unified gateway for many providers, but RAG is currently vendor-locked to OpenAI because only the OpenAI “file_search” tool is supported. In practice, teams (like mine) use pgvector and want to switch models freely (cost, latency, on-prem, or policy reasons) without losing retrieval.
This feature would:
Remove vendor lock-in and let us reuse the same vector DB across all models.
Enable local and air-gapped deployments (vLLM/Ollama) to benefit from standardized retrieval.
Simplify app code: one LiteLLM API, same behavior everywhere.
Encourage best-fit model selection (price/perf) while keeping consistent RAG quality.
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
Twitter / LinkedIn details
No response
The Feature
I’d like LiteLLM to support vector store retrieval for all model providers, not only OpenAI models via the file_search tool. Concretely:
Add a provider-agnostic “vector_search” tool that works with Anthropic, Google, Azure OpenAI, local/vLLM/Ollama, etc.
From your docs:
curl http://localhost:4000/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer $LITELLM_API_KEY"
-d '{
"model": "claude-3-5-sonnet",
"messages": [{"role": "user", "content": "What is litellm?"}],
"tools": [
{
"type": "file_search",
"vector_store_ids": ["T37J8R4WTM"]
}
]
Motivation, pitch
LiteLLM is a unified gateway for many providers, but RAG is currently vendor-locked to OpenAI because only the OpenAI “file_search” tool is supported. In practice, teams (like mine) use pgvector and want to switch models freely (cost, latency, on-prem, or policy reasons) without losing retrieval.
This feature would:
Remove vendor lock-in and let us reuse the same vector DB across all models.
Enable local and air-gapped deployments (vLLM/Ollama) to benefit from standardized retrieval.
Simplify app code: one LiteLLM API, same behavior everywhere.
Encourage best-fit model selection (price/perf) while keeping consistent RAG quality.
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
Twitter / LinkedIn details
No response