-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Open
Labels
tech-debtTechnical DebtTechnical Debt
Description
🤔 What is the technical debt you think should be addressed?
When the inference providers were originally created, shared mixins like OpenAIMixin
and LiteLLMOpenAIMixin
did not exist. As a result, many providers implemented their own logic manually and inconsistently.
Now that these mixins are available and some providers have adopted them, we have a fragmented implementation across the codebase. This results in:
- Duplicated logic (e.g. for streaming, parameter handling, response formatting)
- Inconsistent behavior across providers
- Increased maintenance burden
- Higher likelihood of subtle bugs and divergent implementations
💡 What is the benefit of addressing this technical debt?
- Consistency: All inference providers follow the same behavior.
- Maintainability: Changes (e.g. API updates, bug fixes) can be made in one place.
- Reduced Duplication: Shared logic eliminates repeated code across providers.
- Scalability: Easier to onboard or implement new providers.
- Better Testing: Shared mixins can be tested centrally, increasing reliability.
Inference providers
provider | chat | completions | embeddings | status | notes |
---|---|---|---|---|---|
anthropic | yes | yes | yes | ||
azure openai | yes | yes | yes | ||
bedrock | yes | yes | no | #3410 | via OpenAIChatCompletionToLlamaStackMixin and OpenAICompletionToLlamaStackMixin, openai-compat |
cerebras | yes | yes | no | ||
databricks | yes | no | no | ||
fireworks | yes | yes | yes | ||
gemini | yes | yes | yes | ||
groq | yes | yes | yes | ||
llama | yes | yes | yes | ||
nvidia | yes | yes | yes | ||
ollama | yes | yes | yes | ||
openai | yes | yes | yes | ||
runpod | yes | no | no | TODO | vLLM based, uses openai, openai-compat |
sambanova | yes | yes | yes | ||
tgi | yes | yes | no | ||
hf::serverless | yes | yes | no | TODO | BROKEN: #3415 |
hf::endpoints | yes | yes | no | TODO | |
together | yes | yes | yes | ||
vertexai | yes | yes | no | ||
vllm | yes | yes | no | ||
watsonx | yes | yes | no | TODO | custom |
cdoern and leseb
Metadata
Metadata
Assignees
Labels
tech-debtTechnical DebtTechnical Debt