Repository navigation
Replies: 1 comment
|
I feel pretty strongly that litellm is a bad idea given how often it is the vehicle for a vulnerability. Useful for lots of people. Not something we need in daft. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi, I'd like to propose an optional
litellmAI provider and check whether you'd be open to it before I send a PR. I know #6565 cited the litellm supply-chain incident, so here is the dependency side first:litellm>=1.83.7,<1.106.0, so the malicious 1.82.7 and 1.82.8 uploads can never resolve. The upper bound is a cap in the same style as the other extras (openai<2.39.0,google-genai<2.7.0).daft[litellm]extra, listed inalllike the other providers. Nothing changes in the core dependencies.litellmis only imported when a prompter or embedder is created or called on a worker. Building the provider only checks that it's installed (find_spec). A missing extra raises the usualProviderImportError("Pleasepip install 'daft[litellm]'...").uv.lockchange only addslitellmandfastuuid. Every other litellm dependency is already in the lock, and no existing pins move.Feature summary
A
litellmprovider that implementsprompt()(vialitellm.acompletion) andembed_text()(vialitellm.aembedding). Models are addressed with LiteLLM'sprovider/modelstrings. One provider can then reach Anthropic, AWS Bedrock, Vertex AI, Azure OpenAI, Mistral, Cohere, Ollama and others from Daft workers, without running a proxy. If you already run a LiteLLM Proxy, the existingopenaiprovider withbase_urlworks today. This provider is for calling those APIs directly.Example usage
The structure follows the
openaiprovider: aProvider, descriptor dataclasses, aload_litellmentry in the registry, and the same message handling for text, image/video URLs, numpy images, bytes anddaft.File.Corner cases
return_formatis passed asresponse_format. LiteLLM maps it to each backend's mechanism, and the result is checked withmodel_validate_json. An empty response raises a clearValueError.RateLimitErrorandServiceUnavailableErrorgo through the existingraise_retry_after_from_response, so aRetry-Afterheader becomesRetryAfterErrorand Daft's UDF retries apply.dimensions=, thenlitellm.get_model_info(...)["output_vector_size"], then one probe request. If all three fail, it raises an error asking fordimensions=.supports_overriding_dimensions=Falseis honored.batch_size, default 64) is one request. Unlike the OpenAI provider, it doesn't re-chunk by token count, because token limits differ across backends.Noneinputs become empty strings. A response whose count doesn't match the inputs raises an error.record_token_metricslike the other providers.api_key,api_base,api_version,custom_llm_provider,timeout,num_retries,extra_headers) are set on the provider and can be overridden per call. Any otherprompt()kwargs are passed through to LiteLLM.I have a branch ready with code, docs (
providers.md,prompt.md,embed.mdand the API reference) and mocked unit tests. A socket guard means the tests can't reach the network, and the existingopenaiprovider tests still pass. It's around 600 lines of source plus tests and docs, so I'm happy to split it (for example prompter first, embedder second) if that's easier to review. If you'd rather not take on this dependency, that's fine too.Disclosure: I work on developer relations at LiteLLM.
All reactions