-
Notifications
You must be signed in to change notification settings - Fork 1
Description
Summary
The Braintrust LiteLLM wrapper (wrap_litellm() and patch_litellm()) instruments completion, acompletion, responses, aresponses, embedding, and moderation. LiteLLM also exposes several other documented API functions that are not instrumented at all:
litellm.image_generation()/litellm.aimage_generation()— image generation (DALL-E, etc.)litellm.transcription()/litellm.atranscription()— audio transcription (Whisper, etc.)litellm.speech()/litellm.aspeech()— text-to-speechlitellm.rerank()/litellm.arerank()— reranking for RAG pipelines (Cohere, etc.)
Additionally, litellm.aembedding() (async embedding) is not wrapped — the EmbeddingWrapper class only accepts a sync function.
What is missing
No tracing spans are created when users call the above functions through either wrap_litellm() or patch_litellm(). The LiteLLMWrapper class in py/src/braintrust/wrappers/litellm.py (lines 484-518) only exposes: completion, acompletion, responses, aresponses, embedding, moderation. The patch_litellm() function (lines 630-667) only patches: completion, acompletion, responses, aresponses.
Braintrust docs status
not_found — The Braintrust LiteLLM integration docs at https://www.braintrust.dev/docs do not mention image generation, transcription, speech, or rerank support.
Upstream sources
- LiteLLM image generation: https://docs.litellm.ai/docs/image_generation
- LiteLLM audio transcription: https://docs.litellm.ai/docs/audio_transcription
- LiteLLM text-to-speech: https://docs.litellm.ai/docs/text_to_speech
- LiteLLM rerank: https://docs.litellm.ai/docs/rerank
- LiteLLM async embedding: https://docs.litellm.ai/docs/embedding/supported_embedding
Local repo files inspected
py/src/braintrust/wrappers/litellm.py— full wrapper implementation (668 lines)py/src/braintrust/wrappers/test_litellm.py— tests (no tests for the missing functions)py/noxfile.py— LiteLLM test sessions
Relationship to existing issues
- Issue LiteLLM:
patch_litellm()does not patchembeddingormoderation;aembeddingmissing entirely #115 coverspatch_litellm()not patching the existingembeddingandmoderationwrappers. This issue covers API functions that have no wrapper implementation at all. - Issue OpenAI: Images API (
client.images.generate()) not instrumented #124 covers the OpenAI Images API specifically. This issue covers the LiteLLM multi-provider equivalents.