v1.70.0-nightly
·
33 commits
to 8a880ccd2f07bd696a9ebc03abae83d3287b08b1
since this release
What's Changed
- Capture Gemini reasoning tokens usage in streaming mode by @Dima-Mediator in #10789
- Anthropic unified web search + tool cost tracking support by @krrishdholakia in #10846
- avoid error
AttributeError: 'Message' object has no attribute 'audio'
by @igtm in #10814 - fix: bedrock-sonnet-caching-cost by @shibaboy in #10752
- Fix/issue 10113 embeddings use non default tokenizer by @camfarineau in #10629
- Add
models import
command by @msabramo in #10581 - Add documentation on in-memory and disk caching by @user202729 in #10522
- Bedrock pixtral fix by @ajac-zero in #10439
- added fix to resolve delete API by @jugaldb in #10845
- build(deps): bump undici from 6.21.1 to 6.21.3 in /docs/my-website by @dependabot in #10855
- [Refactor] Make Pagerduty a free feature by @ishaan-jaff in #10857
- [Refactor] Move enterprise_routes within litellm_enterprise by @ishaan-jaff in #10860
- refactor: make send_management_endpoint_alert a free feature by @ishaan-jaff in #10863
- [Feat - Guardrails] Expose /apply_guardrail endpoint for directly calling guardrail by @ishaan-jaff in #10867
- [CI/CD] Test: fix mapped tests by @ishaan-jaff in #10870
- [Feat] UI: Add support for Deleting Guardrail on UI by @ishaan-jaff in #10869
- [UI] Allow adding Bedrock, Presidio, Lakera, AIM guardrails on UI by @ishaan-jaff in #10874
- fix(proxy/_types.py): add missing comma for
/v2/rerank
by @krrishdholakia in #10861 - fix(streaming_handler.py): patch vertex ai meta llama streaming error by @krrishdholakia in #10878
- build(dependencies)📦: Add numpydoc to dependencies in pyproject.toml by @ericmjl in #10736
- [Feat]: Configure LiteLLM to Parse User Headers from Open Web UI by @damgem in #9802
- Feat/support anonymize in aim guardrail by @hxdror in #10757
- Gemini Multimodal Live API support by @krrishdholakia in #10841
- Feat log retention cleanup by @jugaldb in #10872
- Model filter added on logs tab. (#10877) by @ishaan-jaff in #10891
- fix: fix novita ai docs by @jasonhp in #10882
- [Feat] Add Tracing for guardrails in StandardLoggingPayload, Langfuse by @ishaan-jaff in #10890
- [UI] - Add Guardrail Tracing to LiteLLM SpendLogs by @ishaan-jaff in #10893
- Fix example commands in reliability.md by @wwwillchen in #10894
- Feat: add guardrail tracing to OTEL, Arize phoenix by @ishaan-jaff in #10896
- Key hash filter added by @NANDINI-star in #10903
- [Feat] UI: Allow clicking into a guardrail by @ishaan-jaff in #10904
- [Feat] Add Lakera v2 Guardrail Support by @ishaan-jaff in #10880
- Key alias filter on logs by @NANDINI-star in #10905
- [Feat] handle multi-pod deployment for SpendLogs Retention by @jugaldb in #10895
- [UI] Fixes for using Lakera Guardrail on LiteLLM UI by @ishaan-jaff in #10906
- add constant to config settings.md by @jugaldb in #10908
- Remove extra test file by @jugaldb in #10889
- [UI] Allow editing guardrails by @ishaan-jaff in #10907
New Contributors
- @Dima-Mediator made their first contribution in #10789
- @igtm made their first contribution in #10814
- @shibaboy made their first contribution in #10752
- @camfarineau made their first contribution in #10629
- @ajac-zero made their first contribution in #10439
- @damgem made their first contribution in #9802
- @hxdror made their first contribution in #10757
- @wwwillchen made their first contribution in #10894
Full Changelog: v1.69.3-nightly...v1.70.0-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.0-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.0-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 190.0 | 206.64782881083894 | 6.289821899591554 | 0.0 | 1882 | 0 | 171.67403099995227 | 1154.6766310000294 |
Aggregated | Passed ✅ | 190.0 | 206.64782881083894 | 6.289821899591554 | 0.0 | 1882 | 0 | 171.67403099995227 | 1154.6766310000294 |