Releases: BerriAI/litellm
v1.40.4
What's Changed
- feat: clarify slack alerting message by @nibalizer in #4023
- [Admin UI] Analytics - fix div by 0 error on /model/metrics by @ishaan-jaff in #4021
- Use DEBUG level for curl command logging by @grav in #2980
- feat(create_user_button.tsx): allow admin to invite user to proxy via user-email/pwd invite-links by @krrishdholakia in #4028
- [FIX] Proxy redirect to
PROXY_BASE_URL/ui
after logging in by @ishaan-jaff in #4027 - [Feat] Audit Logs for Key, User, ProxyModel CRUD operations by @ishaan-jaff in #4030
New Contributors
- @nibalizer made their first contribution in #4023
Full Changelog: v1.40.3...v1.40.4
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.4
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.4
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 74 | 89.43947919222931 | 6.450062450815326 | 0.0 | 1930 | 0 | 64.37952199996744 | 1143.0389689999743 |
Aggregated | Passed ✅ | 74 | 89.43947919222931 | 6.450062450815326 | 0.0 | 1930 | 0 | 64.37952199996744 | 1143.0389689999743 |
v1.40.3-stable
What's Changed
- feat: clarify slack alerting message by @nibalizer in #4023
New Contributors
- @nibalizer made their first contribution in #4023
Full Changelog: v1.40.3...v1.40.3-stable
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.3-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 140.0 | 166.81647102860174 | 6.3100225495221665 | 0.0 | 1888 | 0 | 109.54055500008053 | 2288.330084999984 |
Aggregated | Passed ✅ | 140.0 | 166.81647102860174 | 6.3100225495221665 | 0.0 | 1888 | 0 | 109.54055500008053 | 2288.330084999984 |
v1.40.3
What's Changed
- [FIX] Proxy - only log cache credentials in debug mode by @ishaan-jaff in #4024
Full Changelog: v1.40.2...v1.40.3
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.3
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 130.0 | 168.35103872813087 | 6.385058663866248 | 0.0 | 1909 | 0 | 109.50845100001061 | 8353.559378 |
Aggregated | Passed ✅ | 130.0 | 168.35103872813087 | 6.385058663866248 | 0.0 | 1909 | 0 | 109.50845100001061 | 8353.559378 |
v1.40.2-stable
Full Changelog: v1.40.1.dev4...v1.40.2-stable
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.2-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 100.0 | 135.25610868094057 | 6.399866394760457 | 0.0 | 1915 | 0 | 82.61822200000779 | 2219.8920350000435 |
Aggregated | Passed ✅ | 100.0 | 135.25610868094057 | 6.399866394760457 | 0.0 | 1915 | 0 | 82.61822200000779 | 2219.8920350000435 |
v1.40.2
What's Changed
- Add simple OpenTelemetry tracer by @yujonglee in #3974
- [FEAT] Add native OTEL logging to LiteLLM by @ishaan-jaff in #4010
- [Docs] Use OTEL logging on LiteLLM Proxy by @ishaan-jaff in #4011
- fix(bedrock): raise nested error response by @pharindoko in #3989
- [Feat] Admin UI - Add, Edit all LiteLLM callbacks on UI by @ishaan-jaff in #4014
- feat(assistants/main.py): add assistants api streaming support by @krrishdholakia in #4012
- feat(utils.py): Support
stream_options
param across all providers by @krrishdholakia in #4015 - fix(utils.py): fix cost calculation for openai-compatible streaming object by @krrishdholakia in #4009
- [Fix] Admin UI Internal Users by @ishaan-jaff in #4016
Full Changelog: v1.40.1...v1.40.2
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.2
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 72 | 86.0339053382131 | 6.392727588765549 | 0.0 | 1913 | 0 | 61.2748209999836 | 896.4834699999642 |
Aggregated | Passed ✅ | 72 | 86.0339053382131 | 6.392727588765549 | 0.0 | 1913 | 0 | 61.2748209999836 | 896.4834699999642 |
v1.40.1.dev4
What's Changed
- Add simple OpenTelemetry tracer by @yujonglee in #3974
- [FEAT] Add native OTEL logging to LiteLLM by @ishaan-jaff in #4010
- [Docs] Use OTEL logging on LiteLLM Proxy by @ishaan-jaff in #4011
- fix(bedrock): raise nested error response by @pharindoko in #3989
- [Feat] Admin UI - Add, Edit all LiteLLM callbacks on UI by @ishaan-jaff in #4014
- feat(assistants/main.py): add assistants api streaming support by @krrishdholakia in #4012
- feat(utils.py): Support
stream_options
param across all providers by @krrishdholakia in #4015 - fix(utils.py): fix cost calculation for openai-compatible streaming object by @krrishdholakia in #4009
- [Fix] Admin UI Internal Users by @ishaan-jaff in #4016
Full Changelog: v1.40.1...v1.40.1.dev4
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1.dev4
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 110.0 | 130.49834083376624 | 6.432223242582805 | 0.0 | 1925 | 0 | 92.76206099997353 | 2155.1117690000297 |
Aggregated | Passed ✅ | 110.0 | 130.49834083376624 | 6.432223242582805 | 0.0 | 1925 | 0 | 92.76206099997353 | 2155.1117690000297 |
v1.40.1.dev2
What's Changed
- Add simple OpenTelemetry tracer by @yujonglee in #3974
Full Changelog: v1.40.1...v1.40.1.dev2
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1.dev2
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 140.0 | 177.0382996107586 | 6.334561220731733 | 0.0 | 1896 | 0 | 114.13910500004931 | 1784.0317350000134 |
Aggregated | Passed ✅ | 140.0 | 177.0382996107586 | 6.334561220731733 | 0.0 | 1896 | 0 | 114.13910500004931 | 1784.0317350000134 |
v1.40.1
What's Changed
- [Feat] return
num_retries
andmax_retries
in exceptions by @ishaan-jaff in #3962 - [FEAT]- set custom AllowedFailsPolicy on litellm.Router by @ishaan-jaff in #3963
- feat(proxy_server.py): log litellm api version to langfuse by @krrishdholakia in #3969
- feat - add batches api to docs by @ishaan-jaff in #3973
- [Fix] Traceloop / OTEL logging fixes + easier docs by @ishaan-jaff in #3970
- add dall-e 3 required comment by @rhtenhove in #3984
- [Feat] Log Raw Request from LiteLLM on Langfuse - when
"log_raw_request": true
by @ishaan-jaff in #3983 - [Feat] Admin UI - Multi-Select Tags, Viewing spend by tags by @ishaan-jaff in #3994
- [Feat] UI - Filter model latency by API Key Alias by @ishaan-jaff in #3995
- feat(assistants/main.py): Azure Assistants API support by @krrishdholakia in #3996
- [Admin UI] Filter Model Latency by Customer, API Key by @ishaan-jaff in #3997
- fix(router.py): use
litellm.request_timeout
as default for router clients by @krrishdholakia in #3992 - [Doc] - Spend tracking with litellm by @ishaan-jaff in #3991
New Contributors
- @rhtenhove made their first contribution in #3984
Full Changelog: v1.40.0...v1.40.1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 120.0 | 139.78250550967104 | 6.395300383667639 | 0.0 | 1913 | 0 | 95.28932899991105 | 1526.2213239999483 |
Aggregated | Passed ✅ | 120.0 | 139.78250550967104 | 6.395300383667639 | 0.0 | 1913 | 0 | 95.28932899991105 | 1526.2213239999483 |
v1.40.0
What's Changed
- fix: fix streaming with httpx client by @krrishdholakia in #3944
- feat(scheduler.py): add request prioritization scheduler by @krrishdholakia in #3954
- [FEAT] Perf improvements - litellm.completion / litellm.acompletion - Cache OpenAI client by @ishaan-jaff in #3956
- fix(http_handler.py): support verify_ssl=False when using httpx client by @krrishdholakia in #3959
- Litellm docker compose start by @krrishdholakia in #3961
Full Changelog: v1.39.6...v1.40.0
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.0
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 120.0 | 133.63252197830545 | 6.467733658247951 | 0.0 | 1936 | 0 | 94.77090299998281 | 801.180971000008 |
Aggregated | Passed ✅ | 120.0 | 133.63252197830545 | 6.467733658247951 | 0.0 | 1936 | 0 | 94.77090299998281 | 801.180971000008 |
v1.39.6
We're launching team member invites (No SSO Required) on v1.39.6 🔥 Invite team member to view LLM Usage, Spend per service https://docs.litellm.ai/docs/proxy/ui
👍 [Fix] Cache Vertex AI clients - Major Perf improvement for VertexAI models
✨ Feat - Send new users invite emails on creation (using 'send_invite_email' on /user/new)
💻 UI - allow users to sign in with with email/password
🔓 [UI] Admin UI Invite Links for non SSO
✨ PR - [FEAT] Perf improvements - litellm.completion / litellm.acompletion - Cache OpenAI client
What's Changed
- Fix warnings from pydantic by @lj-wego in #3670
- Update pydantic version in CI requirements.txt by @lj-wego in #3938
- Allow admin to give invite links to others by @krrishdholakia in #3875
- Update model config definition to use v2 style by @lj-wego in #3943
- Add OIDC + unit test for bedrock httpx by @Manouchehri in #3688
- (fix) Update Mistral model list and prices by @alexpeattie in #3945
- feat -
send_invite_email
on /user/new by @ishaan-jaff in #3942 - [UI] Admin UI Invite Links for non SSO users by @ishaan-jaff in #3950
- [Feat] Admin UI - invite users to view spend by @ishaan-jaff in #3952
- UI - allow users to sign in with with email/password by @ishaan-jaff in #3953
- feat(proxy_server.py): add assistants api endpoints to proxy server by @krrishdholakia in #3936
- [Fix] Cache Vertex AI clients - Perf improvement by @ishaan-jaff in #3935
- fix(bedrock): convert botocore credentials when role is assumed by @pharindoko in #3939
New Contributors
- @lj-wego made their first contribution in #3670
- @alexpeattie made their first contribution in #3945
- @pharindoko made their first contribution in #3939
Full Changelog: v1.39.5...v1.39.6
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.39.6
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
---|---|---|---|---|---|---|---|---|---|
/chat/completions | Passed ✅ | 78 | 90.37559010674164 | 6.5521693586672445 | 0.0 | 1958 | 0 | 65.34477100001368 | 961.3953589999937 |
Aggregated | Passed ✅ | 78 | 90.37559010674164 | 6.5521693586672445 | 0.0 | 1958 | 0 | 65.34477100001368 | 961.3953589999937 |