v1.40.4

What's Changed

• feat: clarify slack alerting message by @nibalizer in #4023
• [Admin UI] Analytics - fix div by 0 error on /model/metrics by @ishaan-jaff in #4021
• Use DEBUG level for curl command logging by @grav in #2980
• feat(create_user_button.tsx): allow admin to invite user to proxy via user-email/pwd invite-links by @krrishdholakia in #4028
• [FIX] Proxy redirect to PROXY_BASE_URL/ui after logging in by @ishaan-jaff in #4027
• [Feat] Audit Logs for Key, User, ProxyModel CRUD operations by @ishaan-jaff in #4030

New Contributors

• @nibalizer made their first contribution in #4023

Full Changelog: v1.40.3...v1.40.4

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	74	89.43947919222931	6.450062450815326	0.0	1930	0	64.37952199996744	1143.0389689999743
Aggregated	Passed ✅	74	89.43947919222931	6.450062450815326	0.0	1930	0	64.37952199996744	1143.0389689999743

v1.40.3-stable

What's Changed

• feat: clarify slack alerting message by @nibalizer in #4023

New Contributors

• @nibalizer made their first contribution in #4023

Full Changelog: v1.40.3...v1.40.3-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.3-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	140.0	166.81647102860174	6.3100225495221665	0.0	1888	0	109.54055500008053	2288.330084999984
Aggregated	Passed ✅	140.0	166.81647102860174	6.3100225495221665	0.0	1888	0	109.54055500008053	2288.330084999984

v1.40.3

What's Changed

• [FIX] Proxy - only log cache credentials in debug mode by @ishaan-jaff in #4024

Full Changelog: v1.40.2...v1.40.3

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.3

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	130.0	168.35103872813087	6.385058663866248	0.0	1909	0	109.50845100001061	8353.559378
Aggregated	Passed ✅	130.0	168.35103872813087	6.385058663866248	0.0	1909	0	109.50845100001061	8353.559378

v1.40.2-stable

Full Changelog: v1.40.1.dev4...v1.40.2-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.2-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	100.0	135.25610868094057	6.399866394760457	0.0	1915	0	82.61822200000779	2219.8920350000435
Aggregated	Passed ✅	100.0	135.25610868094057	6.399866394760457	0.0	1915	0	82.61822200000779	2219.8920350000435

v1.40.2

What's Changed

• Add simple OpenTelemetry tracer by @yujonglee in #3974
• [FEAT] Add native OTEL logging to LiteLLM by @ishaan-jaff in #4010
• [Docs] Use OTEL logging on LiteLLM Proxy by @ishaan-jaff in #4011
• fix(bedrock): raise nested error response by @pharindoko in #3989
• [Feat] Admin UI - Add, Edit all LiteLLM callbacks on UI by @ishaan-jaff in #4014
• feat(assistants/main.py): add assistants api streaming support by @krrishdholakia in #4012
• feat(utils.py): Support stream_options param across all providers by @krrishdholakia in #4015
• fix(utils.py): fix cost calculation for openai-compatible streaming object by @krrishdholakia in #4009
• [Fix] Admin UI Internal Users by @ishaan-jaff in #4016

Full Changelog: v1.40.1...v1.40.2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	72	86.0339053382131	6.392727588765549	0.0	1913	0	61.2748209999836	896.4834699999642
Aggregated	Passed ✅	72	86.0339053382131	6.392727588765549	0.0	1913	0	61.2748209999836	896.4834699999642

v1.40.1.dev4

What's Changed

• Add simple OpenTelemetry tracer by @yujonglee in #3974
• [FEAT] Add native OTEL logging to LiteLLM by @ishaan-jaff in #4010
• [Docs] Use OTEL logging on LiteLLM Proxy by @ishaan-jaff in #4011
• fix(bedrock): raise nested error response by @pharindoko in #3989
• [Feat] Admin UI - Add, Edit all LiteLLM callbacks on UI by @ishaan-jaff in #4014
• feat(assistants/main.py): add assistants api streaming support by @krrishdholakia in #4012
• feat(utils.py): Support stream_options param across all providers by @krrishdholakia in #4015
• fix(utils.py): fix cost calculation for openai-compatible streaming object by @krrishdholakia in #4009
• [Fix] Admin UI Internal Users by @ishaan-jaff in #4016

Full Changelog: v1.40.1...v1.40.1.dev4

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1.dev4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	110.0	130.49834083376624	6.432223242582805	0.0	1925	0	92.76206099997353	2155.1117690000297
Aggregated	Passed ✅	110.0	130.49834083376624	6.432223242582805	0.0	1925	0	92.76206099997353	2155.1117690000297

v1.40.1.dev2

What's Changed

• Add simple OpenTelemetry tracer by @yujonglee in #3974

Full Changelog: v1.40.1...v1.40.1.dev2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1.dev2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	140.0	177.0382996107586	6.334561220731733	0.0	1896	0	114.13910500004931	1784.0317350000134
Aggregated	Passed ✅	140.0	177.0382996107586	6.334561220731733	0.0	1896	0	114.13910500004931	1784.0317350000134

v1.40.1

What's Changed

• [Feat] return num_retries and max_retries in exceptions by @ishaan-jaff in #3962
• [FEAT]- set custom AllowedFailsPolicy on litellm.Router by @ishaan-jaff in #3963
• feat(proxy_server.py): log litellm api version to langfuse by @krrishdholakia in #3969
• feat - add batches api to docs by @ishaan-jaff in #3973
• [Fix] Traceloop / OTEL logging fixes + easier docs by @ishaan-jaff in #3970
• add dall-e 3 required comment by @rhtenhove in #3984
• [Feat] Log Raw Request from LiteLLM on Langfuse - when "log_raw_request": true by @ishaan-jaff in #3983
• [Feat] Admin UI - Multi-Select Tags, Viewing spend by tags by @ishaan-jaff in #3994
• [Feat] UI - Filter model latency by API Key Alias by @ishaan-jaff in #3995
• feat(assistants/main.py): Azure Assistants API support by @krrishdholakia in #3996
• [Admin UI] Filter Model Latency by Customer, API Key by @ishaan-jaff in #3997
• fix(router.py): use litellm.request_timeout as default for router clients by @krrishdholakia in #3992
• [Doc] - Spend tracking with litellm by @ishaan-jaff in #3991

New Contributors

• @rhtenhove made their first contribution in #3984

Full Changelog: v1.40.0...v1.40.1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	120.0	139.78250550967104	6.395300383667639	0.0	1913	0	95.28932899991105	1526.2213239999483
Aggregated	Passed ✅	120.0	139.78250550967104	6.395300383667639	0.0	1913	0	95.28932899991105	1526.2213239999483

v1.40.0

What's Changed

• fix: fix streaming with httpx client by @krrishdholakia in #3944
• feat(scheduler.py): add request prioritization scheduler by @krrishdholakia in #3954
• [FEAT] Perf improvements - litellm.completion / litellm.acompletion - Cache OpenAI client by @ishaan-jaff in #3956
• fix(http_handler.py): support verify_ssl=False when using httpx client by @krrishdholakia in #3959
• Litellm docker compose start by @krrishdholakia in #3961

Full Changelog: v1.39.6...v1.40.0

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.0

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	120.0	133.63252197830545	6.467733658247951	0.0	1936	0	94.77090299998281	801.180971000008
Aggregated	Passed ✅	120.0	133.63252197830545	6.467733658247951	0.0	1936	0	94.77090299998281	801.180971000008

v1.39.6

We're launching team member invites (No SSO Required) on v1.39.6 🔥 Invite team member to view LLM Usage, Spend per service https://docs.litellm.ai/docs/proxy/ui

👍 [Fix] Cache Vertex AI clients - Major Perf improvement for VertexAI models

✨ Feat - Send new users invite emails on creation (using 'send_invite_email' on /user/new)

💻 UI - allow users to sign in with with email/password

🔓 [UI] Admin UI Invite Links for non SSO

✨ PR - [FEAT] Perf improvements - litellm.completion / litellm.acompletion - Cache OpenAI client

What's Changed

• Fix warnings from pydantic by @lj-wego in #3670
• Update pydantic version in CI requirements.txt by @lj-wego in #3938
• Allow admin to give invite links to others by @krrishdholakia in #3875
• Update model config definition to use v2 style by @lj-wego in #3943
• Add OIDC + unit test for bedrock httpx by @Manouchehri in #3688
• (fix) Update Mistral model list and prices by @alexpeattie in #3945
• feat - send_invite_email on /user/new by @ishaan-jaff in #3942
• [UI] Admin UI Invite Links for non SSO users by @ishaan-jaff in #3950
• [Feat] Admin UI - invite users to view spend by @ishaan-jaff in #3952
• UI - allow users to sign in with with email/password by @ishaan-jaff in #3953
• feat(proxy_server.py): add assistants api endpoints to proxy server by @krrishdholakia in #3936
• [Fix] Cache Vertex AI clients - Perf improvement by @ishaan-jaff in #3935
• fix(bedrock): convert botocore credentials when role is assumed by @pharindoko in #3939

New Contributors

• @lj-wego made their first contribution in #3670
• @alexpeattie made their first contribution in #3945
• @pharindoko made their first contribution in #3939

Full Changelog: v1.39.5...v1.39.6

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.39.6

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	78	90.37559010674164	6.5521693586672445	0.0	1958	0	65.34477100001368	961.3953589999937
Aggregated	Passed ✅	78	90.37559010674164	6.5521693586672445	0.0	1958	0	65.34477100001368	961.3953589999937