v1.24.3

What's Changed

[FEAT] Track LLM spend per team by @ishaan-jaff in #1996

[FEAT] Proxy - use custom API endpoints as callbacks by @ishaan-jaff in #1986

docs: https://docs.litellm.ai/docs/proxy/logging#custom-callback-apis-async

[FEAT] Track team_id in langfuse tags, callback metadata by @ishaan-jaff in #1993

[Fix] Admin UI Chat Tab by @ishaan-jaff in #1997

• feat(proxy_server.py): support key-level permissions for pii masking by @krrishdholakia in #1991

Full Changelog: v1.24.1...v1.24.3

v1.24.1

What's Changed

• feat(proxy_server.py): /team/new and /team/info endpoints for team management by @krrishdholakia in #1979
• Implements aioboto3 for sagemaker by @krrishdholakia in #1952
• Update utils.py: switched from pkg_resources to importlib.resources for filename stored in the os.environ(TIKTOKEN_CACHE_DIR) by @sorokine in #1947
• fix(redis.py): fix instantiating redis client from url by @krrishdholakia in #1989
• Add support for Vertex AI custom models deployed on private endpoint by @RenaLu in #1916

New Contributors

• @sorokine made their first contribution in #1947
• @RenaLu made their first contribution in #1916

Full Changelog: v1.23.16...v1.24.1

v1.23.16

👉 support for /moderations endpoint

💥 New models text-moderation-stable, text-moderation-007

⚡️ Admin UI - New Support for Generic SSO Providers Try here: https://docs.litellm.ai/docs/proxy/ui#setup-ssoauth-for-ui

🛠️ Fixes for using models access groups https://docs.litellm.ai/docs/proxy/virtual_keys#grant-access-to-new-model

What's Changed

• [FEAT] Proxy Add /moderations endpoint by @ishaan-jaff in #1974
• fix(lowest_latency.py): fix latency calc (lower better) by @stephenleo in #1935
• [FEAT] Support Generic SSO provider on Admin UI by @ishaan-jaff in #1978
• [FIX] Model Access Groups by @ishaan-jaff in #1980

New Contributors

• @stephenleo made their first contribution in #1935

Full Changelog: v1.23.15...v1.23.16

v1.23.15

What's Changed

• feat(presidio_pii_masking.py): enable output parsing for pii masking by @krrishdholakia in #1970
• fix(proxy_server.py): fix proxy server team id bug by @krrishdholakia in #1971

Full Changelog: v1.23.14...v1.23.15

v1.23.14

Allow your team to test 100+ LLMs, control LLM access by User ID 👉 Try out our new Chat playground

⚡️ Fixes for s3 Cache - when proxy disconnects

⚒️ fix for logging tags to Langfuse

📖 Updated docs on using litellm GHCR docker container

What's Changed

• [FIX] s3 cache proxy - fix notImplemented error by @ishaan-jaff in #1966
• feat(utils.py): enable post call rules for streaming by @krrishdholakia in #1955
• [Feat] Proxy DynamoDB - set arn number on dynamoDB /key/gen by @ishaan-jaff in #1589
• [FEAT] Admin UI - show users playground to test models by @ishaan-jaff in #1968

Full Changelog: v1.23.12...v1.23.14

v1.23.12

🚀 LiteLLM Proxy Server v1.23.11 - Allow your team to create keys for Azure, OpenAI, Bedrock, Sagemaker, Gemini and call 100+ LLMs

What's Changed

• support langfuse tags feature by @deenaawny-github-account in #1943
• build(deps): bump jinja2 from 3.1.2 to 3.1.3 by @dependabot in #1944
• [FEAT] ADMIN UI - Show Model Info by @ishaan-jaff in #1949
• Litellm proxy routing fix by @krrishdholakia in #1946
• [FEAT] UI show user available models when making a key by @ishaan-jaff in #1950

New Contributors

• @deenaawny-github-account made their first contribution in #1943

Full Changelog: v1.23.10...v1.23.12

v1.23.10

What's Changed

• Enable viewing key alias instead of hashed tokens by @krrishdholakia in #1926
• [FEAT] Proxy - set team specific models by @ishaan-jaff in #1933
• feat(proxy_server.py): support for pii masking with microsoft presidio by @krrishdholakia in #1931

Full Changelog: v1.23.9...v1.23.10

v1.23.9

What's Changed

• fix(usage.tsx): do cost breakdown by model by @krrishdholakia in #1928
• [FEAT] Proxy set ssl_certificates on proxy by @ishaan-jaff in #1929

Full Changelog: v1.23.8...v1.23.9

v1.23.8

Full Changelog: v1.23.7...v1.23.8

v1.23.7

• [FEAT] ui - view total proxy spend / budget by @ishaan-jaff in #1915
• [FEAT] Bedrock set timeouts on litellm.completion by @ishaan-jaff in #1919
• [FEAT] Use LlamaIndex with Proxy - Support azure deployments for /embeddings - by @ishaan-jaff in #1921
• [FIX] Verbose Logger - don't double print CURL command by @ishaan-jaff in #1924
• [FEAT] Set timeout for bedrock on proxy by @ishaan-jaff in #1922
• feat(proxy_server.py): show admin global spend as time series data by @krrishdholakia in #1920

1. Bedrock Set Timeouts

Usage - litellm.completion

response = litellm.completion(
    model="bedrock/anthropic.claude-instant-v1",
    timeout=0.01,
    messages=[{"role": "user", "content": "hello, write a 20 pg essay"}],
)

Usage on Proxy config.yaml

model_list:
  - model_name: BEDROCK_GROUP
    litellm_params:
      model: bedrock/cohere.command-text-v14
      timeout: 0.0001

2 View total proxy spend / budget

3. Use LlamaIndex with Proxy - Support azure deployments for /embeddings

Send Embedding requests like this

http://0.0.0.0:4000/openai/deployments/azure-embedding-model/embeddings?api-version=2023-07-01-preview

This allow users to use llama index AzureOpenAI with LiteLLM

Use LlamaIndex with LiteLLM Proxy

import os, dotenv

from dotenv import load_dotenv

load_dotenv()

from llama_index.llms import AzureOpenAI
from llama_index.embeddings import AzureOpenAIEmbedding
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext

llm = AzureOpenAI(
    engine="azure-gpt-3.5",
    temperature=0.0,
    azure_endpoint="http://0.0.0.0:4000",
    api_key="sk-1234",
    api_version="2023-07-01-preview",
)

embed_model = AzureOpenAIEmbedding(
    deployment_name="azure-embedding-model",
    azure_endpoint="http://0.0.0.0:4000",
    api_key="sk-1234",
    api_version="2023-07-01-preview",
)


# response = llm.complete("The sky is a beautiful blue and")
# print(response)

documents = SimpleDirectoryReader("llama_index_data").load_data()
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
index = VectorStoreIndex.from_documents(documents, service_context=service_context)

query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)

Full Changelog: v1.23.5...v1.23.7