LiteLLM Example Configs

[justinh-rahb]

Caution

NONE OF THE BELOW IS RELEVANT ANYMORE, LITELLM HAS BEEN REMOVED FROM OPEN WEBUI

https://docs.openwebui.com/migration#migrating-from-internal-to-external-litellm

Just wanted to dump some configs for common endpoints for future reference.

[!NOTE]
Other than official OpenAI-endpoints, LiteLLM usually requires that an provider be specified before the model string in the Add a model field, example:

azure/gpt35turbo

Model Name can be whatever you want it to appear as in your list.
API Base URL typically only needs to be set for OpenAI-compatible APIs and Azure.

[!WARNING]

Gemini Endpoint:

ONLY Makersuite/AI Studio API keys are supported, VertexAI/GCP endpoints and authentication methods are NOT currently supported.

Anthropic

Model strings:

• claude-2
• claude-2.1
• claude-instant-1.2
• claude-3-sonnet-20240229
• claude-3-opus-20240229
• claude-3-haiku-20240307

[!NOTE]
Anthropic's API requires that max_tokens parameter be sent in the payload, the maximum accepted value is 4096

Claude 2.1

Claude 3 "Sonnet"

Claude 3 "Opus"

Groq

Model strings:

• mixtral-8x7b-32768
• llama2-70b-4096

Mixtral 8x7B

Llama2 70B

Google Gemini

Model strings:

• gemini-pro

Gemini Pro

Mistral

Model strings:

• open-mistral-7b (aka mistral-tiny-2312)
• open-mixtral-8x7b (aka mistral-small-2312)
• mistral-small-latest (aka mistral-small-2402)
• mistral-medium-latest (aka mistral-medium-2312)
• mistral-large-latest (aka mistral-large-2402)

Open Mixtral 8x7B (formerly mistral-small)

Mistral Medium

Mistral Large

Azure OpenAI

Model strings:

• gpt35turbo
• gpt4

GPT 3.5 Turbo

---

[!CAUTION]

OpenAI and Ollama Endpoints

Unless there is a specific reason why the Connections > Ollama or Connections > OpenAI methods are not working for your use-case, the following methods are NOT recommended to be used instead:

OpenAI

Model strings:

• gpt-3.5-turbo
• gpt-4
• gpt-4-turbo-preview
• gpt-4-vision-preview
• See more

GPT 4 Turbo

"OpenAI-compatible" endpoints

Ollama

Llama2

[HydeAndGeek]

ahhhh yes ty!

[PierrunoYT]

Hey it's not working

[PierrunoYT]

Sorry I'm just confused because Pinokio says it's up to date.

[krrishdholakia]

this is amazing @justinh-rahb - if you have any feedback for how we can improve our own docs for this, let me know - https://docs.litellm.ai/docs/

• Krrish, LiteLLM Maintainer

[justinh-rahb]

I get an error when I try to use claude-3-opus-20240229 & claude-3-sonnet-20240229:

AnthropicException - AnthropicException - {
"error": {
"type": "forbidden",
"message": "Request not allowed"
}
}

My guess is invalid API key.

[krrishdholakia]

@abdimussa87 if you run the proxy with --debug you should be able to see the request being sent to anthropic.

[abdimussa87]

It had to do with the region I was in. Now I'm getting this error:
AnthropicException - AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

[justinh-rahb]

It had to do with the region I was in. Now I'm getting this error: AnthropicException - AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

You're running an outdated version of Open WebUI container with an older version of LiteLLM prior to the release of Claude 3 API.

[abdimussa87]

Yup. That was it. Thank you.

[yelban]

No, in v0.1.115 (latest version) open-webui still cannot set LiteLLM to make any Claude 3 model work. Unless...

Upgrade LiteLLM to the latest version v1.34.12.

git clone https://github.com/open-webui/open-webui.git

cd open-webui

Replace litellm==1.30.7 with litellm==1.34.12 in the ./backend/requirements.txt file.

docker build -t ghcr.io/open-webui/open-webui:latest .

Then use the locally created docker image ghcr.io/open-webui/open-webui:latest.

You can use the Claude 3 models added by LiteLLM.

[justinh-rahb]

Look at this, all workin' and stuff:

[yelban]

May I ask if you are also using the Docker container to start open-webui?

When I used the official mirror: ghcr.io/open-webui/open-webui:latest, following the previous example, carefully checking and adding claude 3 litellm has never been successful until modifying the version of litellm in requirements.txt and using a self-built docker image to finally succeed.

[justinh-rahb]

May I ask if you are also using the Docker method to start open-webui?

Docker-installed main branch - v0.1.115, no modifications.

[yelban]

Delete the local open-webui images, After restarting using ghcr.io/open-webui/open-webui:main, I can confirm that the newly added claude 3 model can run correctly.

Docker-installed main branch - v0.1.115, no modifications It works perfectly.

Your response and reminder are greatly appreciated. Thank you. @justinh-rahb

[flefevre]

Thanks a lot for the elements you shared. I am facing a difficulty in setting the access of open webui to litellm.
I do not understand what i need to provide in the litellm-config.yaml file for the sk.
Should i connect to litellm to generate a specific key and put it in master_key?
Thanks for the explanoation.

openwebui | File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 372, in user_api_key_auth openwebui | assert api_key.startswith("sk-") # prevent token hashes from being used openwebui | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ openwebui | AssertionError openwebui | id='dccf746e-2efa-4248-923a-7bfddedea4e4' name='Iagen Innov' email='tot@email.com' role='admin' profile_image_url='/user.png' timestamp=1711742369 openwebui | INFO: 172.22.0.1:0 - "GET /litellm/api/model/info HTTP/1.1" 401 Unauthorized

[justinh-rahb]

Master key is irrelevant in the context of our internal LiteLLM proxy! You seem to be wanting to setup a separate external LiteLLM as it's own container. Use it as if it is an OpenAI endpoint, where Master key is your OPENAI_API_KEY you would use for the "OpenAI Connection", with base URL set to your LiteLLM container's address.

[flefevre]

Thanks for your answer.
You are right. I have installed in two separate docker compose files: 1) open webui and 2) litellm with redis.
We have had an exchange about it here with you and shared it here to help people.: #508 (reply in thread)

[zhixwang]

Hi, thanks for the support! But I'm not clear how to set it up on OpenWebUI. It there a step-by-step tutorial somewhere? :)

[ravidreams]

How to get the LiteLLM API key? What is the LiteLLM API base URL? Are we supposed to install LiteLLM before adding values here? How to install that and get it connected with Openwebui? WHat value to give for RPM?

[flefevre]

Dear all, If I do understand,open webui comes with its own litellm. It is built inside. Perhaps they could allow at term to reference an external litellm, a better practice. François Le sam. 25 mai 2024, 11:49, Ravishankar Ayyakkannu ***@***.***> a écrit :

…

How to get the LiteLLM API ket? What is the LiteLLM API base URL? Are we supposed to install LiteLLM before adding values here? How to install that and get it connected with Openwebui? WHat value to give for RPM? — Reply to this email directly, view it on GitHub <#1038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZRFFWWCPJ7N2RSYWBY6DZEBNBRAVCNFSM6AAAAABEF46NNOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TKNJUG42DS> . You are receiving this because you commented.Message ID: ***@***.***>

[JuanjoJG-WATA]

I recently updated OUI and I tried to add my Azure API Key and endpoint but with no results. The LiteLLM option there used to be is nowhere to be seen

[justinh-rahb]

Here's a guide to help you move your litellm setup out of the open-webui container and into its own separate container. This will allow you to use litellm independently and connect it to open-webui as needed.

1. Download your config.yaml file

Your config.yaml file can be downloaded from the Admin Settings window:

2. Run a standalone litellm container

To run litellm in its own container, use the following docker run command:

docker run -d \
  -p 4000:4000 \
  --name litellm \
  -v ./config.yaml:/app/config.yaml \
  -e LITELLM_MASTER_KEY=sk-12345 \
  --restart always \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000

Replace ./config.yaml with the path to the config.yaml file you downloaded. Set the LITELLM_MASTER_KEY environment variable to any API key value as long as it's not empty.

3. Connect litellm to open-webui

Once you have the litellm container running, you can connect it to the open-webui container. In the open-webui settings, add an "OpenAI" base URL under "Connections" with the value http://host.docker.internal:4000/v1 and set the API key to your LITELLM_MASTER_KEY value (e.g., sk-12345).

By following these steps, you'll have moved your litellm setup out of the open-webui container and into its own separate container. You can now use litellm independently and connect it to open-webui as needed.

[JuanjoJG-WATA]

Thank you for your answer Justin. Unfortunately, I'm not using Docker, is there a way to do so without it?

[justinh-rahb]

Unfortunately, no, not a straightforward one. You could consider our own alternative to litellm, Pipelines which admittedly is not at parity with litellm but we can iterate on it very rapidly. There is an Azure API example though it's possible you may need to help us debug it since afaik none of us devs have access to test it ourselves.

[krrishdholakia]

hey @JuanjoJG-WATA yes you can use litellm without a docker container - via the cli tool

1. Install litellm

pip install 'litellm[proxy]' 

2. Run with config

litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000

docs - https://docs.litellm.ai/docs/proxy/quick_start

[JuanjoJG-WATA]

I made it work after a few tweaks. @justinh-rahb @krrishdholakia @tjbck

First I installed litellm like Krish said in the comment above this one:
pip install 'litellm[proxy]'

Then I changed directory to open-webui and created a yaml file called config.yaml and I filled it with the first 6 lines of this example:
(https://docs.litellm.ai/docs/proxy/quick_start#create-a-config-for-litellm-proxy)

Then I ran the .yaml with:
litellm --config open-webui/config.yaml

In the OpenWebUI interface I went to Settings -> Connections -> Made a new OpenAI API with http://localhost:4000 and also tried with the proper IP, alfo followed by the port to make sure it worked remotely.

Also I made some changes to the .env file. To that I added the following:

I closed both the process of the yaml and openwebui and ran them again to commit the changes.

And voilá, it's working.

[flefevre]

Please have a look to the new version>0.2 Use environmental variable https://docs.openwebui.com/getting-started/env-configuration#litellm Le jeu. 6 juin 2024, 09:24, JuanjoJG-WATA ***@***.***> a écrit :

…

I made it work after a few tweaks. @justinh-rahb <https://github.com/justinh-rahb> @krrishdholakia <https://github.com/krrishdholakia> @tjbck <https://github.com/tjbck> First I installed litellm like Krish said in the comment above this one: pip install 'litellm[proxy]' Then I changed directory to open-webui and created a yaml file called *config.yaml* and I filled it with the first 6 lines of this example: ( https://docs.litellm.ai/docs/proxy/quick_start#create-a-config-for-litellm-proxy ) Then I ran the .yaml with: litellm --config open-webui/config.yaml In the OpenWebUI interface I went to Settings -> Connections -> Made a new OpenAI API with *http://localhost:4000 <http://localhost:4000>* and also tried with the proper IP, alfo followed by the port to make sure it worked remotely. I closed both the process of the yaml and openwebui and ran them again to commit the changes. And voilá, it's working. image.png (view on web) <https://github.com/open-webui/open-webui/assets/165764759/26f6b4dc-99d4-46b3-8dd8-b4c7ece1ff17> — Reply to this email directly, view it on GitHub <#1038 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZRFCPNLG2VOIEMDMFZBLZGAFEJAVCNFSM6AAAAABEF46NNOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMOBWGI2TS> . You are receiving this because you commented.Message ID: ***@***.***>

[JuanjoJG-WATA]

Yes, I forgot to include that in my answer.

[Fusseldieb]

I just realized that this is a minor step back. It shouldn't have been removed just yet.

I've had Anthropic and OpenAI added as LiteLLM models, and it worked great, for both image and text, however that's not working anymore.

I now have three options:

• Move to Pipelines, which don't support images yet, therefore loosing image capabilities of BOTH models
• Stay on the vanilla OpenAI connection, but loosing Anthropic
• Downgrade

Honestly, all three don't sound great haha

[justinh-rahb]

You've forgotten a 4th option:

• Keep using LiteLLM if that's working for you

The only thing we've "taken away" is the UI for configuring it, but good news! LiteLLM has their own.

[Fusseldieb]

Hmm... Didn't knew that.
I run open-webui inside a docker container, so I'm unsure where I would configure it.
Would you mind explaining? Might even benefit some other readers which have the same "issue", until the gap closes.

[justinh-rahb]

https://docs.openwebui.com/migration#migrating-from-internal-to-external-litellm

#1038 (reply in thread)
Was also posted 3 days ago, in this thread you are currently reading ☝️