[Bug]: Fix docker image for multiple deployment Router

[krrishdholakia]

What happened?

Our openai-proxy docker image no longer supports multiple model deployments. This is what we're currently directing users too - https://docs.litellm.ai/docs/routing#handle-multiple-azure-deployments-via-openai-proxy-server.

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

Support for routing between multiple deployments added to the openai-proxy endpoint:

also added support for accepting the model list either via a config.yaml -

litellm/openai_proxy/utils.py

Line 26 in 751e9f0

def load_router_config(router: Optional[litellm.Router]):

or from the model_list param in the request body:

litellm/openai_proxy/main.py

Line 114 in e118ce5

if "model_list" in data:

Remaining Work:

• need to update dockerfile to accept the config.yaml as a build param (and not raise an error if it's not passed in)

[krrishdholakia]

cc: @alabrashJr @mc-marcocheng

[krrishdholakia]

This is live now.

How to deploy router

1. Clone repo

 git clone https://github.com/BerriAI/litellm

2. Create + Modify router_config.yaml (save your azure/openai/etc. deployment info)

cp ./router_config_template.yaml ./router_config.yaml

3. Build + Run docker image

docker build -t litellm-proxy . --build-arg CONFIG_FILE=./router_config.yaml 

docker run --name litellm-proxy -e PORT=8000 -p 8000:8000 litellm-proxy

Test

curl 'http://0.0.0.0:8000/router/completions' \
--header 'Content-Type: application/json' \
--data '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hey"}]
}'

[krrishdholakia]

docs updated as well.

@alabrashJr can you let me know if this solves your problem?