replace openai base with proxy w/ translation

[krrishdholakia]

There’s a lot of different LLM deployment providers. How do I easily replace my OpenAI base with their url as a proxy? - petals-infra/chat.petals.dev#20, https://www.banana.dev/, etc.

```python
def translate_function(model, messages, max_tokens):
	prompt = " ".join(message["content"] for message in messages)
	max_new_tokens = max_tokens
	return {"model": model, "prompt": prompt, "max_new_tokens": max_new_tokens} 

openai.api_base = litellm.translate_api_call(custom_api_base, translate_function)
```

[krrishdholakia]

Tracking 20 repos that might find this helpful:

• Error communicating with OpenAI shroominic/codeinterpreter-api#82
• openai.error.APIConnectionError: Error communicating with OpenAI GAIR-NLP/factool#8
• How to use proxy to valid OenAI api eky danswer-ai/danswer#240
• Using openai.proxy on Anaconda Jupyter Notebook openai/openai-python#471
• https://github.com/bentoml/OpenLLM
• https://github.com/basetenlabs/truss
• https://github.com/pytorch/serve
• https://github.com/apot-group/ml-models-in-production
• https://github.com/premAI-io/prem-app
• https://github.com/InternLM/lmdeploy
• https://github.com/homanp/superagent
• https://github.com/tensorchord/openmodelz
• https://github.com/baileytec-labs/llama-on-lambda
• https://github.com/ludwig-ai/ludwig
• https://github.com/facebookresearch/llama-recipes/blob/main/docs/inference.md
• https://github.com/THUDM/SwissArmyTransformer

[ishaan-jaff]

updates:

• baseten client/url doesn't work - they have a bunch of auth errors. Reached out to their support for help

[ishaan-jaff]

How would this be possible ?
When you set a url for OpenAI base it expects a server

openai.api_base = "https://localhost:500"

The proposed interface there is a function :

openai.api_base = litellm.translate_api_call(custom_api_base, translate_function)

Open to suggestions on this

[ishaan-jaff]

Tried doing something like this:

def custom_function():
  print("custom function called")

openai.api_base = custom_function

but openai calls @app.route('/chat/completions', methods=["POST"])

[ishaan-jaff]

closing this