Allow Custom URL for OpenAI API

[mooangus]

It would be great if you allowed for a custom URL to be specified for OpenAI's API so we could use Azure OpenAI instance instead.

[jankieAI]

or our own locally hosted endpoint like https://github.com/oobabooga/text-generation-webui

[sabaimran]

Ah, we can consider that. What would be the reason behind wanting to do that? I'm assuming because you've already gotten a subscription and want to use credits there or some equivalent?

[mooangus]

Yes. That's correct. My organisation already uses Azure and I can use their end point at no personal cost. As noted by someone else, it would be great if it could just be a custom field to override the OpenAI default. That would allow any custom endpoint for the API.

[difhel]

+1 for waiting that. It would be great to set a custom endpoint in /config.

[aleung]

I checked the source code, Khoj uses langchain chat_models.openai.ChatOpenAI, which supports reading OpenAI API base url from environment OPENAI_API_BASE.

I've tried to add this environment into the docker compose file and it works.

[debanjum]

Thanks a ton @aleung for looking into this! Glad that a simple change like updating the OPENAI_API_BASE environment variable works. I was able to reproduce this on my end as well using the LiteLLM proxy server and the Mistral API.

Let me update our docs with more complete instructions on how to use this soon.

[tupini07]

@mooangus I'm curious whether you actually managed to get the connection with Azure working after this change was implemented? Azure uses a different route for the endpoints than OpenAI, and I think there're also some difference in how the payload is built, so I was wondering if simply having the ability to change the URL allowed you to connect to Azure OpenAI?

Perhaps there's some more change necessary to support this: the openai package provides a specific constructor for AzureOpenAI clients. This page highlights the differences between the two https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints