Feature Request: Support proxy URLs

[duncangrubbs]

Problem Summary

There is no easy way to configure a pipeline to hit anything besides the default service provider endpoint for LLM models. For example, the default llm_models in the registry for OpenAI hardcode the endpoints from Endpoints.

Current Solution(s)

You can, in theory, override this by creating a custom model in the registry and replacing endpoint=Endpoints.CHAT.value, when initializing OpenAI. However, since you are not changing the value of the Endpoints enum, you still get errors during validation. To get around this, you can implement a custom model class that extends REST but doesn't do any endpoint validation. However this feels like overkill for what otherwise would be such a simple change.

Although I have not gotten it to work, another approach would be to use the langchain.OpenAI.v1 model and then pass openai_api_base to the config since langchain supports proxies. However, from my testing there isn't stable langchain support for the chat completions API, and this also just feels like overkill.

Proposed Solution

I think the ideal solution would be to essentially support the openai_api_base parameter natively. Something like

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
name = "gpt-4"
endpoint = "YOUR_PROXY_URL"

The model would still default endpoint=Endpoints.CHAT.value, but would be overridden if the user passed a value. The validation step could stay, but instead of raising an exception it could just log a warning that you are sending requests to a proxy.

[rmitsch]

Thanks for this suggestion! We have this on our backlog, one of the next releases will include native support for custom endpoints 🙂

[duncangrubbs]

@rmitsch That's great to hear! I look forward to the release