Enhancement: Change OPENAI_REVERSE_PROXY to only expect a network host / port.

[Mist-Hunter]

Contact Details

No response

What features would you like to see added?

Thanks so much for this great project. There aren't many working front ends like yours let alone with all these great features. I do have a question, similar to the comment below on how best to point LibreChat at my localai container. It's already pretty much working, but I was wondering how feasible this change below might be.

I don't know how dumb an idea this might be, but is it possible to alter the code so that OPENAI_REVERSE_PROXY is not expected to provide any api paths /v1 /v1/chat/completions, and those are just presumed? I'm guessing the path is built in for other use cases I'm not aware of, but in the event it's not, it would make integrating with localai fully automatic.

More details

#403 (comment)

In the comment above, you point out how to hard code the models in the event that you are running your own reverse proxy (or using the OPENAI_REVERSE_PROXY env as a way to access local llms like via localai).

If I set

OPENAI_REVERSE_PROXY: http://host.docker.internal:8070/v1

Librechat will grab the model list from localai without issue from /v1/models, but will fail to post to /v1 because that's not the right end point.

If I set

OPENAI_REVERSE_PROXY: http://host.docker.internal:8070/v1/chat/completions

Then my posts go through and I can chat with the localai models, but only if I've hard coded them ahead of time.

Which components are impacted by your request?

No response

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[Mist-Hunter]

I noticed the CHATGPT_REVERSE_PROXY in some of the documentation so I tried

CHATGPT_REVERSE_PROXY: http://host.docker.internal:8070/v1
OPENAI_REVERSE_PROXY: http://host.docker.internal:8070/v1/chat/completions

Just to see what would happen, and it polled the models as expect and was able to post chats as expected. This gets me where I wanted to be.

[danny-avila]

Hi, thanks for using LibreChat! This is related to #1026 except the point about fetching models. What endpoint is it for LocalAI to fetch models? I can try setting it up today to test

[danny-avila]

Actually I'm a little confused here. It should be fetching models correctly with this:

OPENAI_REVERSE_PROXY: http://host.docker.internal:8070/v1/chat/completions

I'm testing LocalAI in a minute

[Mist-Hunter]

In local ai, I see the models @ http://localai:port/v1/models

[danny-avila]

I think I know what's going on. I'm not gonna dive deep into LocalAI setup but basically, you can only start posting to models out-of-the-box if they can receive a messages payload, which I believe you need some template to format the JSON messages.

I think without a template, the payload will need to receive a single text prompt, which makes it work:

Note I am using OPENAI_REVERSE_PROXY with the OpenAI endpoint, I am now able to post messages in this endpoint while also being able to select models:

I remember being able to use LocalAI no problem without forcing the prompt payload, but I must've been using a model that has OpenAI formatting out-of-the-box

[danny-avila]

Relevant LocalAI docs: https://localai.io/advanced/index.html

For now, as a quick patch, I can add an environment variable to force the prompt payload, but to make the UI setup more flexible, I can make this an option on the frontend soon

[danny-avila]

Actually, i just realized this is happening due to calling /v1/completions and not /v1/chat/completions, idk why I set my reverse proxy without chat lol

In any case, calling /chat/completions needs a template or the LLM will just continue your response

I will add an option to check for /completions and force the prompt in this case, also allowing the user to force prompt payload with .env var

[Mist-Hunter]

Thanks for looking into this so quickly! I'll have to check this out.