-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#1155: Add support for OpenAI-compatible endpoint in LLM and Embed #1197
Conversation
LocalAI configs if you wanted to attempt utilization yourself:
|
Can we merge this PRs? It seems only add more config and change only the OpenAI Chat and Embeding to support the local endpoints. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding support for this.
Failing test is easy fix. Should not require the config or env for base var. Default should be None. Might get around to this this week. One line change imo |
yes, Merging this PR for now. Will fix the tests in my follow up PR. |
Description
Delivers using the OpenAI-specific term
base_url
for LLM andapi_base
for Embed. Tested on LocalAI.io for LLM and Embed. Original issued asked forendpoint
support, butbase_url
seemed superior to match OpenAI kwargs.Fixes #1155
Type of change
How Has This Been Tested?
It appears perfectly functional from LocalAI logs. The RAG operation itself returned "I dont know Elon's net worth", but all the RAG search results and LLM templating looked right in the debug. I personally plan to use EmbedChain with this change for my RAG ingest and search but to use my own template for synthesizing search results since running it against a 7b model looks like it will need a little more tinkering to behave as expected than what default prompt provides (maybe there is an obvious way to change the prompt I am missing??). I did not run any unit tests.
EDIT: I found the prompt override nvm: https://github.com/embedchain/embedchain/blob/9afc6878c82ee71332fa09aebecea93dd7829e7f/configs/full-stack.yaml#L18C5-L28C102
Checklist:
Maintainer Checklist