Own Api LLM

[brightebyte]

Feature request

The Project Basaran lets you host an api that is similar to the OpenAI api, but with a self hosted llm. Support for such custom API LLMs would be great

Motivation

Further democritizig LLMs

Your contribution

I could test it

[joshuasundance-swca]

Basaran is designed so that you can use the openai Python library with a different base url. See their example here. This should make it easy to use Basaran in LangChain.

I was able to call a local Basaran server by setting environment variables OPENAI_API_BASE and OPENAI_API_KEY before creating a langchain.llms.openai.OpenAI object, or by passing those values in as keyword arguments. You could also set model_name.

I tested using 0.19.0 because 0.21.1 seems to have a breaking change.

docker run -p 80:80 -e MODEL=bigscience/bloomz-560m hyperonym/basaran:0.19.0

>>> import langchain
>>> # 172.28.0.3 is the IP of the basaran server on my docker network
>>> llm = langchain.llms.openai.OpenAI(
...     openai_api_base='http://172.28.0.3/v1',
...     openai_api_key='this_value_does_not_matter',
...     model_name='bigscience/bloomz-560m',
... )
>>> llm.openai_api_base, llm.openai_api_key, llm.model_name
('http://172.28.0.3/v1', 'this_value_does_not_matter', 'bigscience/bloomz-560m')
>>> llm('Large language models can be used to')
' generate a wealth of data'
>>>

It's worth noting that if you do it just like this without further modification, tracing will probably just say you were using OpenAI. That may or may not be a problem for you.

If you wanted to, you could probably create a Basaran LLM class similar to how ChatAnyscale was implemented. See the API reference here and the source code here.

According to Basaran's README, it looks like implementing chat models may be more difficult.

[dosubot]

Hi, @brightebyte

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you opened this issue to request support for custom API LLMs in the Basaran project to enable users to host a self-hosted LLM similar to OpenAI's. joshuasundance-swca provided guidance on using the openai Python library with a different base URL in Basaran and suggested creating a Basaran LLM class similar to the implementation of ChatAnyscale, which received a "THUMBS_UP" reaction from mateuszkurdziel.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!