Any way to use OpenLLM endpoints with OpenAI and LangChain?

[newparad1gm]

I see now there is a way to use OpenLLM with OpenAI's API by passing in the OpenLLM domain in as base_url to openai.OpenAI.

My question is can this be continued to be used with LangChain's llms and chat_models like ChatOpenAI? My main issue is I cannot currently use the OpenLLM class from langchain.llms because that requires an install of the openllm package, and I'm getting a break with starting this up in Django in a Celery worker locally on my Mac due to kqueue problems with trio. The OpenLLM model is remotely hosted. So I was wondering if I could use the ChatOpenAI class instead with the OpenLLM domain to use in the service context for vector generation and loading that index. I am currently trying to use it with LangChain and llama-index like

from langchain.chat_models import ChatOpenAI
from llama_index import LLMPredictor, PromptHelper, ServiceContext

llm = ChatOpenAI(openai_api_base='http://openllm-endpoint/v1', openai_api_key='na', model='meta-llama--Llama-2-7b-chat-hf')
prompt_helper = PromptHelper(4096, chunk_overlap_ratio=0.1, chunk_size_limit=1024)
llm_predictor = LLMPredictor(llm=llm)
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper, embed_model='local')

But I am getting the error: Unknown model 'meta-llama--Llama-2-7b-chat-hf'. Please provide a valid OpenAI model name in: gpt-4, etc, as I am running LLaMa 2 on the server and trying to use that model, and this is tripping on the openai_utils.py check for valid OpenAI model names. I guess I'm asking if there is a way to get around this check or fake the model name on the OpenLLM server hosting the model, or do I need to go to LangChain to see what they can do?

[newparad1gm]

Alright, I figured out a way to do this through OpenAILike on llama-index, so like this:

from langchain.chat_models import ChatOpenAI
from llama_index import LLMPredictor, PromptHelper, ServiceContext
from llama_index.llms import OpenAILike

llm = OpenAILike(api_base='http://openllm-endpoint/v1', api_key='na', is_local=True, context_window=4096, model=model, is_chat_model=True, is_function_calling_model=True)
prompt_helper = PromptHelper(4096, chunk_overlap_ratio=0.1, chunk_size_limit=1024)
llm_predictor = LLMPredictor(llm=llm)
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper, embed_model='local')

The only issue is that the /v1/chat/completions endpoint on the LLaMa2 OpenLLM server is breaking with the 500 error (the /v1/completions is working):

    "message": "Exception: Exception caught during generation: Response payload is not completed (check server log)",
    "type": "invalid_request_error",

Although it its working locally for me on a self deployed facebook/opt-1.3 model and another remotely hosted model with Falcon

[aarnphm]

What is the error you saw in the logs on the remote server?

[newparad1gm]

This is run on an AWS container, and the error log is like this

[llama-2-7-b-deployment-756f5c6d64-f6t6g] Traceback (most recent call last):
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/openllm/_llm.py", line 107, in generate_iterator
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     async for out in generator:
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/bentoml/_internal/runner/runner_handle/remote.py", line 330, in async_stream_method
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     async for b, end_of_http_chunk in resp.content.iter_chunks():
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/aiohttp/streams.py", line 60, in __anext__
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     rv = await self._stream.readchunk()
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/aiohttp/streams.py", line 431, in readchunk
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     await self._wait("readchunk")
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/aiohttp/streams.py", line 302, in _wait
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     await waiter
[llama-2-7-b-deployment-756f5c6d64-f6t6g] aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
[llama-2-7-b-deployment-756f5c6d64-f6t6g] 
[llama-2-7-b-deployment-756f5c6d64-f6t6g] The above exception was the direct cause of the following exception:
[llama-2-7-b-deployment-756f5c6d64-f6t6g] 
[llama-2-7-b-deployment-756f5c6d64-f6t6g] Traceback (most recent call last):
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/openllm/entrypoints/openai.py", line 187, in chat_completions
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     async for res in result_generator:
[llama-2-7-b-deployment-756f5c6d64-f6t6g]   File "/usr/local/lib/python3.9/dist-packages/openllm/_llm.py", line 117, in generate_iterator
[llama-2-7-b-deployment-756f5c6d64-f6t6g]     raise RuntimeError(f'Exception caught during generation: {err}') from err
[llama-2-7-b-deployment-756f5c6d64-f6t6g] RuntimeError: Exception caught during generation: Response payload is not completed
[llama-2-7-b-deployment-756f5c6d64-f6t6g] 2023-12-12T15:29:47+0000 [ERROR] [api_server:5] Error generating completion: Exception caught during generation: Response payload is not completed 

Wondering if there was a config issue with the load balance or some other GPU issue? I am running this with stream=False.

I also saw that you can just start it with the OpenLLMAPI on llama_index.llms.openllm, and that only needs openllm-client. I'm guessing I can just install that hopefully and run with it.

[aarnphm]

What is the whole traceback?

You can install openllm-client standalone.