Add support for streaming LLM generation

[FBR65]

Hi,

the Error is thrown in line 61 litellm.py:

---

results.append(result["choices"][0]["message"]["content"])

'CustomStreamWrapper' object is not subscriptable

---

Call is
from txtai.pipeline import LLM

MODEL_NAME = "huggingface/TheBloke/leo-hessianai-70B-chat-GPTQ"
llm = LLM(path=MODEL_NAME,method="litellm", api_base=api_base,stream=True)

---

This works fine:

import litellm
from litellm import completion

MODEL_NAME = "huggingface/TheBloke/leo-hessianai-70B-chat-GPTQ"
messages = [{"content": "C", "role": "user"}] # LiteLLM follows the OpenAI format
api_base = "http://127.0.0.1:8080"

CALLING ENDPOINT

response=completion(model=MODEL_NAME, messages=messages, api_base=api_base,stream=True)
for part in response:
print(part.choices[0].delta.content or "")

[davidmezzetti]

Thank you for the issue. I'll see if streaming support can be added in.

[davidmezzetti]

It's been a while on this but streaming will be added shortly.