Streaming responses

[niutech]

This is a follow-up on issue #46. Is it possible to stream responses from LLM in Q&A? For example, LiteLLM allows streaming responses by adding stream=True. Thanks!

[Saksham65]

Hi there,
Thanks for your question. As discussed internally:

1. Current Support: Our LiteLLMChat wrapper does not support using stream=True to split a single LLM output into multiple messages over time. The current wrapper returns the full response once generated.

2. Technical Considerations: Implementing streaming would require a significant redesign—specifically around how UDFs return results (for instance, allowing them to yield partial outputs). This isn’t supported by the current engine.

3. Kafka and Output Connectors: While you can stream question and answer pairs to systems like Kafka by adjusting how the webserver handles responses, splitting one answer into multiple parts is not currently implemented.

[niutech]

Thank you for your reply. Is it possible to bypass UDFs altogether and just stream raw responses from LiteLLM?