does llamaindex Workflow provide any support to stream the LLM's output？

[qiankunli]

In many cases, a Workflow step is a LLM call. For such steps, does Workflow provide any support to stream the LLM's output through methods like async_response_gen, This way, we can stream the responses of a multi-step/multi-turn LLM inference task to the front end, thereby improving user experience.

result = await w.run(query="Where is the capital of China?")
async for chunk in result.async_response_gen():
    print(chunk, end="", flush=True)
    print('\n')

output

The capital 
of 
China 
is Beijing.

If async_response_gen supports this request. Can you explain the principle of async_response_gen, and how the LLM response is injected into the workflow through an LLM call?

[dosubot[bot]]

Hey @qiankunli! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's tackle this together!

Yes, the LlamaIndex Workflow supports streaming the LLM's output using the async_response_gen method. This method is part of the StreamingAgentChatResponse class and is designed to handle the streaming of LLM responses in a multi-step or multi-turn inference task.

The async_response_gen method works by asynchronously yielding parts of the response as they become available. It does this by checking an asynchronous queue (aqueue) or an asynchronous chat stream (achat_stream). If is_writing_to_memory is True, it retrieves text deltas from the aqueue, handling exceptions and timeouts to ensure smooth streaming. If is_writing_to_memory is False, it streams directly from achat_stream. This allows the system to process and display parts of the response in real-time, improving user experience by providing immediate feedback during interactions ^[1][2].

In your example, the async_response_gen method is used to stream the response of an LLM call, allowing you to print each chunk of the response as it is generated. This is particularly useful for applications that require real-time interaction with users.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other | Bug Report}

[logan-markewich]

@qiankunli use the event streaming api

For example

class LLMStream(Event):
  delta: str

# then, a step in your class
@step
async def run_llm(self, ctx: Context, ev: SomeEvent) -> SomeOtherEvent:
  resp = await llm.astream_chat(some_messages):
  async for r in resp:
    ctx.write_event_to_stream(LLMStream(delta=r.delta or ""))
  ...
  
# then when running the worklow
handler = workflow.run(...)
async for ev in handler.stream_events():
  if isinsantce(ev, LLMStream):
    print(ev.delta, end="", flush=True)
response = await handler

https://docs.llamaindex.ai/en/stable/module_guides/workflow/#streaming-events