Should AutoRecLab stream LLM output?

[baaelke]

Problem

Currently, AutoRecLab sends prompts to the LLM and blocks until the full response is returned. During long steps, the user sees only a label for minutes straight. There is no indication of what the agent is doing, whether it's stuck etc.

Idea

Stream the LLM's output to the terminal in real time, so users can follow the agent's reasoning as it happens. It could include token by token streaming of the agent's code generation, plans, scoring and feedback but also chain of thought output. This could then be configurable.

Beneftis

• Transparency: users can see why the agent made decisions, not just the final code
• Option of aborting: if the agent is clearly going in the wrong direction, the user could have the option to interrupt the current iteration and effectively skip it. This would spare time and API costs
• Easier debugging

Question for discussion

• Stream everything or only reasoning?
• Logging vs display? Should streamed output also be written to the log file, or only displayed in the terminal? Preserving full CoT in logs would maybe be too much.
• --stream flag? I do not think it should be on by default but enabling this output-streaming with a flag seems like a good feature.
• Is this feature necessary? AutoRecLab is designed to run in the background for hours. Still, IMO the option to see LLM output would benefit the tool since it "shows" progress.