I think the current implementation of preemptive generation is wrong.

[Panmax]

The current implementation of preemptive generation relies on the _generate_reply process of AgentActivity. This poses no problem in pure voice conversation agent scenarios, but if business logic is handled in on_user_turn_completed, llm_node, or tts_node (such as modifying elements on the page), issues may arise. Due to the existence of preemptive generation, similar transcription results may trigger on_user_turn_completed and tts_node multiple times, which can have an erroneous impact on the state machine.

I believe a more reasonable way to implement preemptive generation is to simply call the large model in advance, cache the result, and only go through those node processes when it is confirmed that the preemptive generation data will be used.

[longcw]

but if business logic is handled in on_user_turn_completed, llm_node, or tts_node (such as modifying elements on the page), issues may arise.

what do you mean issues may raise from these? the preemptive generation is designed to start LLM and TTS generations right after the transcript is received, the agent frame work will wait for a delay to confirm the user speech is done.

if there is new transcript comes in or the contents modified in on_user_turn_completed, the preemptive generation will be canceled and not used.

I think it's better if you can share some error cases.

I believe a more reasonable way to implement preemptive generation is to simply call the large model in advance, cache the result, and only go through those node processes when it is confirmed that the preemptive generation data will be used.

This is exactly what preemptive generation does.

[bay3s]

Hey guys,

Can we revisit this - agree with @Panmax here, we should perhaps consider caching tool calls and other execution directives from preemptive generation rather than running those in the course of preemptive generation.

Reason being that tool calls and parameters might change during preemptive generation itself, and the final results that we expect might not be the same as what get executed during the course of an intermediate generation.

Best,
Sid

[longcw]

@bay3s

caching tool calls and other execution directives from preemptive generation rather than running those

this is already the current behavior, the tools are executed after generation is authorized.