Environment: livekit-agents 1.5.1, livekit-plugins-openai realtime path against the xAI Grok realtime API (grok-voice-think-fast-1.0), server-side VAD. Verified the same decision logic in 1.6.5 agent_activity.py: fnc_executed_ev._reply_required → interrupt() + _realtime_reply_task(...) with no check on whether the triggering response already produced audio.
Behavior: reasoning-class realtime models frequently bundle an audible assistant message AND a function_call in ONE response — despite prompt instructions to call tools silently. The framework's post-tool reply decision (reply_required, i.e. the tool returned a value) is computed after that bundled audio has already played, and fires unconditionally: interrupt() (a blanket response.cancel) followed by a fresh response.create. The caller audibly hears the agent answer the same turn twice.
Wire capture from a real session (timestamps to the ms):
09:39:40.532 input_audio_buffer.committed
09:39:40.755 response.created (VAD-created, id=df812bbb…)
09:39:41.048 response.output_item.added (message, content=[] at add time)
09:39:41.050 response.content_part.added (audio) ← ~6.4s of audible speech streams
09:39:47.451 response.output_item.added (function_call)
09:39:47.453 response.done
09:39:47.479 FUNCTION_TOOLS_EXECUTED reply_required=True
09:39:47.768 response.created (framework continuation, id=19a93a3c…, client response.create)
→ a second audible message answering the SAME turn again
Gap between the bundled audio and the duplicate: ~0.3s. In a live test the caller reacted with "Can you stop speaking?"
Why applications cannot fix this at their layer (we tried, extensively): at function_tools_executed time an app-side guard knows the turn already delivered audio, but cannot know whether that audio was a throwaway filler ("one moment…" → continuation NEEDED) or the complete reply (continuation = audible duplicate). No wire signal distinguishes the two at decision time. We shipped both guard polarities: cancelling the continuation froze the agent mid-call whenever the bundle was a filler; allowing it produces the duplicate whenever the bundle was complete. Prompt instructions ("call the tool silently") are exactly what reasoning models fail to follow here — that is what creates the bundle.
The framework CAN know: AgentActivity/RealtimeSession sees the response that carried the function_call, including whether it contained audio output items/deltas and how much played out — information the app never gets at decision time.
Ask — gate the post-tool reply on whether the function-call response already produced audible output, or expose the decision. Any of these shapes would unblock:
- A config/capability like
tool_reply_generation="skip_if_audio".
- Expose
had_audio (or an output-item summary of the triggering response) on FunctionToolsExecutedEvent, so cancel_tool_reply()-style decisions can be informed rather than blind.
- A user callback consulted before
_realtime_reply_task when the triggering response contained an audible message item.
Related: #4554 (Gemini speaks twice after function calls — closed with a prompt-instruction workaround; the trace above is a case where prompt instructions are structurally unreliable) and #2407 (auto_tool_reply_generation configurability). This report adds the wire-level trace of the bundling and evidence that both app-side workaround polarities fail.
Happy to provide full dual-direction JSONL wire traces for both the duplicate case and the freeze case.
Environment: livekit-agents 1.5.1,
livekit-plugins-openairealtime path against the xAI Grok realtime API (grok-voice-think-fast-1.0), server-side VAD. Verified the same decision logic in 1.6.5agent_activity.py:fnc_executed_ev._reply_required→interrupt()+_realtime_reply_task(...)with no check on whether the triggering response already produced audio.Behavior: reasoning-class realtime models frequently bundle an audible assistant message AND a
function_callin ONE response — despite prompt instructions to call tools silently. The framework's post-tool reply decision (reply_required, i.e. the tool returned a value) is computed after that bundled audio has already played, and fires unconditionally:interrupt()(a blanketresponse.cancel) followed by a freshresponse.create. The caller audibly hears the agent answer the same turn twice.Wire capture from a real session (timestamps to the ms):
Gap between the bundled audio and the duplicate: ~0.3s. In a live test the caller reacted with "Can you stop speaking?"
Why applications cannot fix this at their layer (we tried, extensively): at
function_tools_executedtime an app-side guard knows the turn already delivered audio, but cannot know whether that audio was a throwaway filler ("one moment…" → continuation NEEDED) or the complete reply (continuation = audible duplicate). No wire signal distinguishes the two at decision time. We shipped both guard polarities: cancelling the continuation froze the agent mid-call whenever the bundle was a filler; allowing it produces the duplicate whenever the bundle was complete. Prompt instructions ("call the tool silently") are exactly what reasoning models fail to follow here — that is what creates the bundle.The framework CAN know:
AgentActivity/RealtimeSessionsees the response that carried thefunction_call, including whether it contained audio output items/deltas and how much played out — information the app never gets at decision time.Ask — gate the post-tool reply on whether the function-call response already produced audible output, or expose the decision. Any of these shapes would unblock:
tool_reply_generation="skip_if_audio".had_audio(or an output-item summary of the triggering response) onFunctionToolsExecutedEvent, socancel_tool_reply()-style decisions can be informed rather than blind._realtime_reply_taskwhen the triggering response contained an audible message item.Related: #4554 (Gemini speaks twice after function calls — closed with a prompt-instruction workaround; the trace above is a case where prompt instructions are structurally unreliable) and #2407 (
auto_tool_reply_generationconfigurability). This report adds the wire-level trace of the bundling and evidence that both app-side workaround polarities fail.Happy to provide full dual-direction JSONL wire traces for both the duplicate case and the freeze case.