Bug Description
xAI's realtime API streams partial input transcriptions through the
conversation.item.input_audio_transcription.completed channel with
status: "in_progress", and sends the final with status: "completed".
(OpenAI, by contrast, streams partials via ...transcription.delta and sends
...completed only once.)
The xAI RealtimeSession overrides this handler but still delegates to the OpenAI
base implementation, which ignores status and always emits
InputTranscriptionCompleted(is_final=True):
# livekit/plugins/xai/realtime/realtime_model.py
def _handle_conversion_item_input_audio_transcription_completed(self, event):
if remote_item := self._remote_chat_ctx.get(event.item_id):
if remote_item.item.type == "message" and remote_item.item.text_content == event.transcript:
remote_item.item.content.clear()
super()._handle_conversion_item_input_audio_transcription_completed(event) # emits is_final=True
So every in-progress partial is surfaced as a final transcription. Downstream,
AgentActivity finalizes a user ChatMessage and emits conversation_item_added
on each is_final=True, so one user utterance produces many
conversation_item_added (and user_input_transcribed(is_final=True)) events
with the same item_id and a growing transcript.
Observed (grok-voice realtime), single utterance:
completed status=in_progress "What is"
completed status=in_progress "What is my name? What is your"
completed status=in_progress "What is my name? What is your Wi-Fi?"
completed status=completed "What is my name? What is your Wi-Fi?"
-> 4 user conversation items emitted instead of 1.
This is xAI-specific (xAI's status-based streaming), so the fix belongs in the
xAI plugin's RealtimeSession, not in the shared OpenAI base handler.
Expected Behavior
The xAI RealtimeSession should honor xAI's status field: an "in_progress"
transcription is an interim and should be emitted as is_final=False (like a
delta), so it does not finalize a conversation item; only status="completed"
should yield is_final=True (and the single final conversation item).
Suggested handling in the xAI override (its own implementation for the realtime
session, without changing OpenAI base behavior):
def _handle_conversion_item_input_audio_transcription_completed(self, event):
# xAI sends interim transcripts as "completed" with status="in_progress"
if getattr(event, "status", None) == "in_progress":
self.emit(
"input_audio_transcription_completed",
llm.InputTranscriptionCompleted(
item_id=event.item_id,
transcript=event.transcript,
is_final=False,
),
)
return
# status == "completed" (or absent): treat as final via base handling
if remote_item := self._remote_chat_ctx.get(event.item_id):
if remote_item.item.type == "message" and remote_item.item.text_content == event.transcript:
remote_item.item.content.clear()
super()._handle_conversion_item_input_audio_transcription_completed(event)
Result:
- in_progress -> is_final=False -> interim only, no duplicate conversation item.
- completed -> is_final=True -> exactly one final conversation item.
Interim user_input_transcribed events are preserved for UI placeholders.
Reproduction Steps
1. Start an AgentSession with livekit-plugins-xai RealtimeModel
(e.g. grok-voice-think-fast-1.0), input audio transcription enabled.
2. Subscribe to conversation_item_added (and/or input_audio_transcription_completed).
3. Speak one sentence.
4. Observe multiple is_final=True / conversation_item_added events for the same
item_id with a growing transcript, instead of a single final one.
Operating System
macOS
Models Used
No response
Package Versions
livekit-agents === 1.5.17
livekit-plugins-xai == 1.5.17
livekit-plugins-openai == 1.5.17
Session/Room/Call IDs
No response
Proposed Solution
Additional Context
No response
Screenshots and Recordings
No response
Bug Description
xAI's realtime API streams partial input transcriptions through the
conversation.item.input_audio_transcription.completedchannel withstatus: "in_progress", and sends the final withstatus: "completed".(OpenAI, by contrast, streams partials via
...transcription.deltaand sends...completedonly once.)The xAI RealtimeSession overrides this handler but still delegates to the OpenAI
base implementation, which ignores
statusand always emitsInputTranscriptionCompleted(is_final=True):So every in-progress partial is surfaced as a final transcription. Downstream,
AgentActivity finalizes a user ChatMessage and emits
conversation_item_addedon each
is_final=True, so one user utterance produces manyconversation_item_added(anduser_input_transcribed(is_final=True)) eventswith the same item_id and a growing transcript.
Observed (grok-voice realtime), single utterance:
completed status=in_progress "What is"
completed status=in_progress "What is my name? What is your"
completed status=in_progress "What is my name? What is your Wi-Fi?"
completed status=completed "What is my name? What is your Wi-Fi?"
-> 4 user conversation items emitted instead of 1.
This is xAI-specific (xAI's status-based streaming), so the fix belongs in the
xAI plugin's RealtimeSession, not in the shared OpenAI base handler.
Expected Behavior
The xAI RealtimeSession should honor xAI's
statusfield: an "in_progress"transcription is an interim and should be emitted as is_final=False (like a
delta), so it does not finalize a conversation item; only status="completed"
should yield is_final=True (and the single final conversation item).
Suggested handling in the xAI override (its own implementation for the realtime
session, without changing OpenAI base behavior):
Result:
Interim user_input_transcribed events are preserved for UI placeholders.
Reproduction Steps
1. Start an AgentSession with livekit-plugins-xai RealtimeModel (e.g. grok-voice-think-fast-1.0), input audio transcription enabled. 2. Subscribe to conversation_item_added (and/or input_audio_transcription_completed). 3. Speak one sentence. 4. Observe multiple is_final=True / conversation_item_added events for the same item_id with a growing transcript, instead of a single final one.Operating System
macOS
Models Used
No response
Package Versions
Session/Room/Call IDs
No response
Proposed Solution
Additional Context
No response
Screenshots and Recordings
No response