Skip to content

XAI :livekit-plugins-xai: RealtimeSession treats in-progress input transcriptions (status="in_progress") as final, duplicating user conversation items #6271

Description

@areebkhan-tech

Bug Description

xAI's realtime API streams partial input transcriptions through the
conversation.item.input_audio_transcription.completed channel with
status: "in_progress", and sends the final with status: "completed".
(OpenAI, by contrast, streams partials via ...transcription.delta and sends
...completed only once.)

The xAI RealtimeSession overrides this handler but still delegates to the OpenAI
base implementation, which ignores status and always emits
InputTranscriptionCompleted(is_final=True):

# livekit/plugins/xai/realtime/realtime_model.py
def _handle_conversion_item_input_audio_transcription_completed(self, event):
    if remote_item := self._remote_chat_ctx.get(event.item_id):
        if remote_item.item.type == "message" and remote_item.item.text_content == event.transcript:
            remote_item.item.content.clear()
    super()._handle_conversion_item_input_audio_transcription_completed(event)  # emits is_final=True

So every in-progress partial is surfaced as a final transcription. Downstream,
AgentActivity finalizes a user ChatMessage and emits conversation_item_added
on each is_final=True, so one user utterance produces many
conversation_item_added (and user_input_transcribed(is_final=True)) events
with the same item_id and a growing transcript.

Observed (grok-voice realtime), single utterance:
completed status=in_progress "What is"
completed status=in_progress "What is my name? What is your"
completed status=in_progress "What is my name? What is your Wi-Fi?"
completed status=completed "What is my name? What is your Wi-Fi?"
-> 4 user conversation items emitted instead of 1.

This is xAI-specific (xAI's status-based streaming), so the fix belongs in the
xAI plugin's RealtimeSession, not in the shared OpenAI base handler.

Expected Behavior

The xAI RealtimeSession should honor xAI's status field: an "in_progress"
transcription is an interim and should be emitted as is_final=False (like a
delta), so it does not finalize a conversation item; only status="completed"
should yield is_final=True (and the single final conversation item).

Suggested handling in the xAI override (its own implementation for the realtime
session, without changing OpenAI base behavior):

def _handle_conversion_item_input_audio_transcription_completed(self, event):
    # xAI sends interim transcripts as "completed" with status="in_progress"
    if getattr(event, "status", None) == "in_progress":
        self.emit(
            "input_audio_transcription_completed",
            llm.InputTranscriptionCompleted(
                item_id=event.item_id,
                transcript=event.transcript,
                is_final=False,
            ),
        )
        return
    # status == "completed" (or absent): treat as final via base handling
    if remote_item := self._remote_chat_ctx.get(event.item_id):
        if remote_item.item.type == "message" and remote_item.item.text_content == event.transcript:
            remote_item.item.content.clear()
    super()._handle_conversion_item_input_audio_transcription_completed(event)

Result:

  • in_progress -> is_final=False -> interim only, no duplicate conversation item.
  • completed -> is_final=True -> exactly one final conversation item.
    Interim user_input_transcribed events are preserved for UI placeholders.

Reproduction Steps

1. Start an AgentSession with livekit-plugins-xai RealtimeModel
   (e.g. grok-voice-think-fast-1.0), input audio transcription enabled.
2. Subscribe to conversation_item_added (and/or input_audio_transcription_completed).
3. Speak one sentence.
4. Observe multiple is_final=True / conversation_item_added events for the same
   item_id with a growing transcript, instead of a single final one.

Operating System

macOS

Models Used

No response

Package Versions

livekit-agents === 1.5.17
livekit-plugins-xai == 1.5.17
livekit-plugins-openai == 1.5.17

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions