google.realtime.RealtimeModel sends AudioSegmentEnd too late

[filipre]

Bug Description

Hi!
I'm a developer at Beyond Presence where we provide a virtual avatar for voice agents, see this setup. We noticed an issue with the Gemini Realtime model where the AudioSegmentEnd is sent too late from either LiveKit or the Google RealtimeModel (Gemini) while OpenAI Realtime works fine. This makes it difficult for us to react on time and sometimes causes lags for the user. On some occasions, there is also a residual audio chunk, too.

Expected Behavior

OpenAI sends audio data like this:

rtc.AudioFrame -> ... -> rtc.AudioFrame -> AudioSegmentEnd

Gemini on the other hand looks like this:

Case A: Without residual audio

rtc.AudioFrame -> ... -> rtc.AudioFrame -> [nothing for several seconds] -> AudioSegmentEnd

Case B: With residual audio

rtc.AudioFrame -> ... -> rtc.AudioFrame -> [nothing for several seconds] -> rtc.AudioFrame -> AudioSegmentEnd

Reproduction Steps

1. Talk to an agent using Gemini Realtime model
2. Notice that all `rtc.AudioFrame` come in really fast except the last one
3. Notice that `AudioSegmentEnd` comes in too late. It should come in right after the last audio chunk.

Operating System

Linux, macOS

Models Used

google.realtime.RealtimeModel

Package Versions

"livekit-agents == 1.6.5",
    "livekit-plugins-bey == 1.6.5",
    "livekit-plugins-google==1.6.5",
    "livekit-plugins-openai>=1.6.5",
    "python-dotenv ~= 1.0",

Session/Room/Call IDs

No response

Proposed Solution

Right after the last audio chunk, a AudioSegmentEnd should be sent to indicate that the audio has finished.

Additional Context

No response

Screenshots and Recordings

I confirmed the issue in our logs. Notice that there is a 4s delay between the last audio chunk and the segment end

[filipre]

Hey @longcw , sadly the issue still happens for us in version 1.6.6. There is still a delay between the rtc.AudioFrame and AudioSegmentEnd. Could you reopen it? Also do you need more information?

[longcw]

@filipre yes, please share a reproducible example if possible.

[filipre]

Here is a minimal setup that reproduces the issue on 1.6.6. Notice that this VideoGenerator here won't mirror the audio but just returns zero data for audio and video. It shouldn't matter for the issue though.

agent.py (Voice agent)

import argparse
import sys
from functools import partial
from typing import Optional

from dotenv import load_dotenv
from livekit import rtc
from livekit.agents import (
    AutoSubscribe,
    JobContext,
    RoomOutputOptions,
    WorkerOptions,
    WorkerType,
    cli,
)
from livekit.agents.voice import Agent, AgentSession
from livekit.plugins import bey, google, openai
from livekit.agents.voice.avatar import AvatarSession as BaseAvatarSession, DataStreamAudioOutput


async def entrypoint(ctx: JobContext, avatar_id: Optional[str]) -> None:
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    print(f"Agent identity: {ctx.room.local_participant.identity}")

    # Gemini
    local_agent_session = AgentSession(
        llm=google.realtime.RealtimeModel(
            voice="Puck",
            temperature=0.8,
            instructions="You are a helpful assistant",
        ),
    )

    local_agent_session.output.replace_audio_tail(
        DataStreamAudioOutput(
            room=ctx.room,
            destination_identity="bey-avatar-agent",
            wait_remote_track=rtc.TrackKind.KIND_VIDEO,
        ),
    )

    await local_agent_session.start(
        agent=Agent(instructions="Talk to me!"),
        room=ctx.room,
    )


if __name__ == "__main__":
    load_dotenv()

    parser = argparse.ArgumentParser(description="Run a LiveKit agent with Bey avatar.")
    parser.add_argument("--avatar-id", type=str, help="Avatar ID to use.")
    args = parser.parse_args()

    sys.argv = [sys.argv[0], "dev"]  # overwrite args for the LiveKit CLI
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=partial(entrypoint, avatar_id=args.avatar_id),
            worker_type=WorkerType.ROOM,
        )
    )

avatar.py (Video Generation)

import asyncio

from livekit.agents.voice.avatar import AudioSegmentEnd, VideoGenerator
from livekit import rtc, api
import logging
from livekit.agents.voice.avatar import (
    AvatarOptions,
    AvatarRunner,
    DataStreamAudioReceiver,
)
from livekit.agents.voice.room_io import ATTRIBUTE_PUBLISH_ON_BEHALF
from collections.abc import (
    AsyncGenerator,
    Coroutine,
)

URL = "..."
API_KEY = "..."
API_SECRET = "..."
ROOM = "..."
AGENT = "..."

class OurVideoGenerator(VideoGenerator):
    def __init__(
        self,
        width: int = 1920,
        height: int = 1080,
        sample_rate: int = 16_000,
        num_channels: int = 1,
    ) -> None:
        self._width = width
        self._height = height
        self._sample_rate = sample_rate
        self._num_channels = num_channels

        y_size = width * height
        uv_size = (width // 2) * (height // 2)
        self._black_video_data = bytes([0]) * y_size + bytes([128]) * (2 * uv_size)

        samples_per_channel = sample_rate * 20 // 1000
        self._silence_audio_data = b"\x00\x00" * samples_per_channel * num_channels
        self._samples_per_channel = samples_per_channel

    async def push_audio(
        self, frame: rtc.AudioFrame | AudioSegmentEnd
    ) -> None:
        if isinstance(frame, AudioSegmentEnd):
            logging.info("Audio segment ended.")
            return

        if isinstance(frame, rtc.AudioFrame):
            logging.info("Audio frame received")

    def clear_buffer(self) -> Coroutine[None, None, None]:
        logging.info("Clearing audio buffer.")

    def __aiter__(
        self,
    ) -> AsyncGenerator[
        rtc.VideoFrame | rtc.AudioFrame | AudioSegmentEnd, None
    ]:
        return self.loop()

    async def loop(self) -> AsyncGenerator[
        rtc.VideoFrame | rtc.AudioFrame | AudioSegmentEnd, None
    ]:
        logging.info("Loop started")
        while True:
            frame = rtc.VideoFrame(
                width=self._width,
                height=self._height,
                type=rtc.VideoBufferType.I420,
                data=self._black_video_data,
            )
            audio_frame = rtc.AudioFrame(
                data=self._silence_audio_data,
                sample_rate=self._sample_rate,
                num_channels=self._num_channels,
                samples_per_channel=self._samples_per_channel,
            )
            yield frame
            yield audio_frame
            await asyncio.sleep(1 / 30)  # Simulate 30 FPS


async def main():
    room = rtc.Room()
    url = URL
    room_name = ROOM

    token = (
        api.AccessToken(api_key=API_KEY, api_secret=API_SECRET)
        .with_identity("bey-avatar-agent")
        .with_name("Avatar")
        .with_grants(api.VideoGrants(room_join=True, room=room_name))
        .with_kind("agent")
        .with_attributes({ATTRIBUTE_PUBLISH_ON_BEHALF: AGENT})
        .to_jwt()
    )

    await room.connect(url, token)

    avatar_options = AvatarOptions(
        video_width=1920,
        video_height=1080,
        video_fps=25,
        audio_sample_rate=16000,
        audio_channels=1,
    )
    video_gen = OurVideoGenerator(
        width=avatar_options.video_width,
        height=avatar_options.video_height,
        sample_rate=avatar_options.audio_sample_rate,
        num_channels=avatar_options.audio_channels,
    )

    runner = AvatarRunner(
        room=room,
        audio_recv=DataStreamAudioReceiver(room),
        video_gen=video_gen,
        options=avatar_options,
    )
    await runner.start()

    await asyncio.sleep(1000)

if __name__ == "__main__":
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s %(levelname)s %(message)s",
    )
    asyncio.run(main())

The output I see is the following:

2026-07-21 17:31:14,948 INFO Clearing audio buffer.
2026-07-21 17:31:17,071 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached
2026-07-21 17:31:17,071 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached
2026-07-21 17:31:17,071 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached
2026-07-21 17:31:17,072 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached
2026-07-21 17:31:18,714 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached
2026-07-21 17:31:18,714 INFO Audio frame received
2026-07-21 17:31:18,714 INFO Audio frame received
2026-07-21 17:31:18,714 INFO Audio frame received
2026-07-21 17:31:18,724 INFO Audio frame received
2026-07-21 17:31:18,724 INFO Audio frame received
2026-07-21 17:31:18,724 INFO Audio frame received
2026-07-21 17:31:18,728 INFO Audio frame received
2026-07-21 17:31:18,728 INFO Audio frame received
2026-07-21 17:31:18,728 INFO Audio frame received
2026-07-21 17:31:18,731 INFO Audio frame received
2026-07-21 17:31:18,732 INFO Audio frame received
2026-07-21 17:31:18,735 INFO Audio frame received
2026-07-21 17:31:18,807 INFO Audio frame received
2026-07-21 17:31:18,807 INFO Audio frame received
2026-07-21 17:31:18,832 INFO Audio frame received
2026-07-21 17:31:18,834 INFO Audio frame received
2026-07-21 17:31:18,875 INFO Audio frame received
2026-07-21 17:31:18,920 INFO Audio frame received
2026-07-21 17:31:18,930 INFO Audio frame received
2026-07-21 17:31:18,993 INFO Audio frame received
2026-07-21 17:31:18,994 INFO Audio frame received
2026-07-21 17:31:18,994 INFO ignoring text stream with topic 'lk.transcription', no callback attached
2026-07-21 17:31:19,021 INFO Audio frame received
2026-07-21 17:31:19,127 INFO Audio frame received
2026-07-21 17:31:19,128 INFO Audio frame received
2026-07-21 17:31:19,179 INFO Audio frame received
2026-07-21 17:31:19,179 INFO Audio frame received
2026-07-21 17:31:19,200 INFO Audio frame received
2026-07-21 17:31:19,201 INFO Audio frame received
2026-07-21 17:31:19,211 INFO Audio frame received
2026-07-21 17:31:19,212 INFO Audio frame received
2026-07-21 17:31:19,231 INFO Audio frame received
2026-07-21 17:31:19,278 INFO Audio frame received
2026-07-21 17:31:19,292 INFO Audio frame received
2026-07-21 17:31:19,313 INFO Audio frame received
2026-07-21 17:31:19,361 INFO Audio frame received
2026-07-21 17:31:19,367 INFO Audio frame received
# ... here is the delay ...
2026-07-21 17:31:22,371 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached
2026-07-21 17:31:22,373 INFO Audio frame received
2026-07-21 17:31:22,373 INFO Audio segment ended.
2026-07-21 17:36:57,351 INFO ignoring byte stream with topic 'lk.agent.session', no callback attached

Does this help?

[longcw]

I didn't reproduce it with the latest version with this example https://github.com/livekit/agents/tree/livekit-agents%401.6.5/examples/avatar_agents/audio_wave by changing the llm to gemini live.

16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.1
16:24:25 INFO avatar-example Audio frame received, duration=0.06
16:24:25 INFO avatar-example Audio segment ended
16:24:29 DEBUG livekit.agents notifying lk.playback_finished {"payload": "{\"playback_position\": 4.559999999999993, \"interrupted\": false, \"synchronized_transcript\": null}"}