ToolFlag.IGNORE_ON_ENTER Is Broken With Amazon Nova 2 Sonic

[ShaneAtPennymac]

Bug Description

When using the ToolFlag.IGNORE_ON_ENTER flag on a function_tool with AWS nova 2 sonic, multiple generate_reply calls in an on_enter function won't play audio. IE, the first generate_reply call will generate audio, however subsequent generate_reply calls will not. This only affects livekit-agents and livekit-plugins-aws[realtime] versions greater or equal to 1.6.5. Version 1.6.4 is does not contain the defect.

Expected Behavior

ToolFlag.IGNORE_ON_ENTER won't affect audio being played from an on_enter function. The flag is only supposed to exclude any tool from any generate_reply calls made inside the agent's on_enter method.

Reproduction Steps

Test file:

from livekit.agents import AgentServer, WorkerOptions, JobContext, AutoSubscribe, function_tool, RunContext
from livekit.agents.cli import run_app
from livekit.agents.llm import ToolFlag
from livekit.plugins import aws
from livekit.agents import TurnHandlingOptions, AgentSession, Agent
import dotenv


# loads the following env vars in .env file.
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
# AWS_SESSION_TOKEN
# AWS_REGION
# LIVEKIT_URL
# LIVEKIT_API_KEY
# LIVEKIT_API_SECRET
dotenv.load_dotenv()


def main() -> None:
    server = AgentServer.from_server_options(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            agent_name='fake-agent-name',  # CHANGE THIS
            load_threshold=0.99,
            max_retry=6,
            port=8080,
            drain_timeout=900,
            prometheus_port=9095,
            multiprocessing_context="spawn",
        )
    )
    run_app(server)

async def entrypoint(ctx: JobContext) -> None:
    session = AgentSession(
        llm=aws.realtime.RealtimeModel(
            model="amazon.nova-2-sonic-v1:0",
            tool_choice="auto",
            max_tokens=6000,
            temperature=0,
            region='us-west-2',
            turn_detection="HIGH",
        ),
        tts=aws.TTS(voice="Ruth", region='us-west-2'),
        turn_handling=TurnHandlingOptions(
            interruption={"resume_false_interruption": True, "false_interruption_timeout": 1.0},
        ),
    )
    await session.start(room=ctx.room, agent=TestAgent())
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)


class TestAgent(Agent):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(
            *args,
            instructions='you are test agent. just have a conversation',
            tools=[test_tool],
            **kwargs,
        )

    async def on_enter(self):
        await self.session.generate_reply(instructions='say hello')  # will generate audio
        await self.session.generate_reply(instructions='introduce yourself') # doesn't generate audio
        await self.session.generate_reply(instructions='say goodbye')  # doesn't generate audio


@function_tool(flags=ToolFlag.IGNORE_ON_ENTER)
def test_tool(context: RunContext) -> None:
    """Call this tool when the user requests that you execute the test tool."""
    context.session.generate_reply(instructions='announce that you just called the test tool')


if __name__ == "__main__":
    main()

Call into the agent, and the generate_reply calls in the on_enter function will not play audio. Removing ToolFlag.IGNORE_ON_ENTER from the function tool and calling back will play audio

Operating System

macOS Tahoe/Linux Debian GNU/Linux 12 (bookworm)

Models Used

Nova 2 Sonic

Package Versions

livekit-agents==1.6.8
livekit-plugins-aws[realtime]==1.6.8
livekit==1.1.14
livekit-api==1.2.0

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response