[AI-269] Eager turn taking by tschellenbach · Pull Request #169 · GetStream/Vision-Agents

tschellenbach · 2025-11-11T19:17:01Z

Eager turn taking support
Fix screensharing support for Gemini & OpenAI
Agent testing for wait for participant
Agent testing for screensharing
Remove STT & Turn keeping event duplication

Summary by CodeRabbit

Release Notes

New Features
- Introduced eager end-of-turn detection for faster turn transitions in speech-to-text processing
- Enhanced LLM turn lifecycle management with explicit turn state tracking and completion handling
- Added turn detection capability flag to STT implementations
Improvements
- Refined video track forwarding logic with better priority handling
- Strengthened event system with runtime validation for async handlers
- Improved event messaging for unknown event types
Documentation
- Added documentation on turn detection configuration and eager turn-end event emission
Tests
- Added comprehensive test coverage for agent track handling and participant presence detection

…_test

…equirement for event handlers

… class; add is_global_banned field in StreamEdge class

coderabbitai · 2025-11-11T19:17:07Z

Caution

Review failed

The pull request is closed.

Walkthrough

This PR introduces eager turn detection for vision agents. It relocates AgentOptions to a new types module with TrackInfo and LLMTurn dataclasses, implements pending LLM turn coordination in the Agent class, adds eager turn detection fields to STT and turn detection events, refactors event emission signatures, updates plugin implementations, and adds comprehensive test coverage.

Changes

Cohort / File(s)	Summary
Core types restructuring `agents-core/vision_agents/core/agents/agent_options.py`, `agents-core/vision_agents/core/agents/agent_types.py`	AgentOptions moved from agent_options.py to agent_types.py; new TrackInfo and LLMTurn dataclasses added to agent_types.py with fields for track metadata and LLM turn lifecycle management.
Agent implementation `agents-core/vision_agents/core/agents/agents.py`	Added pending LLM turn handling (_pending_turn, _finish_llm_turn), async event handlers (_on_vad_audio, _on_turn_event), turn_detection_enabled property, enhanced event subscriptions with on_turn_ended handling, and turn-based transcript accumulation logic.
Event system `agents-core/vision_agents/core/events/manager.py`	Added runtime validation to enforce async handlers; improved warning messaging for unknown events during preparation.
STT enhancements `agents-core/vision_agents/core/stt/events.py`, `agents-core/vision_agents/core/stt/stt.py`	Added eager_end_of_turn boolean field to STTTranscriptEvent; added turn_detection attribute and _emit_turn_ended_event method to STT class.
Turn detection `agents-core/vision_agents/core/turn_detection/events.py`, `agents-core/vision_agents/core/turn_detection/turn_detection.py`	Added eager_end_of_turn field to TurnEndedEvent; refactored _emit_end_turn_event signature to accept individual parameters (participant, confidence, trailing_silence_ms, duration_ms, eager_end_of_turn) instead of event object.
Plugin import updates `plugins/moondream/vision_agents/plugins/moondream/detection/moondream_local_processor.py`, `plugins/moondream/vision_agents/plugins/moondream/vlm/moondream_local_vlm.py`, `plugins/smart_turn/vision_agents/plugins/smart_turn/smart_turn_detection.py`	Updated imports of AgentOptions and default_agent_options from agent_options to agent_types; updated _emit_end_turn_event call sites to use keyword arguments.
Plugin STT implementation `plugins/deepgram/vision_agents/plugins/deepgram/deepgram_stt.py`, `plugins/deepgram/tests/test_deepgram_stt.py`	Added turn_detection attribute and eager_turn_detection parameter to Deepgram STT; implemented eager end-of-turn detection via EagerEndOfTurn event; integrated _emit_turn_ended_event calls with eager_end_of_turn flag.
Video forwarder management `plugins/gemini/vision_agents/plugins/gemini/gemini_realtime.py`, `plugins/openai/vision_agents/plugins/openai/rtc_manager.py`	Simplified watch_video_track to remove and reattach handlers instead of stopping/restarting forwarders; added _current_video_forwarder tracking in RTCManager for safe forwarder switching.
Vogent turn detection `plugins/vogent/vision_agents/plugins/vogent/vogent_turn_detection.py`	Removed TurnEndedEvent import; updated _emit_end_turn_event call to use keyword arguments (participant, confidence, trailing_silence_ms, duration_ms).
Minor updates `plugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.py`, `plugins/openai/vision_agents/plugins/openai/openai_realtime.py`	Removed empty custom field from ChannelMemberRequest; updated TODO comments with additional implementation notes.
Documentation and examples `docs/ai/instructions/ai-stt.md`, `docs/ai/instructions/ai-turn-detector.md`, `examples/01_simple_agent_example/simple_agent_example.py`	Added turn keeping documentation section with eager_end_of_turn example; updated turn detector docs to reflect new _emit_end_turn_event signature; updated simple example to use eager_turn_detection and gemini-2.5-flash-lite.
Test coverage `tests/test_agent.py`, `tests/test_agent_tracks.py`	Added comprehensive test suite for Agent.wait_for_participant with mock implementations; added test suite validating track handling, priority-based forwarding, and processor/LLM integration.

Sequence Diagram

sequenceDiagram
    participant STT as STT Module
    participant Agent as Agent
    participant LLM as LLM
    participant TTS as TTS
    
    rect rgb(200, 220, 255)
        Note over STT,TTS: Eager Turn Detection Flow
        STT->>STT: Detect eager end-of-turn (EagerEndOfTurn event)
        STT->>Agent: Emit TurnEndedEvent (eager_end_of_turn=true)
        Agent->>Agent: Store pending LLM turn (_pending_turn)
        Agent->>LLM: Request response
        LLM->>Agent: Return response (async)
        Agent->>Agent: _finish_llm_turn()
        Agent->>TTS: Trigger speech synthesis
    end
    
    rect rgb(220, 255, 220)
        Note over STT,TTS: Normal Turn Detection Flow
        STT->>STT: Detect normal end-of-turn
        STT->>Agent: Emit TurnEndedEvent (eager_end_of_turn=false)
        Agent->>Agent: Process turn completion
        Agent->>LLM: Request response
        LLM->>Agent: Return response
        Agent->>TTS: Trigger speech synthesis
    end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

Agent pending turn logic (agents.py _pending_turn, _finish_llm_turn, event handler coordination): Verify async state transitions and ensure no race conditions in turn completion flow.
Turn detection event refactoring (turn_detection.py signature change): Validate all call sites pass correct parameters and that existing behavior is preserved across all turn detector implementations.
Eager turn detection implementation (deepgram_stt.py): Confirm proper handling of EagerEndOfTurn detection, threshold logic, and event emission with eager_end_of_turn flag.
Video forwarder switching (gemini_realtime.py, rtc_manager.py): Ensure handler removal/reattachment logic prevents frame loss or double-forwarding during transitions.
Event handler async enforcement (events/manager.py): Review validation logic and ensure error messages guide users correctly.

Possibly related PRs

PR #70: Both modify Agent turn/turn-detection and transcript-to-LLM flow with pending turn state and turn-detection emission handling.
PR #157: Both relocate AgentOptions to agent_types.py and update call sites across Moondream and other plugins.
PR #163: Both modify Agent._on_turn_event handling and pass event.participant into LLM response logic.

Suggested labels

tests, turn-detection, eager-turn, events, refactor

Suggested reviewers

yarikdevcom
maxkahan

Poem

Bell jars of silence shatter—
eager turns spring forth unbidden,
pending dreams float through circuits,
while forwarders dance their careful waltz.
The agent breathes in sync,
transcripts bloom like dark flowers.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch simplify_turn_events

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between ec457b9 and 71883ac.

📒 Files selected for processing (23)

agents-core/vision_agents/core/agents/agent_options.py (0 hunks)
agents-core/vision_agents/core/agents/agent_types.py (1 hunks)
agents-core/vision_agents/core/agents/agents.py (9 hunks)
agents-core/vision_agents/core/events/manager.py (2 hunks)
agents-core/vision_agents/core/stt/events.py (1 hunks)
agents-core/vision_agents/core/stt/stt.py (4 hunks)
agents-core/vision_agents/core/turn_detection/events.py (1 hunks)
agents-core/vision_agents/core/turn_detection/turn_detection.py (1 hunks)
docs/ai/instructions/ai-stt.md (1 hunks)
docs/ai/instructions/ai-turn-detector.md (1 hunks)
examples/01_simple_agent_example/simple_agent_example.py (2 hunks)
plugins/deepgram/tests/test_deepgram_stt.py (1 hunks)
plugins/deepgram/vision_agents/plugins/deepgram/deepgram_stt.py (4 hunks)
plugins/gemini/vision_agents/plugins/gemini/gemini_realtime.py (1 hunks)
plugins/getstream/vision_agents/plugins/getstream/stream_edge_transport.py (0 hunks)
plugins/moondream/vision_agents/plugins/moondream/detection/moondream_local_processor.py (1 hunks)
plugins/moondream/vision_agents/plugins/moondream/vlm/moondream_local_vlm.py (1 hunks)
plugins/openai/vision_agents/plugins/openai/openai_realtime.py (1 hunks)
plugins/openai/vision_agents/plugins/openai/rtc_manager.py (2 hunks)
plugins/smart_turn/vision_agents/plugins/smart_turn/smart_turn_detection.py (2 hunks)
plugins/vogent/vision_agents/plugins/vogent/vogent_turn_detection.py (1 hunks)
tests/test_agent.py (1 hunks)
tests/test_agent_tracks.py (1 hunks)

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tschellenbach and others added 14 commits November 8, 2025 10:44

agent testing

a70bfd2

docs for event flow

46b4d8b

fix openai screenshare

3485356

fix gemini

55396f4

test for wait for participant

866cd66

Merge branch 'main' of github.com:GetStream/Vision-Agents into agents…

e289f2a

…_test

eager turn taking

ad6a91f

working eager turn detection

4ecf2a9

wip

5269779

something isn't right with events

681bd04

Refactor VAD audio handler to be asynchronous and enforce coroutine r…

6f2e99c

…equirement for event handlers

Enhance event registration logging and register TurnEndedEvent in STT…

a9855a6

… class; add is_global_banned field in StreamEdge class

merge

79539a8

cleanup and logging

2ae0c43

github-actions bot added agents-core plugins examples docs labels Nov 11, 2025

cleanup

71883ac

tschellenbach marked this pull request as ready for review November 11, 2025 19:42

tschellenbach merged commit f5d37ea into main Nov 11, 2025
5 of 6 checks passed

tschellenbach deleted the simplify_turn_events branch November 11, 2025 19:42

coderabbitai bot mentioned this pull request Nov 17, 2025

Format code with ruff #200

Merged

This was referenced Nov 25, 2025

Silero & Vogent update #214

Merged

[AI-326] Elevenlabs Scribe 2 improvements #216

Merged

This was referenced Jan 15, 2026

Agent HTTP server #284

Merged

Security camera example #247

Merged

Add limits to AgentLauncher #302

Merged

coderabbitai bot mentioned this pull request Feb 6, 2026

Decouple vision agents from getstream #330

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AI-269] Eager turn taking#169

[AI-269] Eager turn taking#169
tschellenbach merged 15 commits intomainfrom
simplify_turn_events

tschellenbach commented Nov 11, 2025 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Nov 11, 2025 •

edited

Loading

Review failed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tschellenbach commented Nov 11, 2025 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Uh oh!

coderabbitai bot commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tschellenbach commented Nov 11, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Nov 11, 2025 •

edited

Loading