Bug Description
While using Deepgram STT with its default turn detection and VAD (without Silero VAD), the user_state is switching from speaking to listening every 2 seconds, even with minor background sound, NOT EVEN NOISE This frequent toggling prevents accurate detection of true user inactivity.
A filler prompt is intended to be triggered when user is away, using livekit internal parameter "user_away_timeout" of 5 seconds. However, due to continuous user_state fluctuations caused by background sound, the system incorrectly interprets these transitions as user activity. As a result, the condition where both agent_state and user_state remain in listening for 5 seconds is not reliably met.
Expected Behavior
Expecting, filler to be triggered every 5 secs, after being agent_state and user_state both in listening.
Reproduction Steps
turn_handling=TurnHandlingOptions(
turn_detection="stt",
),
vad=None #ctx.proc.userdata["vad"],
@session.on("agent_state_changed")
def on_agent_state_changed(ev):
nonlocal _no_input_timer, _no_input_gen
logger.info(f"[{_ts()}][{backend_call_id}] STATE: {ev.old_state} → {ev.new_state}")
raw_event_log.append({
"elapsed": _elapsed(),
"event": "STATE",
"from": str(ev.old_state),
"to": str(ev.new_state),
"timestamp": datetime.now(timezone.utc).isoformat(),
})
if ev.new_state == "listening":
_no_input_gen += 1
gen = _no_input_gen
if _no_input_timer and not _no_input_timer.done():
_no_input_timer.cancel()
async def _fire_no_input(g: int):
await asyncio.sleep(_NO_INPUT_TIMEOUT_S)
if g == _no_input_gen:
logger.info(f"[{_ts()}][{backend_call_id}] NO_INPUT: prompting after {_NO_INPUT_TIMEOUT_S}s silence")
await session.say(_NO_INPUT_PROMPT, allow_interruptions=True)
@session.on("user_state_changed")
def on_user_state_changed(ev):
nonlocal _no_input_timer, _no_input_gen
logger.info(f"[{_ts()}][{call_id}] USER_STATE: {ev.old_state} → {ev.new_state}")
if ev.new_state == "speaking":
_no_input_gen += 1
if _no_input_timer and not _no_input_timer.done():
_no_input_timer.cancel()
_no_input_timer = NoneOperating System
linux
Models Used
deepgram nova-3
Package Versions
livekit-agents ~=1.2 (compatible with 1.2.x)
livekit-plugins-deepgram >=1.3.3
livekit-plugins-elevenlabs >=1.3.3
livekit-plugins-google >=1.3.3
livekit-plugins-openai >=1.3.3
livekit-plugins-noise-cancellation ~=0.2
Session/Room/Call IDs
No response
Proposed Solution
Additional Context
No response
Screenshots and Recordings
No response
Bug Description
While using Deepgram STT with its default turn detection and VAD (without Silero VAD), the user_state is switching from speaking to listening every 2 seconds, even with minor background sound, NOT EVEN NOISE This frequent toggling prevents accurate detection of true user inactivity.
A filler prompt is intended to be triggered when user is away, using livekit internal parameter "user_away_timeout" of 5 seconds. However, due to continuous user_state fluctuations caused by background sound, the system incorrectly interprets these transitions as user activity. As a result, the condition where both agent_state and user_state remain in listening for 5 seconds is not reliably met.
Expected Behavior
Expecting, filler to be triggered every 5 secs, after being agent_state and user_state both in listening.
Reproduction Steps
turn_handling=TurnHandlingOptions( turn_detection="stt", ), vad=None #ctx.proc.userdata["vad"], @session.on("agent_state_changed") def on_agent_state_changed(ev): nonlocal _no_input_timer, _no_input_gen logger.info(f"[{_ts()}][{backend_call_id}] STATE: {ev.old_state} → {ev.new_state}") raw_event_log.append({ "elapsed": _elapsed(), "event": "STATE", "from": str(ev.old_state), "to": str(ev.new_state), "timestamp": datetime.now(timezone.utc).isoformat(), }) if ev.new_state == "listening": _no_input_gen += 1 gen = _no_input_gen if _no_input_timer and not _no_input_timer.done(): _no_input_timer.cancel() async def _fire_no_input(g: int): await asyncio.sleep(_NO_INPUT_TIMEOUT_S) if g == _no_input_gen: logger.info(f"[{_ts()}][{backend_call_id}] NO_INPUT: prompting after {_NO_INPUT_TIMEOUT_S}s silence") await session.say(_NO_INPUT_PROMPT, allow_interruptions=True) @session.on("user_state_changed") def on_user_state_changed(ev): nonlocal _no_input_timer, _no_input_gen logger.info(f"[{_ts()}][{call_id}] USER_STATE: {ev.old_state} → {ev.new_state}") if ev.new_state == "speaking": _no_input_gen += 1 if _no_input_timer and not _no_input_timer.done(): _no_input_timer.cancel() _no_input_timer = NoneOperating System
linux
Models Used
deepgram nova-3
Package Versions
Session/Room/Call IDs
No response
Proposed Solution
Additional Context
No response
Screenshots and Recordings
No response