Summary
Text-mode simulations (lk agent simulate --scenarios …) intermittently fail
with Agent did not respond within 60.0s on turn N even though the captured
transcript for the same run shows a complete, well-formed conversation —
including the agent's reply on the allegedly silent turn, the final booking
confirmation, and the farewell. The agent-side log shows no pending work at
that moment: the session closes with reason: user_initiated, and the session
report uploads normally.
Environment
- livekit-agents 1.6.7 (Python 3.14), plugins: openai/azure 1.6.7
- LiveKit CLI 2.18.2, LiveKit Cloud project (eu)
- Agent architecture: single
Agent, all tools are strict single-turn
function tools (no AgentTasks, nothing outlives a turn); LLM is Azure
OpenAI chat with parallel_tool_calls=False
- Simulations run in text mode (no STT/TTS/VAD)
Observed behaviour
Across ~20 runs on two days (12-scenario and 15-scenario suites, run both
locally spawned and cloud-to-cloud):
- 3–11 of the jobs per run fail with
Agent did not respond within 60.0s on turn N; N varies uniformly (turns 3–11) and is not correlated with any
scenario, tool, or prompt. Worst observed: 11 of 15 jobs failed this way
in a single run with --concurrency 2 and zero provider rate-limit
errors, while every one of the 15 transcripts was complete and every
end-state assertion passed.
- For every such failure, the transcript in the same output contains the
agent's response on turn N and the conversation continues to a clean end.
- The agent worker log for the affected room shows normal
conversation_item_added events, then session closed {"reason": "user_initiated"} and a successful report upload — no error,
no hung task, no retry.
- The same suites pass their end-state assertions (bookings/requests created
via on_simulation_end) in every completed conversation.
What we ruled out
- Long-running tools: our final architecture was rebuilt so no tool spans a
turn, after reading remote_session.py — we understand the contract as
"the simulator calls interrupt(force=True) before each user turn and
waits for session.run() to complete". Our agent satisfies this.
- Model latency: the failures also occur on turns whose reply is visible in
the transcript within seconds.
- Rate limits: runs without any 429s still show the same distribution.
Expected
A turn that the transcript records as answered should not be scored as
"did not respond within 60.0s".
Repro
We can share scenario files and full CLI output/agent logs privately if
helpful. A minimal public repro is hard because the failures are
probabilistic (~25–40 % of jobs per run) — happy to run instrumented builds.
Summary
Text-mode simulations (
lk agent simulate --scenarios …) intermittently failwith
Agent did not respond within 60.0s on turn Neven though the capturedtranscript for the same run shows a complete, well-formed conversation —
including the agent's reply on the allegedly silent turn, the final booking
confirmation, and the farewell. The agent-side log shows no pending work at
that moment: the session closes with
reason: user_initiated, and the sessionreport uploads normally.
Environment
Agent, all tools are strict single-turnfunction tools (no AgentTasks, nothing outlives a turn); LLM is Azure
OpenAI chat with
parallel_tool_calls=FalseObserved behaviour
Across ~20 runs on two days (12-scenario and 15-scenario suites, run both
locally spawned and cloud-to-cloud):
Agent did not respond within 60.0s on turn N; N varies uniformly (turns 3–11) and is not correlated with anyscenario, tool, or prompt. Worst observed: 11 of 15 jobs failed this way
in a single run with
--concurrency 2and zero provider rate-limiterrors, while every one of the 15 transcripts was complete and every
end-state assertion passed.
agent's response on turn N and the conversation continues to a clean end.
conversation_item_addedevents, thensession closed {"reason": "user_initiated"}and a successful report upload — no error,no hung task, no retry.
via
on_simulation_end) in every completed conversation.What we ruled out
turn, after reading
remote_session.py— we understand the contract as"the simulator calls
interrupt(force=True)before each user turn andwaits for
session.run()to complete". Our agent satisfies this.the transcript within seconds.
Expected
A turn that the transcript records as answered should not be scored as
"did not respond within 60.0s".
Repro
We can share scenario files and full CLI output/agent logs privately if
helpful. A minimal public repro is hard because the failures are
probabilistic (~25–40 % of jobs per run) — happy to run instrumented builds.