frontdesk: end-to-end simulation example with dynamic tool mocking#6089
Merged
Conversation
The mock_tools context manager stores mocks in a ContextVar that does not
reliably reach a live session's tool-execution tasks, and the with-block
ergonomics don't fit entrypoints. Add a second call shape,
mock_tools(AgentClass, mocks, session=session), that registers the mocks on
the session immediately and for its lifetime: assign semantics (call again to
replace, pass {} to clear), stored in a WeakKeyDictionary so mocks die with
the session. Tool execution merges both registries, with the context-manager
form taking precedence so existing tests are unchanged.
…c tool mocks The scenario's userdata drives the whole run: available_slots seeds a deterministic FakeCalendar (replacing the random/cal.com calendar), expected_booking grades the run on final calendar state in on_simulation_end, and the agent's tools run mocked through mock_tools targeting the live session. Both mocks close over the same calendar, so booking through the mocked schedule_appointment changes what the mocked list_available_slots returns next; the mock still collects the attendee email via GetEmailTask. All simulation glue lives in simulation.py; scenarios reference absolute dates against a pinned clock (FRONTDESK_NOW). The entrypoint waits for the first participant before resolving simulation_context(), since the simulator may join after the agent. FakeCalendar now records bookings, raises SlotUnavailableError on unknown slots, and uses a seedable RNG. All 10 scenarios pass against LiveKit Cloud (lk agent simulate).
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
AgentSession was only imported under TYPE_CHECKING on the retry branch this was developed against, not on main.
Comment on lines
+48
to
+64
| async def schedule_appointment(ctx: RunContext, slot_id: str) -> str | None: | ||
| if not (slot := slots_map.get(slot_id)): | ||
| raise ToolError(f"error: slot {slot_id} was not found") | ||
|
|
||
| email_result = await beta.workflows.GetEmailTask( | ||
| chat_ctx=ctx.session.current_agent.chat_ctx | ||
| ) | ||
| if ctx.speech_handle.interrupted: | ||
| return None | ||
|
|
||
| ctx.disallow_interruptions() | ||
|
|
||
| await cal.schedule_appointment( | ||
| start_time=slot.start_time, attendee_email=email_result.email_address | ||
| ) | ||
| local = slot.start_time.astimezone(tz) | ||
| return f"The appointment was successfully scheduled for {local:%A, %B %d, %Y at %H:%M}." |
Contributor
There was a problem hiding this comment.
🔍 Simulation mock functions rely on _run_mock parameter trimming working correctly
The simulation mocks in simulation.py declare fewer parameters than the real tools — e.g., list_available_slots() takes no args while the real tool takes self, ctx, range. This works because _run_mock in run_result.py:1086-1120 inspects the mock's signature and trims fnc_args/fnc_kwargs to match. For list_available_slots() with 0 positional params, all args are dropped. For schedule_appointment(ctx, slot_id) with 2 positional params, fnc_args[:2] captures (self_agent, run_ctx) — meaning ctx receives the agent instance and slot_id receives the RunContext, NOT the actual slot_id string. This would be a bug if the mock relied on slot_id being the actual slot ID, but looking at the mock implementation, it uses keyword arguments from fnc_kwargs which includes slot_id from the LLM's parsed arguments. The prepare_function_arguments function at llm/utils.py:584 uses signature.bind(**{...}) which maps named params, so slot_id ends up in kwargs. The trimming of positional args doesn't matter because the real values come through kwargs matching by name.
Was this helpful? React with 👍 or 👎 to provide feedback.
bcherry
approved these changes
Jun 13, 2026
The simulation attributes now ride the agent dispatch and land on the Job itself (livekit/protocol#1629, livekit/livekit#4598), so simulation_context() reads lk.simulator.dispatch from job.attributes first and resolves before the room even connects. The participant attribute scan stays as a fallback for servers that predate job attributes. Requires livekit-protocol>=1.1.17 (first release with Job.attributes). frontdesk: drop the wait_for_participant workaround from the entrypoint.
The dispatch attributes always ride the Job now (agent-service sends them on the RoomAgentDispatch), so the resolution is synchronous and final at job start. The simulator token keeps only the lk.simulator marker, which the SDK still uses to end the job when the simulator disconnects.
The scenarios use absolute dates authored against a fixed baseline (2026-06-12). Previously that only lined up if FRONTDESK_NOW was set in the environment; forgetting it left the calendar on the real clock, so every seeded slot was in the past and got filtered out (the agent saw an empty calendar and could not book). Now the entrypoint pins the clock from the scenario (simulation.SIMULATION_NOW, or a per-scenario userdata 'now' override) as soon as simulation_context() resolves, before anything reads now(). FRONTDESK_NOW still works for non-simulated runs.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Replaces the pin_now module global + FRONTDESK_NOW env var with a plain Calendar.now() method. The FakeCalendar takes a fixed 'now' (the scenario clock) and the agent receives the calendar's clock as now=cal.now, so its sense of 'today' matches the availability it sees. Production / cal.com calendars keep wall-clock now(). No hidden global state or env var: the clock is data flowing through constructors.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Removes userdata.booked_times, a parallel booking list the real tool maintained and the simulation mock silently didn't — so a successful mocked booking was mis-tagged as 'not booked' by on_session_end. Bookings are now recorded only on the Calendar (both FakeCalendar and CalComCalendar track scheduled_appointments), which whoever calls schedule_appointment updates automatically. on_session_end and on_simulation_end now read the same source, and the mock needs no booking bookkeeping at all. Also translate SlotUnavailableError to ToolError in the mock to match the real tool's contract (both Devin findings).
…teps The current date was baked into the system instructions, which changes per session/day and defeats the prompt cache, and it lacked the time of day. It's now a get_current_time tool the model calls on demand: the system prompt stays static (cache-friendly), the time is always current down to the minute, and the mock leaves it alone so it reports the pinned scenario clock under simulation. Drops the max_tool_steps=1 override (back to the SDK default of 3) so a turn can chain get_current_time -> list_available_slots. Switches the LLM to google/gemma-4-31b-it.
…tions # Conflicts: # examples/frontdesk/agent.py # livekit-agents/livekit/agents/voice/run_result.py # livekit-agents/pyproject.toml # uv.lock
Merge brought in main's time-of-day greeting, which read the wall clock directly; route it through self._now() so it respects the pinned simulation clock like the rest of the agent.
The example test constructs FrontDeskAgent(timezone=...) without a clock,
so 'now' defaults to wall-clock (the simulation entrypoint still passes
the calendar's pinned clock). The tests also allow the agent to call
get_current_time before listing availability, since resolving a relative
date ('tomorrow') may prompt that tool call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the frontdesk example a full showcase of agent simulations, alongside
hotel_receptionist:userdata.available_slotsseeds a deterministicFakeCalendarviactx.simulation_context(), replacing the random/cal.com calendar under simulation.on_simulation_endcompares what was actually booked againstuserdata.expected_booking— a specific slot,null(the agent must not book), or omitted (conversation-only grading) — and vetoes the run on mismatch.mock_toolsgains a no-context-manager call shape targeting a liveAgentSession. Both frontdesk mocks close over the same calendar, so booking through the mockedschedule_appointmentremoves the slot from the next mockedlist_available_slots— one scenario asserts exactly that. The mocked booking still collects the attendee email viaGetEmailTask.scenarios.yaml): happy paths plus adversarial callers — unavailable times, refusing all alternatives, an empty calendar, weekend insistence, probing for unlisted near-times. Scenarios use absolute dates against a pinned clock (FRONTDESK_NOW), documented at the top of the file.All simulation glue is isolated in
examples/frontdesk/simulation.py; the agent code stays production-shaped.SDK changes
mock_tools(AgentClass, mocks, session=session)registers mocks on the session directly (the ContextVar of thewithform doesn't reliably reach a live session's tool tasks). Assign semantics — call again to replace,{}to clear — stored in aWeakKeyDictionaryso mocks die with the session. The context-manager form is unchanged and takes precedence;tests/test_tools.pypasses as-is.simulation_context()now resolves fromjob.attributes: the simulation attributes ride the agent dispatch and land on theJobitself (agent: add attributes map to AgentDispatch and Job protocol#1629, agent: thread attributes map from dispatch to job livekit#4598, served by livekit/agents-private#179), so the context is available as soon as the entrypoint runs — nowait_for_participantneeded, and the resolution is final at job start (no participant fallback). The simulator token keeps only thelk.simulatormarker, still used to end the job when the simulator disconnects. Bumpslivekit-protocol>=1.1.17(first release withJob.attributes).Deploy note: livekit/agents-private#179 must be live before this releases — this SDK no longer reads the dispatch from participant attributes.
Verification
lk agent simulate agent.py --scenarios scenarios.yaml: all 10 scenarios pass (run), each job loggingtext simulation: disabling STT/TTS/VAD and audio I/O.tests/test_tools.py(85 passed) for the existing mock path, plus aFakeLLM-driven session run covering session-mock interception, removal via{}, and context-manager precedence.nativesdk): aCreateRoomwithRoomAgentDispatch.attributesdelivered both keys onjob.attributes, andsimulation_context()resolved run/job/scenario beforectx.connect(), with no participants in the room.