feat: agent-simulation support#2075
Merged
Merged
Conversation
🦋 Changeset detectedLatest commit: 39349a8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
….50.2 The protocol package now re-exports SimulationDispatch, Scenario, and SimulationMode from its root, so drop the hand-rolled protojson decoding in favor of SimulationDispatch.fromJsonString.
Preserve the simulation tests and adopt main's protocol dependency and telemetry coverage while resolving conflicts. Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve dynamic text simulation behavior and expose the generated protocol surface so the JS port matches Python 1.6.6, including finalize error and veto handling. Co-authored-by: Cursor <cursoragent@cursor.com>
Expose a finalize-specific error carrying the agent verdict so callback failures remain visible without dropping a previously recorded veto. Co-authored-by: Cursor <cursoragent@cursor.com>
Preserve the concurrent framework-info API alongside connected finalize transport coverage and typed veto errors. Co-authored-by: Cursor <cursoragent@cursor.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Keep both simulator attribute keys in the public types module so participant and dispatch handling cannot drift between duplicate declarations. Co-authored-by: Cursor <cursoragent@cursor.com>
Use the module-scope job context import because the reverse dependency is type-only and does not create a runtime cycle. Co-authored-by: Cursor <cursoragent@cursor.com>
toubatbrian
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of the Python 1.6.6 simulation surface so JS agents are simulatable:
JobContext.simulationContext()resolves the scenario dispatch from the job'slk.simulator.dispatchattributelk.simulator) leavesdefineAgent({ onSimulationEnd })runs on the session host'sfinalizeSimulationrequest and can veto a pass viactx.fail()ScenarioGroup,SimulationRun,SimulationRun_Job,simulationJob, and client-sideRemoteSession.finalizeSimulation()use the generated@livekit/protocolsurfaceFinalizeSimulationErrorpreserves a generateduserVerdictrecorded before a callback failure while still propagating the callback errortypes.tsgetJobContextis imported at module scope; the reverse dependency is type-only, so no runtime cycle requires an inline dynamic importPython source parity
The authoritative release is
livekit-agents@1.6.6(e731eac7c). Actual fetched source patches covered the simulation foundation #5992 (5d427a497), #6014 (82f02c73d), and #6018 (2d2adbef2); the consolidated simulation glue #5688 (e41b88082); follow-ups #6037 (272a2bb34), #6036 (07761a97e), #6053 (1e2405943), #6055 (a236c5320), #6061 (9e5be77d0), and #6069 (20397c8ec); and example coverage #6089 (52a9881be). The JS port uses the framework's existing AsyncLocal job-context and generated Buf classes; Python-only process/decorator machinery is not copied. As in Python 1.6.6,simulationJobis public but remains unset during finalize because the current finalize protocol carries the run ID and verdict, not a simulation-job payload.Python's generic
_send_request()currently has the same fail-then-throw client bug: it raises onresponse.errorbeforefinalize_simulation()can readuser_verdict. Per the correctness-priority requirement, JS fixes this with a finalize-specific typed error rather than copying the bug; generic request error handling remains unchanged.Focused simulation-protocol proof
QCLI cannot synthesize
lk.simulator.dispatch, create a participant carryinglk.simulator, or issue the finalize-simulation remote protocol, so these focused tests are authoritative for those boundaries:agents/src/job.test.ts: dispatch attribute decoding, malformed/missing dispatch behavior, and simulator-participant disconnect shutdownagents/src/voice/remote_session.test.ts: a genuinely connected in-memory transport pair runs the publicRemoteSessionclient against a realSessionHost; it proves normal finalize, callback-error propagation, fail-then-throw preservation onFinalizeSimulationError.userVerdict, and unchanged generic request errorsagents/src/voice/agent_session_text_only.test.ts: dynamic text/audio resolution, audio-model gating, andrecordingOptions.audio = falseagents/src/simulation.test.ts: generated scenario/run/job exports andsimulationJoblifecycleVerification on
39349a8468c0f7519c0436c654d3c32ff36b06bf: 50 focused tests passed, core typecheck passed, scoped lint and formatting passed, and fullpnpm buildpassed. The workspace-wide API check remains blocked by pre-existing tooling issues:plugins/liveavatarhas noapi-extractor.json, and core API Extractor 7.43 cannot parse the merged main entrypoint'sexport * asdeclaration.Cue voice E2E
Secondary regression smoke only; this does not prove simulation finalization (the connected client-host tests above do).
39349a8468c0f7519c0436c654d3c32ff36b06bfsid_35821b106dc4, ordinary--mode voiceuser_input_transcribed(.is_final=true) -> agent_state_changed(.new_state="AS_SPEAKING") -> conversation_item_added(.item.message.role="ASSISTANT")[20s]I can hear you and will answer normally for the static import exact head voice session.)deepgram/nova-3STT,openai/gpt-4.1-miniLLM,cartesia/sonic-3TTS, andturn-detector-v1EOT~/.cue-cli/sessions/sid_35821b106dc4/recording.wav(stereo PCM, 48 kHz, 24.445 s) andrecordings/002_run/recording.wav(stereo PCM, 48 kHz, 10.715 s), plus command/session JSON and the complete event stream