feat(metrics): port playbackLatency metric from python (#5524)#1323
feat(metrics): port playbackLatency metric from python (#5524)#1323toubatbrian merged 3 commits intomainfrom
playbackLatency metric from python (#5524)#1323Conversation
Ports livekit/agents#5524 to TypeScript. Adds a new `playbackLatency` field on `MetricsReport` measuring the delay (in seconds) between forwarding the first audio frame and the `AudioOutput` reporting that playback started. - `_AudioOut` tracks `startedForwardingAt` (ms) inside `forwardAudio` - pipeline-reply and tts-say paths in `AgentActivity` capture `audioOut.startedForwardingAt` in their `onFirstFrame` callback and populate `playbackLatency = (startedSpeakingAt - startedForwardingAt) / 1000` on the assistant `ChatMessage` metrics. Near-zero for the default room output; meaningful when a remote avatar worker is in the chain and reports playback via `lk.playback_started` RPC.
🦋 Changeset detectedLatest commit: 1a154ea The changes in this PR will be included in the next version bump. This PR includes changesets to release 26 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 1 potential issue.
⚠️ 1 issue in files not directly in the diff
⚠️ playbackLatency metric silently dropped during telemetry trace serialization (agents/src/telemetry/traces.ts:302-311)
The PR adds playbackLatency to the MetricsReport interface (agents/src/llm/chat_context.ts:102) and computes it in agent_activity.ts, but does not update the telemetry serialization in agents/src/telemetry/traces.ts. The ProtoMetricsReport interface (traces.ts:302-311) is missing a playbackLatency field, and the chatItemToProto serialization logic (traces.ts:384-409) never copies it. Every other field in MetricsReport is faithfully mapped in ProtoMetricsReport and serialized — playbackLatency is the only one silently dropped, so the metric will never appear in OpenTelemetry traces.
View 3 additional findings in Devin Review.
Without this, the metric is silently dropped when ChatItems are serialized into OTel traces. Addresses Devin Review finding on PR #1323.
|
Verified locally and on observability dashboard. |
4 participants
Summary
Ports livekit/agents#5524 ("feat(metrics): add playback_start_latency metric") to agents-js.
Adds a new
playbackLatencyfield onMetricsReport(and therefore on assistantChatMessage.metrics) measuring the delay between the first audio frame being forwarded to theAudioOutputand the output reporting that playback actually started.This metric is typically near-zero in the default pipeline (the room output self-reports
playbackStartedwhen it pushes the first frame to the track and so does not account for network delivery to the client), and becomes significant when a remote avatar worker is in the chain — in that scenario the avatar reports playback via thelk.playback_startedRPC and the latency reflects the avatar/network leg.This is an automated port created by the Claude Code routine triggered by @toubatbrian. cc @toubatbrian @livekit/agent-devs as reviewers.
Changes
agents/src/llm/chat_context.tsplaybackLatency?: number(seconds) toMetricsReport. Documented with a tsdoc block matching the Python field's docstring.agents/src/voice/generation.tsstartedForwardingAt?: number(ms —Date.now()) to the_AudioOutinterface.forwardAudio, setsout.startedForwardingAt = Date.now()the first time a frame is appended toout.audio. Mirrors the Python_audio_forwarding_taskchange (out.started_forwarding_at = time.time()at the same point in the loop).agents/src/voice/agent_activity.tsTwo replyKind paths populate
assistantMetrics; both are updated:ttsTask(thesay()path) —onFirstFramenow also capturesreplyStartedForwardingAtfromaudioOut.startedForwardingAt(or falls back toreplyStartedSpeakingAtfor the text-only / no-audio case).replyAssistantMetrics.playbackLatency = (replyStartedSpeakingAt - replyStartedForwardingAt) / 1000is populated alongsidestartedSpeakingAt/stoppedSpeakingAt._pipelineReplyTaskImpl(LLM → TTS pipeline) — same treatment viaagentStartedForwardingAtand aplaybackLatencyentry onassistantMetrics.Each touched location has an inline
// Ref: python livekit-agents/livekit/agents/voice/<file>.py - <range> linescomment as required byCLAUDE.md.Implementation nuances (where exact code-level parity is not possible)
Time units. Python tracks all timestamps as seconds-floats (
time.time()); JS in this repo uses milliseconds (Date.now()) by convention (perCLAUDE.md)._AudioOut.startedForwardingAtis therefore stored in ms, and we divide the(startedSpeakingAt - startedForwardingAt)difference by1000when writing it intoMetricsReport.playbackLatency(which, per the existingstartedSpeakingAt/stoppedSpeakingAt/e2eLatencyfields, is in seconds). End result:playbackLatencyis in seconds, matching Python'splayback_latency._on_first_framereceivesaudio_out. Python upgrades the realtime path's callback to take theaudio_outviafunctools.partial(_on_first_frame, audio_out=audio_out). JS already createsonFirstFrameas a closure inside the relevant scope, so the equivalent change is just a parameter (audioOut: _AudioOut | null) — nopartial-equivalent needed. The text-only branch passesnull.Realtime generation path is intentionally not touched. In Python,
_realtime_generation_task_implalready populatesassistant_metrics["started_speaking_at"]etc. and the diff just addsplayback_latencyto that block. In JS, the realtime generation task (_realtime_generation_task_impl/realtimeReplyTaskImpl) does not currently trackstarted_speaking_at/stopped_speaking_aton the assistantChatMessageat all (theChatMessages are created without ametricsfield — seeagent_activity.ts:2754and:2781). AddingplaybackLatencythere in isolation would be misleading without the surrounding metrics. This pre-existing gap is unrelated to PR #5524 and is left as a separate follow-up.Example port skipped. The Python diff updates
examples/avatar_agents/audio_wave/agent_worker.pyto log the metric. agents-js doesn't carry an equivalent avatar example in this repo, and the change in Python is illustrative (@session.on("conversation_item_added")already exists in JS, so users can observe the new field with the same pattern).Test plan
pnpm build:agents— cleanpnpm lint --filter=@livekit/agents— only pre-existing warnings, no new onespnpm test agents/src/llm/chat_context— 39/39 passpnpm test agents/src/voice— 136/136 passpatchbump on@livekit/agents)restaurant_agent.ts/realtime_agent.tsagainst the Agent Playground and verifymetrics.playbackLatencyappears on assistant messages — recommend a reviewer with playground access do this; my port runs in an isolated environment without LiveKit credentials.Reviewers
cc @toubatbrian @livekit/agent-devs
Generated by Claude Code