Feature Request: Add `utterance_end_latency` to `STTMetrics` for streaming STT

[kimdwkimdw]

Feature Type

Would make my life easier

Feature Description

Problem

The current STTMetrics provides audio_duration, duration, and streamed, but the duration field is always 0 for streaming STT as documented:

duration — For non-streaming STT, the amount of time (seconds) it took to create the transcript. Always 0 for streaming STT.

This means there is no responsiveness metric for streaming STT in the current metrics system. Every other pipeline component has one:

Component	Responsiveness Metric
LLM	ttft (Time to First Token)
TTS	ttfb (Time to First Byte)
EOU	end_of_utterance_delay
STT (streaming)	❌ None

For operators running production voice agents, this gap makes it impossible to:

• Monitor streaming STT engine performance over time
• Compare different STT providers (e.g., Deepgram vs. Google vs. AssemblyAI) under the same pipeline conditions
• Alert on STT latency regressions
• Attribute end-to-end latency to specific pipeline stages

Proposed Solution

Add a new field utterance_end_latency to STTMetrics, defined as:

The wall-clock delay from the end of user speech (VAD END_OF_SPEECH) to the arrival of the final transcript (FINAL_TRANSCRIPT).

This is conceptually identical to TTFS (Time To Final Segment) — an industry metric introduced by Pipecat for benchmarking streaming STT in real-time voice agent applications. TTFS has become the de facto standard for evaluating streaming STT responsiveness, with benchmark results published across major providers.

Why utterance_end_latency (TTFS) matters

In a voice agent pipeline, the end-to-end response latency is roughly:

total_latency ≈ STT_latency + eou.end_of_utterance_delay + llm.ttft + tts.ttfb

Today, STT_latency for streaming mode is a black box — it cannot be measured through STTMetrics. Adding utterance_end_latency closes this observability gap and gives operators the full picture.

Workarounds / Alternatives

Implementation

A draft implementation is available in PR #4966:

👉 #4966

Summary of changes

• Adds utterance_end_latency (type: float, default: 0) to the STTMetrics dataclass
• Records _end_of_speech_time on SpeechEventType.END_OF_SPEECH in SpeechHandle
• Computes latency on SpeechEventType.FINAL_TRANSCRIPT as time.time() - _end_of_speech_time
• Emits the value via the existing metrics_collected event — no new events or APIs required
• Fully backward-compatible: the field defaults to 0 and only populates when both events occur

Example output

STT utterance_end_latency=0.342s  (provider=deepgram, model=nova-3, streamed=true)

Additional Context

Related Issues

• How to measure latency properly? #3824 — "How to measure latency properly?" — Users are already trying to piece together STT latency from other metrics
• Metrics Latency Calculation #2522 — "Metrics Latency Calculation" — Earlier request for end-to-end traceability across pipeline stages
• Incorrect transcription_delay when using STT turn detection mode #4388 — "Incorrect transcription_delay when using STT turn detection mode" — Shows the community need for accurate STT timing
• min_endpointing_delay behaves differently in VAD vs STT turn detection mode #4325 — "min_endpointing_delay behaves differently in VAD vs STT turn detection mode" — Suggests adding a "silence-to-commit" metric

Additional Context

• Pipecat's stt-benchmark: https://github.com/pipecat-ai/stt-benchmark — Open-source framework measuring TTFS and semantic WER across STT providers. Pipecat has integrated TTFS as a first-class configuration parameter (ttfs_p99_latency) since v0.0.102.
• Industry adoption: Providers like Speechmatics, Deepgram, and others are actively referencing TTFS as the key latency metric for streaming STT in voice agent contexts.
• LiveKit observability docs: https://docs.livekit.io/deploy/observability/data/#speech-to-text-stt-

[chenghao-mou]

This is available as transcription_delay in the EOU metric.

[kimdwkimdw]

@chenghao-mou

transcription_delay measures turn completion, not pure STT latency, because it still includes EOU/endpointing effects. utterance_end_latency is needed to isolate how quickly STT returns FINAL for the audio that actually produced the transcript.

#4966
Check utterance_end_latency vs transcription_delay section

[chenghao-mou]

transcription_delay is the final utterance latency calculated based on max(last_final_transcript_time - last_speaking_time, 0). It is calculated in the audio_recognition.py file.

We don't emit it for every utterance because there are many factors:

• Not all STT emit end_time properly or accurately. In fact, many of them have some of kind of delay or misalignment as they are inferred from word timestamps.
• Between-utterance delay is acceptable, while the final one for the turn end is more critical for the end2end latency

For all the stated purposes:

• Monitor streaming STT engine performance over time
• Compare different STT providers (e.g., Deepgram vs. Google vs. AssemblyAI) under the same pipeline conditions
• Alert on STT latency regressions
• Attribute end-to-end latency to specific pipeline stages

I think transcription_delay should be sufficient. I previously thought about using cursor diff for STT latency: #2301 but didn't find enough use case for that.

[kimdwkimdw]

Thanks for the thorough response.

Totally fair that transcription_delay works for end-to-end turn latency. The thing I keep running into though is attribution — when transcription_delay spikes, there's no way to tell whether the STT provider was slow to finalize or the endpointing logic added the delay. Both get folded into last_final_transcript_time - last_speaking_time.

In my testing across Deepgram / Google / OpenAI, measuring just END_OF_SPEECH → FINAL_TRANSCRIPT showed pretty clear per-provider differences that transcription_delay alone doesn't surface.

On the timestamp accuracy point — yeah, exact end_time is definitely unreliable across providers. But most providers' timestamps overlap closely enough that a cursor-based approach (like you explored in #2301) would still give a useful-enough signal, even if it's not perfect.

FWIW, the concept of TTFS (Time To Final Segment) is gaining traction in the voice agent space:

• Per-provider P99 defaults are already being tracked (e.g., Deepgram 0.35s, OpenAI Realtime 1.66s)
• Open-source benchmarks across 20+ providers are now treated as an industry reference

To be honest, I've been rebasing my fork on every release just to keep this metric around for STT evaluation, and it's getting pretty tedious. Would really love to see it land upstream, even with partial provider coverage. Happy to adjust #4966 however makes sense.

[chenghao-mou]

TTFS (Time to Final Segment) | Time from user finishing speech to receiving the final segment/transcript. | High. This is the ‘transcription latency’ people mean when talking about voice agents.

According to the page linked, TTFS is defined as "finishing speech" to the final transcript, exactly what we are reporting as transcription delay. The one you are asking for, from what I can see, is more like "Finals Latency"?

I can see the point that you are making. But I think there should be an easier way to do this. I will take a deeper look (but it is currently not a high priority for us)