feat(stt): back-date START_OF_SPEECH onset via server-provided timestamp

[gsharp-aai]

Summary

When turn_detection="stt" is used alongside a local VAD plugin (e.g. Silero), the framework records _speech_start_time from whichever event handler — VAD or STT — sets it first.

When local VAD fires for the audio, this works fine — the VAD handler back-dates _speech_start_time via time.time() - speech_duration - inference_duration and the existing None-guard prevents the later STT START_OF_SPEECH event from overwriting it.

But when the local VAD does not fire for that audio (different model version, different acoustic threshold, different preprocessing — common at quiet/borderline volumes), _speech_start_time stays None until the STT START_OF_SPEECH arrives — at which point the framework falls back to time.time() at message arrival. If the STT's speech-onset signal and its first transcript arrive close together, the framework's _speech_start_time and _last_speaking_time end up pinned near the same wall-clock instant. Result: MetricsReport.started_speaking_at ≈ stopped_speaking_at, i.e. speech_duration ≈ 0s for any turn the local VAD missed.

This is unphysical (real audio was transcribed), breaks downstream analytics keyed on speech duration, and creates state inconsistency where a user turn commits without ever entering a meaningful "speaking" window.

What this PR changes

Framework

1. Adds an optional speech_start_time: float | None = None field on SpeechEvent. Plugins that receive a separate speech-onset signal with timing can populate it; when left None the framework's STT SOS handler falls back to time.time() at message arrival (its current behavior).
2. Updates the STT SOS handler in audio_recognition.py to set _speech_start_time from ev.speech_start_time only when it's still None (i.e. local VAD hasn't fired first). When VAD has already set it, the VAD-back-dated value is preserved.
3. Extends the RecognitionHooks.on_start_of_speech protocol with a required speech_start_time: float parameter and threads the authoritative onset through it from both SOS handlers. Each handler computes its own authoritative onset locally (VAD back-dates from the VAD event; STT reads _speech_start_time) and passes a concrete value in — no ambiguity about which input wins at call time. All downstream state — DynamicEndpointing._utterance_started_at, _user_speaking_span.start_time, UserStateChangedEvent.created_at — reads from that single value.
4. Simplifies AgentActivity.on_start_of_speech to drop its internal VAD-event back-dating and time.time() fallback, since the caller now always provides the authoritative onset.

AssemblyAI plugin

5. Parses the SpeechStarted.timestamp field (stream-relative ms) that the plugin currently discards, converts it to wall-clock via a _stream_wall_start anchor recorded when the first audio frame is sent, and populates SpeechEvent.speech_start_time on the emitted START_OF_SPEECH event.

Why this is a safe fallback

Strictly additive. Every turn where local VAD fires is unaffected — _speech_start_time is already set by the VAD handler (its back-date is more accurate, computed locally with no network delay), the None-guard preserves it, and the same value flows through the hook to every downstream consumer. The STT-provided timestamp is only consulted when _speech_start_time is still None at STT START_OF_SPEECH arrival, i.e. exactly the case where local VAD missed the audio the STT caught.

Provider-side fallback (if you'd prefer not to add the field)

If a SpeechEvent schema change isn't desirable, the same outcome can be achieved without touching the framework: a plugin can pass through the back-dated time on the existing SpeechData.start_time field by attaching a synthetic SpeechData to the START_OF_SPEECH event's alternatives list. Functionally equivalent but semantically off (SpeechData is meant for transcription hypotheses, not event metadata), so the explicit field is preferred. Happy to switch if maintainers prefer to defer the schema change.

Files changed

• livekit-agents/livekit/agents/stt/stt.py — new optional field on SpeechEvent
• livekit-agents/livekit/agents/voice/audio_recognition.py — tighten RecognitionHooks.on_start_of_speech to require speech_start_time: float; set _speech_start_time from ev.speech_start_time under the None-guard; pass self._speech_start_time to the hook at the STT SOS call site; pass the locally-computed back-date at the VAD SOS call site
• livekit-agents/livekit/agents/voice/agent_activity.py — accept the required speech_start_time kwarg on AgentActivity.on_start_of_speech; internal fallback logic removed
• livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py — anchor _stream_wall_start on first frame; parse SpeechStarted.timestamp and populate SpeechEvent.speech_start_time

Test plan

• make format lint type-check pass
• Unit/integration coverage for the new field — happy to add on request

[tinalenguyen]

hi @gsharp-aai, thank you for the PR! it seems like this is a specialized framework change for the plugin, is the SpeechStarted event's latency expected behavior? i would lean towards plugin-side logic to mitigate this, let me know your thoughts

[gsharp-aai]

Hey @tinalenguyen! Thanks for the comment. Just to separate concerns clearly:

Framework change
Our API, along with a couple of others (OpenAI Realtime's input_audio_buffer.speech_started carries audio_start_ms; Speechmatics' StartOfTurn can also include timing), sends timestamp data alongside its start-of-speech signal (SpeechStarted for us).

Currently there's no way for a plugin to pass this information through on the START_OF_SPEECH event. The framework change proposed here is to trust a speech_start_time if provided by the plugin, and otherwise fall back to the message-arrival time (today's behavior). But ultimately would still be on the provider to pass this data through if they want it to be trusted.

Plugin change

is the SpeechStarted event's latency expected behavior

Yes — SpeechStarted on our end has some inherent latency. We gate SpeechStarted behind the first partial transcript, a deliberate server-side choice to prevent phantom events on noise that does not end up being valid speech. This typically isn't an issue when local VAD fires, since self._speech_start_time is already set by the VAD handler and the existing None-guard preserves it. The issue shows up specifically when local VAD does not detect speech but our server does, and a transcript is received.

Because SpeechStarted and the final transcript may arrive close together in that case, the framework's time.time() calls at STT SOS and STT EOS land at nearly the same wall-clock instant, and started_speaking_at ≈ stopped_speaking_at — resulting in MetricsReport.speech_duration ≈ 0. To avoid this, we'd like to use the server-provided onset from SpeechStarted.timestamp to back-date _speech_start_time in that fallback case. VAD-fires-first turns are unaffected.

———

Overall, definitely open to plugin-side logic here! But I believe there would need to be at least some code on the framework side to let plugins propagate onset timing through START_OF_SPEECH events? Happy to iterate on whatever shape the team would prefer.

[tinalenguyen]

Thank you for the context @gsharp-aai, that makes a lot of sense. To ensure accuracy for user_speaking spans, I'm open to adding that field, I'll get back to you on what the rest of the team thinks

[tinalenguyen]

we can add speech_start_time to SpeechEvent, i just added a small comment. otherwise everything looks good to me!

[tinalenguyen]

maybe we can add a condition where:

self._speech_start_time = ev.speech_start_time if ev.speech_start_time < self._speech_start_time else self._speech_start_time

for when the vad detects activity before the stt as well

[gsharp-aai]

Open to this! Just want to flag it changes behavior from the current PR. Two shapes:

1. Fallback only (current PR as-is): _speech_start_time is only set from STT when VAD hasn't already set it. VAD wins when it fires, preserving current behavior.
2. Earlier of VAD or STT (your suggestion): every STT SOS compares both and picks the earlier onset, even when VAD already fired.

I leaned toward #1 since local VAD's back-date is usually more accurate than the server timestamp (no network delay, no clock skew) plus less of a behavioral change (in relation to what currently exists), but happy to flip to #2 if you think the "STT caught it earlier" case is common enough to trust by default.

Let me know which shape the team prefers!

[chenghao-mou]

we have a field called start_time_offset in stt stream that plays a similar role, and it is assigned when the stream is initialized:

stream.start_time_offset = time.time() - _audio_input_started_at

I think we can add a second field stream.start_time so that other STT implementations can use it as well.

[gsharp-aai]

Good call!

Key consideration is that I believe "server-provided onset timestamp" would be anchored to whatever zero-point that provider defines, which will of course vary by each provider's sever-side implementation. Because of that, I was thinking that the framework can't reliably pin a single wall-clock moment that aligns with every provider's "zero" simultaneously (each plugin knows its own server's semantics and should probably own the anchoring moment).

What about putting the field on the base class (shared, discoverable, other plugins can adopt), seeding a framework default at init so plugins that don't override still get some value, and letting each plugin overwrite it at whatever moment corresponds to its own server's zero? The framework can handle resetting it on retries centrally, same pattern as start_time_offset.

Shape:

# base class SpeechStream
self._start_time: float = time.time()   # framework default

@property
def start_time(self) -> float: ...

@start_time.setter
def start_time(self, value: float) -> None: ...
# Plus a reset in _main_task across retries, same pattern as start_time_offset.

What do you think?

Edit: updated to seed a framework default and let plugins overwrite it, instead of leaving it as purely plugin-set.

[chenghao-mou]

That sounds reasonable. The framework provides a default, and plugins can override it if needed.

[gsharp-aai]

The framework provides a default, and plugins can override it if needed.

Updated PR to reflect this

[gsharp-aai]

@tinalenguyen @chenghao-mou Thank you for the reviews!

I have updated the PR based on this feedback, plus added a few tests. Ready for re-review when the team is able. Thank you!