Google STT: Add voice_activity_timeout support for faster endpointing (Chirp 3 has 2.4s latency)

[AhmadIbrahiim]

Feature Type

I cannot use LiveKit without it

Feature Description

Google STT (especially Chirp 3) has ~2.4s latency on short utterances like "hi" or "yes" because the plugin doesn't expose Google's voice_activity_timeout configuration.

Google Cloud Speech-to-Text V2 API supports StreamingRecognitionFeatures.VoiceActivityTimeout with:

• speech_start_timeout - How long to wait for speech to begin
• speech_end_timeout - How much silence before finalizing (min 500ms, max 60s)

Without this, Google STT is impractical for real-time voice agents. Users expect sub-second responses for short utterances, but Chirp 3 waits ~2.4s before returning the transcript.

Reference: https://docs.cloud.google.com/speech-to-text/docs/voice-activity-events

Workarounds / Alternatives

1. Switch to Deepgram (has eot_threshold) - but lose Chirp 3's accuracy
2. Use latest_short or telephony models - but they also lack timeout config
3. PR Add google stt voice activity timeout #4361 adds this feature - pending review

Additional Context

Measured latency for "hi" with Chirp 3: 2468ms
Expected with speech_end_timeout=0.5: ~500-800ms

PR implementing this fix: #4361

[ali-boost]

Hey - I want to report something that baffled me.

There is an issue here with these new fields, i think they are misunderstood - they are controlling the stream and not the VAD.

speech_start_timeout | google.protobuf.duration_pb2.Duration Duration to timeout the stream if no speech begins. If this is set and no speech is detected in this duration at the start of the stream, the server will close the stream.

speech_end_timeout | google.protobuf.duration_pb2.Duration. Duration to timeout the stream after speech ends. If this is set and no speech is detected in this duration after speech was detected, the server will close the stream.

https://docs.cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.StreamingRecognitionFeatures.VoiceActivityTimeout

Using the example from the PR:

stt = google.STT(
    model="chirp_3",
    speech_start_timeout=10.0,
    speech_end_timeout=0.5,  # 500ms silence = end of turn
)

speech_start_timeout=10.0 --> stream will close if no speech in the first 10s
speech_end_timeout=0.5 --> stream will close if 0.5s passed since last speech

PR is functionally fine, it's the documentation and what it says about the latency is the main issue.

[xzneozx96]

Hey - I want to report something that baffled me.

There is an issue here with these new fields, i think they are misunderstood - they are controlling the stream and not the VAD.

speech_start_timeout | google.protobuf.duration_pb2.Duration Duration to timeout the stream if no speech begins. If this is set and no speech is detected in this duration at the start of the stream, the server will close the stream.
speech_end_timeout | google.protobuf.duration_pb2.Duration. Duration to timeout the stream after speech ends. If this is set and no speech is detected in this duration after speech was detected, the server will close the stream.

https://docs.cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.StreamingRecognitionFeatures.VoiceActivityTimeout

Using the example from the PR:

stt = google.STT(
    model="chirp_3",
    speech_start_timeout=10.0,
    speech_end_timeout=0.5,  # 500ms silence = end of turn
)

speech_start_timeout=10.0 --> stream will close if no speech in the first 10s speech_end_timeout=0.5 --> stream will close if 0.5s passed since last speech

PR is functionally fine, it's the documentation and what it says about the latency is the main issue.

Agree, it seem that the once speech_end_timeout is fired, it closes the entire stream mid-sentence if set too short:
chirp_3 + speech_end_timeout + VAD Stream dies after first turn

[AhmadIbrahiim]

PR is functionally fine, it's the documentation and what it says about the latency is the main issue.

I can update the documentation -- Sorry for the inconvenience

[AhmadIbrahiim]

Fixed here: #5219