Skip to content

fix(livekit-agents): forward the STT hooks through MultiSpeakerAdapter - #6679

Merged
longcw merged 3 commits into
livekit:mainfrom
LHMQ878:fix/multi-speaker-adapter-forward-stt-hooks
Aug 4, 2026
Merged

fix(livekit-agents): forward the STT hooks through MultiSpeakerAdapter#6679
longcw merged 3 commits into
livekit:mainfrom
LHMQ878:fix/multi-speaker-adapter-forward-stt-hooks

Conversation

@LHMQ878

@LHMQ878 LHMQ878 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

stt.MultiSpeakerAdapter advertises the wrapped STT's capabilities but forwards none of the framework hooks those capabilities gate.

__init__ passes the wrapped recognizer's capabilities through unchanged:

super().__init__(capabilities=stt.capabilities)
self._stt = stt

So when the wrapped STT sets keyterms=True / chat_context=True — both shipped diarization plugins do (deepgram, assemblyai) — the adapter claims them too. The framework then pushes keyterms and conversation items at the adapter, prewarms it, and reads model/provider off it for metrics. MultiSpeakerAdapter defined none of those, so all of it landed on the STT base class and went nowhere.

The drop is silent precisely because the capability is advertised: the base _update_session_keyterms / _push_conversation_item warn-and-skip paths only fire when the capability is False.

stt/stream_adapter.py — the other STT wrapper in the same package — forwards all seven, which is what makes this look like an oversight rather than a deliberate difference:

forwarded to the wrapped STT StreamAdapter MultiSpeakerAdapter
wrapped_stt (stream_adapter.py:38) yes no
model / provider (:42, :46) yes no
_update_session_keyterms (:50) yes no
_push_conversation_item (:53) yes no
prewarm (:81) yes no
metrics_collected re-emit (:36, :84) yes no
aclose detach (:87) yes no

Measured on bbf163f

A diarization-capable recording STT that also supports keyterms and chat context, through the adapter vs. called directly:

MultiSpeakerAdapter wrapped STT directly
capabilities.keyterms True True
capabilities.chat_context True True
keyterms the recognizer received None ['livekit', 'webrtc']
conversation items the recognizer received 0 1
recognizer prewarmed False True
model 'unknown' 'recording-model-v1'
provider 'unknown' 'recording-provider'
metrics reaching a listener on the adapter 0 1

And through the real framework path rather than a direct hook call — KeytermDetector, which AgentActivity binds to the session's STT:

detector = KeytermDetector(static_keyterms=["LiveKit", "WebRTC", "Deepgram"])
detector.start(session, stt=adapter)

  wrapped recognizer keyterms = None
  expected                    = ['LiveKit', 'WebRTC', 'Deepgram']

detector.set_static_keyterms([..., "Cartesia"])   # mid-call update
  wrapped recognizer keyterms = None

So enabling MultiSpeakerAdapter on a Deepgram or AssemblyAI STT silently turns off keyterm boosting and context carryover for that session, and mislabels the STT in metrics as unknown/unknown.

Change

Forward the same seven members StreamAdapter does, in the same shape.

Two deliberate details, both copied from the siblings rather than invented here:

  • aclose only detaches the metrics handler and does not close the wrapped STT — matching StreamAdapter.aclose and stt.FallbackAdapter.aclose. The caller constructed the inner STT and owns its lifecycle.
  • Forwarding the inner metrics means suppressing the adapter'''s own, or the same audio gets measured twice. StreamAdapterWrapper makes _metrics_monitor_task a no-op (stream_adapter.py:107) and stt.FallbackAdapter sets _recognize_metrics_needed = False (fallback_adapter.py:111); this does both. Measured 2 events per recognition on both the recognize() and stream() paths with the forwarding alone, 1 with the suppression. The kept event is the wrapped plugin'''s, which carries its label and its real model/provider metadata.
  • The metrics override still has to reset the retry counter. A bare no-op monitor — the shape the siblings use — also drops the _num_retries = 0 the base does on each final transcript, and _main_task gives up once that exceeds max_retry. Measured with an inner stream that transcribes then drops the connection every attempt (max_retry=3): with the bare no-op, four good transcripts still left _num_retries at 3 and the stream died with failed to recognize speech after 3 attempts; with the reset kept, recognition continued. StreamAdapterWrapper is safe with a bare no-op only because it builds itself with max_retry=0 (stream_adapter.py:16, :101) so the counter is never consulted; MultiSpeakerAdapterWrapper passes the caller'''s conn_options through, so its budget is live. I verified StreamAdapter does not accumulate under the same pattern.

Test

tests/test_multi_speaker_adapter.py (new module, pytestmark = pytest.mark.unit), 10 tests — one per forwarded member, one that drives the real KeytermDetector end to end instead of calling the hook directly, two that pin the metrics count on each path, and one that pins the retry-counter reset.

The 7 delegation tests confirmed red on the unmodified tree before being kept:

FAILED test_wrapped_stt_exposed - AttributeError: 'MultiSpeakerAdapter' object has no attribute 'wrapped_stt'
FAILED test_keyterms_reach_the_wrapped_stt - assert [] == [['LiveKit', 'WebRTC']]
FAILED test_keyterm_detector_reaches_the_wrapped_stt - assert [] == [['Acme']]
FAILED test_conversation_items_reach_the_wrapped_stt - assert [] == ['your room is booked']
FAILED test_prewarm_reaches_the_wrapped_stt - assert False
FAILED test_model_and_provider_come_from_the_wrapped_stt - assert ('unknown', 'unknown') == ('recording-model-v1', 'recording-provider')
FAILED test_metrics_are_forwarded_and_detached_on_aclose - assert [] == [STTMetrics(...)]

The 2 metrics tests were confirmed red at the intermediate state (forwarding without suppression): AssertionError: one usage report, 2 metrics events. The retry test was confirmed red with a bare no-op monitor: AssertionError: recognition died after 4 good transcripts.

MultiSpeakerAdapter itself had no coverage before this — tests/test_multi_speaker_primary_speaker.py exercises the private _PrimarySpeakerDetector only, never the adapter.

Verification

  • ruff check and ruff format --check on both files — clean.
  • mypy (repo config, strict) on livekit.agents.sttSuccess: no issues found in 5 source files.
  • pytest --unit: I could not install the full workspace on this machine — bithuman 2.7.0 publishes no Windows wheel, so uv sync --all-extras --dev fails outright. I ran the suite against an editable livekit-agents instead, which leaves 16 modules erroring on absent plugin packages. Captured the sorted FAILED/ERROR set with the change (and the new test file) removed and restored: identical, 20 entries both ways (16 collection errors + 4 pre-existing failures in test_ivr_activity.py and test_tokenizer_xml_markup.py, unrelated). Passing count 1557 → 1567, i.e. exactly the 10 new tests.

I did not touch CHANGELOG.md or any package manifest, per CONTRIBUTING.

MultiSpeakerAdapter wraps another STT and passes `capabilities=stt.capabilities`
straight to `super().__init__`, so it advertises whatever the wrapped recognizer
supports -- including `keyterms` and `chat_context`, which both shipped
diarization plugins (deepgram, assemblyai) set. But it forwarded none of the
hooks those capabilities gate, so everything the framework pushed at it was
dropped, and dropped silently: the base `_update_session_keyterms` /
`_push_conversation_item` warn-and-skip only fires when the capability is False,
and here it is True.

Measured against the wrapped STT called directly: keyterms received None instead
of the pushed list, 0 conversation items instead of 1, prewarmed False,
model/provider "unknown" instead of the plugin's values, and 0 metrics reaching
a listener on the adapter. Driving the real KeytermDetector reproduces it end to
end -- `detector.start(session, stt=adapter)` with static keyterms leaves the
wrapped recognizer's keyterms unset, and so does a mid-call
`set_static_keyterms`.

Forward the seven members `stt/stream_adapter.py:38-88` already forwards, in the
same shape: `wrapped_stt`, `model`, `provider`, `_update_session_keyterms`,
`_push_conversation_item`, `prewarm`, the `metrics_collected` re-emit, and an
`aclose` that detaches it. `aclose` deliberately does not close the wrapped STT,
matching StreamAdapter and stt.FallbackAdapter -- the caller constructed it and
owns it.
@LHMQ878
LHMQ878 requested a review from a team as a code owner August 3, 2026 11:52
devin-ai-integration[bot]

This comment was marked as resolved.

Re-emitting the wrapped STT's `metrics_collected` in the previous commit made the
adapter forward the inner measurement while still producing its own for the same
audio, so every recognition was reported twice -- once via `STT.recognize`, once
via `RecognizeStream._metrics_monitor_task`. Measured 2 events per recognition on
both paths where there should be 1.

Both sibling wrappers pair the forwarding with suppressing their own: the stream
adapter's wrapper makes `_metrics_monitor_task` a no-op
(`stt/stream_adapter.py:107`) and `stt.FallbackAdapter` additionally sets
`_recognize_metrics_needed = False` (`stt/fallback_adapter.py:111`). Do both here.
The wrapped STT's measurement is the one worth keeping -- it carries that
plugin's label plus its real model/provider metadata, where the adapter's carried
the adapter's own.

Adds `test_recognize_reports_usage_once` and `test_stream_reports_usage_once`,
both confirmed failing ("one usage report, 2 metrics events") with the forwarding
in place and the suppression removed.
@LHMQ878

LHMQ878 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Good catch, and confirmed — thank you. I measured it: with the forwarding alone the adapter reported 2 metrics events per recognition where there should be 1, on both paths (STT.recognize and RecognizeStream._metrics_monitor_task).

I had copied only half the pattern. Both sibling wrappers pair "re-emit the inner metrics" with "stop producing my own":

  • StreamAdapterWrapper._metrics_monitor_task is a no-op (stt/stream_adapter.py:107)
  • stt.FallbackAdapter additionally sets _recognize_metrics_needed = False (stt/fallback_adapter.py:111)

Fixed in 6ac65a5 by doing both. After it: 1 event per recognition on each path, and it is the wrapped plugin'''s measurement rather than the adapter'''s — so it carries that plugin'''s label and its real model/provider metadata instead of unknown/unknown.

Added test_recognize_reports_usage_once and test_stream_reports_usage_once, both confirmed failing at the intermediate state with AssertionError: one usage report, 2 metrics events, so the pairing cannot silently come apart again.

devin-ai-integration[bot]

This comment was marked as resolved.

…ter metrics

Suppressing the wrapper's own usage measurement by making `_metrics_monitor_task`
a bare no-op also dropped the `_num_retries = 0` that the base implementation
does on every final transcript. `RecognizeStream._main_task` gives up once
`_num_retries` exceeds `max_retry`, so without the reset brief failures spread
across a long call accumulate instead of being forgiven by successful
recognition, and recognition eventually stops for good.

Measured with an inner stream that delivers a good transcript and then drops the
connection on every attempt, `max_retry=3` on the wrapper: with the bare no-op,
four successful transcripts still left `_num_retries` at 3 and the stream raised
`failed to recognize speech after 3 attempts`; with the reset kept, recognition
kept going and `_num_retries` stayed 0.

`StreamAdapterWrapper` gets away with a bare no-op because it constructs itself
with `max_retry=0` (`stt/stream_adapter.py:16`, `:101`), so `_main_task` re-raises
on the first error and the counter is never consulted.
`MultiSpeakerAdapterWrapper` instead passes the caller's `conn_options` straight
through, so its retry budget is live and the reset matters. Verified StreamAdapter
does not accumulate under the same hiccup pattern.

Keep the reset and skip only the metrics emission. Adds
`test_successful_transcript_forgives_earlier_hiccups`, confirmed red with the bare
no-op ("recognition died after 4 good transcripts") and green with the reset,
while both metrics-count tests stay green.
@LHMQ878

LHMQ878 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Also confirmed — and this one I would not have caught by reading. Thank you.

I measured it with an inner stream that delivers a good transcript and then drops the connection on every attempt, max_retry=3 on the wrapper:

bare no-op monitor reset kept
good transcripts delivered 4, then dead keeps going
_num_retries after those transcripts 3 0
stream raised failed to recognize speech after 3 attempts yes no

So brief hiccups spread over a long call accumulated instead of being forgiven, exactly as described.

Worth noting why the siblings get away with the bare no-op I copied: StreamAdapterWrapper constructs itself with max_retry=0 (stt/stream_adapter.py:16, :101), so _main_task re-raises on the first error and never consults the counter. MultiSpeakerAdapterWrapper passes the caller'''s conn_options straight through, so its retry budget is live and the reset does real work. I re-ran the same hiccup pattern against StreamAdapter to check it isn'''t affected — it isn'''t.

Fixed in 523f77a: the override now skips only the metrics emission and keeps _num_retries = 0 on FINAL_TRANSCRIPT. Added test_successful_transcript_forgives_earlier_hiccups, confirmed red with the bare no-op (recognition died after 4 good transcripts) and green with the reset, with both metrics-count tests still green.

Full suite re-run: failure set byte-identical to the pre-change baseline, 1557 → 1567 passing, i.e. exactly the 10 new tests.

@longcw
longcw merged commit 59c2fae into livekit:main Aug 4, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants