fix(livekit-agents): forward the STT hooks through MultiSpeakerAdapter - #6679
Conversation
MultiSpeakerAdapter wraps another STT and passes `capabilities=stt.capabilities` straight to `super().__init__`, so it advertises whatever the wrapped recognizer supports -- including `keyterms` and `chat_context`, which both shipped diarization plugins (deepgram, assemblyai) set. But it forwarded none of the hooks those capabilities gate, so everything the framework pushed at it was dropped, and dropped silently: the base `_update_session_keyterms` / `_push_conversation_item` warn-and-skip only fires when the capability is False, and here it is True. Measured against the wrapped STT called directly: keyterms received None instead of the pushed list, 0 conversation items instead of 1, prewarmed False, model/provider "unknown" instead of the plugin's values, and 0 metrics reaching a listener on the adapter. Driving the real KeytermDetector reproduces it end to end -- `detector.start(session, stt=adapter)` with static keyterms leaves the wrapped recognizer's keyterms unset, and so does a mid-call `set_static_keyterms`. Forward the seven members `stt/stream_adapter.py:38-88` already forwards, in the same shape: `wrapped_stt`, `model`, `provider`, `_update_session_keyterms`, `_push_conversation_item`, `prewarm`, the `metrics_collected` re-emit, and an `aclose` that detaches it. `aclose` deliberately does not close the wrapped STT, matching StreamAdapter and stt.FallbackAdapter -- the caller constructed it and owns it.
Re-emitting the wrapped STT's `metrics_collected` in the previous commit made the
adapter forward the inner measurement while still producing its own for the same
audio, so every recognition was reported twice -- once via `STT.recognize`, once
via `RecognizeStream._metrics_monitor_task`. Measured 2 events per recognition on
both paths where there should be 1.
Both sibling wrappers pair the forwarding with suppressing their own: the stream
adapter's wrapper makes `_metrics_monitor_task` a no-op
(`stt/stream_adapter.py:107`) and `stt.FallbackAdapter` additionally sets
`_recognize_metrics_needed = False` (`stt/fallback_adapter.py:111`). Do both here.
The wrapped STT's measurement is the one worth keeping -- it carries that
plugin's label plus its real model/provider metadata, where the adapter's carried
the adapter's own.
Adds `test_recognize_reports_usage_once` and `test_stream_reports_usage_once`,
both confirmed failing ("one usage report, 2 metrics events") with the forwarding
in place and the suppression removed.
|
Good catch, and confirmed — thank you. I measured it: with the forwarding alone the adapter reported 2 metrics events per recognition where there should be 1, on both paths ( I had copied only half the pattern. Both sibling wrappers pair "re-emit the inner metrics" with "stop producing my own":
Fixed in 6ac65a5 by doing both. After it: 1 event per recognition on each path, and it is the wrapped plugin'''s measurement rather than the adapter'''s — so it carries that plugin'''s label and its real Added |
…ter metrics
Suppressing the wrapper's own usage measurement by making `_metrics_monitor_task`
a bare no-op also dropped the `_num_retries = 0` that the base implementation
does on every final transcript. `RecognizeStream._main_task` gives up once
`_num_retries` exceeds `max_retry`, so without the reset brief failures spread
across a long call accumulate instead of being forgiven by successful
recognition, and recognition eventually stops for good.
Measured with an inner stream that delivers a good transcript and then drops the
connection on every attempt, `max_retry=3` on the wrapper: with the bare no-op,
four successful transcripts still left `_num_retries` at 3 and the stream raised
`failed to recognize speech after 3 attempts`; with the reset kept, recognition
kept going and `_num_retries` stayed 0.
`StreamAdapterWrapper` gets away with a bare no-op because it constructs itself
with `max_retry=0` (`stt/stream_adapter.py:16`, `:101`), so `_main_task` re-raises
on the first error and the counter is never consulted.
`MultiSpeakerAdapterWrapper` instead passes the caller's `conn_options` straight
through, so its retry budget is live and the reset matters. Verified StreamAdapter
does not accumulate under the same hiccup pattern.
Keep the reset and skip only the metrics emission. Adds
`test_successful_transcript_forgives_earlier_hiccups`, confirmed red with the bare
no-op ("recognition died after 4 good transcripts") and green with the reset,
while both metrics-count tests stay green.
|
Also confirmed — and this one I would not have caught by reading. Thank you. I measured it with an inner stream that delivers a good transcript and then drops the connection on every attempt,
So brief hiccups spread over a long call accumulated instead of being forgiven, exactly as described. Worth noting why the siblings get away with the bare no-op I copied: Fixed in 523f77a: the override now skips only the metrics emission and keeps Full suite re-run: failure set byte-identical to the pre-change baseline, 1557 → 1567 passing, i.e. exactly the 10 new tests. |
Summary
stt.MultiSpeakerAdapteradvertises the wrapped STT's capabilities but forwards none of the framework hooks those capabilities gate.__init__passes the wrapped recognizer's capabilities through unchanged:So when the wrapped STT sets
keyterms=True/chat_context=True— both shipped diarization plugins do (deepgram,assemblyai) — the adapter claims them too. The framework then pushes keyterms and conversation items at the adapter, prewarms it, and readsmodel/provideroff it for metrics.MultiSpeakerAdapterdefined none of those, so all of it landed on theSTTbase class and went nowhere.The drop is silent precisely because the capability is advertised: the base
_update_session_keyterms/_push_conversation_itemwarn-and-skip paths only fire when the capability isFalse.stt/stream_adapter.py— the other STT wrapper in the same package — forwards all seven, which is what makes this look like an oversight rather than a deliberate difference:StreamAdapterMultiSpeakerAdapterwrapped_stt(stream_adapter.py:38)model/provider(:42,:46)_update_session_keyterms(:50)_push_conversation_item(:53)prewarm(:81)metrics_collectedre-emit (:36,:84)aclosedetach (:87)Measured on
bbf163fA diarization-capable recording STT that also supports keyterms and chat context, through the adapter vs. called directly:
MultiSpeakerAdaptercapabilities.keytermscapabilities.chat_contextNone['livekit', 'webrtc']model'unknown''recording-model-v1'provider'unknown''recording-provider'And through the real framework path rather than a direct hook call —
KeytermDetector, whichAgentActivitybinds to the session's STT:So enabling
MultiSpeakerAdapteron a Deepgram or AssemblyAI STT silently turns off keyterm boosting and context carryover for that session, and mislabels the STT in metrics asunknown/unknown.Change
Forward the same seven members
StreamAdapterdoes, in the same shape.Two deliberate details, both copied from the siblings rather than invented here:
acloseonly detaches the metrics handler and does not close the wrapped STT — matchingStreamAdapter.acloseandstt.FallbackAdapter.aclose. The caller constructed the inner STT and owns its lifecycle.StreamAdapterWrappermakes_metrics_monitor_taska no-op (stream_adapter.py:107) andstt.FallbackAdaptersets_recognize_metrics_needed = False(fallback_adapter.py:111); this does both. Measured 2 events per recognition on both therecognize()andstream()paths with the forwarding alone, 1 with the suppression. The kept event is the wrapped plugin'''s, which carries its label and its realmodel/providermetadata._num_retries = 0the base does on each final transcript, and_main_taskgives up once that exceedsmax_retry. Measured with an inner stream that transcribes then drops the connection every attempt (max_retry=3): with the bare no-op, four good transcripts still left_num_retriesat 3 and the stream died withfailed to recognize speech after 3 attempts; with the reset kept, recognition continued.StreamAdapterWrapperis safe with a bare no-op only because it builds itself withmax_retry=0(stream_adapter.py:16,:101) so the counter is never consulted;MultiSpeakerAdapterWrapperpasses the caller'''sconn_optionsthrough, so its budget is live. I verified StreamAdapter does not accumulate under the same pattern.Test
tests/test_multi_speaker_adapter.py(new module,pytestmark = pytest.mark.unit), 10 tests — one per forwarded member, one that drives the realKeytermDetectorend to end instead of calling the hook directly, two that pin the metrics count on each path, and one that pins the retry-counter reset.The 7 delegation tests confirmed red on the unmodified tree before being kept:
The 2 metrics tests were confirmed red at the intermediate state (forwarding without suppression):
AssertionError: one usage report, 2 metrics events. The retry test was confirmed red with a bare no-op monitor:AssertionError: recognition died after 4 good transcripts.MultiSpeakerAdapteritself had no coverage before this —tests/test_multi_speaker_primary_speaker.pyexercises the private_PrimarySpeakerDetectoronly, never the adapter.Verification
ruff checkandruff format --checkon both files — clean.mypy(repo config, strict) onlivekit.agents.stt—Success: no issues found in 5 source files.pytest --unit: I could not install the full workspace on this machine —bithuman2.7.0 publishes no Windows wheel, souv sync --all-extras --devfails outright. I ran the suite against an editablelivekit-agentsinstead, which leaves 16 modules erroring on absent plugin packages. Captured the sortedFAILED/ERRORset with the change (and the new test file) removed and restored: identical, 20 entries both ways (16 collection errors + 4 pre-existing failures intest_ivr_activity.pyandtest_tokenizer_xml_markup.py, unrelated). Passing count 1557 → 1567, i.e. exactly the 10 new tests.I did not touch
CHANGELOG.mdor any package manifest, per CONTRIBUTING.