Feature request: per-run target-language segments on `stt.SpeechData`

[MSameerAbbas]

Feature Type

Would make my life easier

Feature Description

stt.SpeechData exposes per-segment data on the source side (source_languages, source_texts) but on the target side only a single concatenated text and a single language. Under Soniox two-way translation with code-switched input, the translated text legitimately contains spans in both target languages, but downstream consumers can't tell which spans are which because:

1. SpeechData.language is set to the first translated token's language only — first-wins in livekit-plugins-soniox's _TokenAccumulator.update.
2. The Soniox plugin already builds the per-run translated list (final._lang_segments, the same coalescing pass it uses to build the source side) but discards it in send_endpoint_transcript — only final_original._lang_segments is forwarded as source_languages / source_texts.

The proposal is purely additive and symmetric: add target_languages / target_texts to SpeechData (per consecutive-language run, mirroring the existing source-side semantics), and have the Soniox plugin populate them from final._lang_segments.

Why per-run, not per-token

The source side is already per-run — _TokenAccumulator.update coalesces consecutive same-language tokens into a single (lang, text) entry. Asking for the target side to be per-run keeps the API symmetric, requires the smallest possible plugin diff (the per-run list is already computed), and matches what consumers actually want for highlighting / display ([en, es, en] instead of [en, en, en, en, es, es, es, en, en]). Anyone who needs word-level granularity can already use the existing SpeechData.words: list[TimedString] field.

Reproduction

from livekit.agents import AgentSession
from livekit.plugins import soniox

session = AgentSession(
    stt=soniox.STT(
        params=soniox.STTOptions(
            model="stt-rt-v4",
            language_hints=["en", "es"],
            language_hints_strict=True,
            translation=soniox.TranslationConfig(
                type="two_way", language_a="en", language_b="es",
            ),
        )
    ),
)

Speak: "No hablo español but I speak English"

Resulting SpeechData on the FINAL_TRANSCRIPT event:

SpeechData(
    text             = "I don't speak Spanish, pero hablo inglés.",
    language         = "en",                                  # misleading: second clause is Spanish
    source_languages = ["es", "en"],                          # per-run, correct
    source_texts     = ["No hablo español, ", " but I speak English."],
    # target_languages / target_texts: not surfaced today
)

Expected after this feature:

SpeechData(
    text             = "I don't speak Spanish, pero hablo inglés.",
    language         = "en",                                  # unchanged for back-compat (first run)
    source_languages = ["es", "en"],
    source_texts     = ["No hablo español, ", " but I speak English."],
    target_languages = ["en", "es"],                          # NEW
    target_texts     = ["I don't speak Spanish, ", "pero hablo inglés."],   # NEW
)

Proposed solution

In livekit-agents, extend stt.SpeechData:

@dataclass
class SpeechData:
    ...
    target_languages: list[LanguageCode] | None = None
    """Per consecutive-language-run target languages, parallel to `target_texts`.
    Mirrors the existing `source_languages` / `source_texts` semantics on the source side.
    Populated by STT services that support translation; None when translation is off."""
    target_texts: list[str] | None = None
    """The translated transcription segments in the target language(s).
    Each entry corresponds to the same-indexed entry in `target_languages`."""

In livekit-plugins-soniox, mirror the existing source-side block in send_endpoint_transcript:

src_segs = final_original._lang_segments
source_languages = [LanguageCode(lang) for lang, _ in src_segs] or None
source_texts     = [t for _, t in src_segs] or None

# NEW — target side, symmetric to source
tgt_segs = final._lang_segments
target_languages = [LanguageCode(lang) for lang, _ in tgt_segs] or None
target_texts     = [t for _, t in tgt_segs] or None

self._event_ch.send_nowait(
    stt.SpeechEvent(
        type=SpeechEventType.FINAL_TRANSCRIPT,
        alternatives=[
            final.to_speech_data(
                self.start_time_offset,
                source_languages=source_languages,
                source_texts=source_texts,
                target_languages=target_languages,
                target_texts=target_texts,
            )
        ],
    )
)

final._lang_segments is already computed by the existing _TokenAccumulator.update coalescing logic — no new accumulation work, no new state, no protocol-level changes. Consumers that don't read the new fields are unaffected (back-compat preserved).

The same change naturally applies to the INTERIM_TRANSCRIPT path that already merges source-side segments via _merge_lang_segments.

Workarounds / Alternatives

No response

Additional Context

No response

[chenghao-mou]

Thanks for the issue! Do you want to create a PR for this?

[MSameerAbbas]

Follow-up: same plugin gap surfaces symmetrically when translation is OFF

While testing the same plugin in non-translation mode, I noticed the same SpeechData-collapse symptom on the source side. I want to flag it here rather than open a new issue because the root cause is identical to what this issue already describes — same plugin, same function, same dropped accumulator — and the right fix handles both halves together.

What I observed (translation OFF, code-switched utterance)

Spoken: "こんにちは、君の名前は何だ。 My name is Sam." (Japanese then English)

Resulting SpeechData on the FINAL_TRANSCRIPT event:

SpeechData(
    text="こんにちは、君の名前は何だ。 My name is Sam.",   # both runs concatenated
    language="ja",                                       # first-wins -- the en run is lost
    source_languages=None,                               # not populated when translation is off
    source_texts=None,                                   # not populated when translation is off
)

Per-token language data exists in the Soniox response (every token has language), but downstream consumers can't tell that the second clause is English. Code-switched dictation in non-translation mode looks monolingual to whoever reads SpeechData.

Why it's the same plugin gap

The original issue points out that final._lang_segments is built per-run but discarded in send_endpoint_transcript. That's exactly what's happening here too — just on the other side, in the other mode:

Translation mode	final._lang_segments semantically holds	What send_endpoint_transcript reads	Net effect
ON	per-run target segments (the translation)	final_original._lang_segments for source_*, ignores final._lang_segments	target-side per-run data dropped (the original issue's symptom)
OFF	per-run source segments (no translation overlay)	final_original._lang_segments (empty in this mode) for source_*, ignores final._lang_segments	source-side per-run data dropped (the symptom above)

Same accumulator. Same line of code that drops it. Just looked at through two different lenses depending on whether translation is on.

Unified fix in send_endpoint_transcript

The fix proposed in this issue (adding target_* and populating them from final._lang_segments) handles the translation-on case. Falling back to final._lang_segments for source_* when final_original._lang_segments is empty handles the translation-off case. Both changes land in the same block:

def send_endpoint_transcript() -> None:
    nonlocal is_speaking
    if final.text:
        # When translation is on, `final_original` holds the source side and
        # `final` holds the target side. When translation is off, `final_original`
        # is empty and `final` IS the source side. The fallback unifies both.
        src_segs = final_original._lang_segments or final._lang_segments
        source_languages = [LanguageCode(lang) for lang, _ in src_segs] or None
        source_texts     = [t for _, t in src_segs] or None

        # NEW target-side fields (per this issue). Only populated when translation
        # is on -- otherwise `final_original` is empty and there's no "translation"
        # to expose distinct from the source.
        if final_original._lang_segments:
            tgt_segs = final._lang_segments
            target_languages = [LanguageCode(lang) for lang, _ in tgt_segs] or None
            target_texts     = [t for _, t in tgt_segs] or None
        else:
            target_languages = None
            target_texts     = None

        self._event_ch.send_nowait(
            stt.SpeechEvent(
                type=SpeechEventType.FINAL_TRANSCRIPT,
                alternatives=[
                    final.to_speech_data(
                        self.start_time_offset,
                        source_languages=source_languages,
                        source_texts=source_texts,
                        target_languages=target_languages,
                        target_texts=target_texts,
                    )
                ],
            )
        )
        ...

Properties:

• Translation off (today: source_* is None): final_original._lang_segments is empty, fallback uses final._lang_segments, consumers get per-run source segments. target_* is None (no overlay exists, by definition).
• Translation on, no code-switch (today: source_* is a single segment): unchanged. final_original._lang_segments has one entry, final._lang_segments has one entry, both flow through.
• Translation on, code-switched (today's broken case): both halves flow. source_* carries the per-run originals (as before), target_* carries the per-run translations (the new fields).

The same INTERIM_TRANSCRIPT path (_merge_lang_segments-driven) gets the symmetric treatment.

No protocol-level changes, no new accumulation work — the per-run data is already being computed by the existing _TokenAccumulator.update coalescing logic on both accumulators. The diff is purely about exposing it.

Reproduction (translation OFF)

from livekit.agents import AgentSession
from livekit.plugins import soniox

session = AgentSession(
    stt=soniox.STT(
        params=soniox.STTOptions(
            model="stt-rt-v4",
            language_hints=["ja", "en"],
            language_hints_strict=False,
            # translation deliberately omitted
        )
    ),
)

Speak: "こんにちは、君の名前は何だ。 My name is Sam."

• Today: SpeechData.source_languages is None, SpeechData.source_texts is None, SpeechData.language == "ja", single-language utterance from the consumer's perspective.
• Expected after the unified fix: source_languages == ["ja", "en"], source_texts == ["こんにちは、君の名前は何だ。", " My name is Sam."].

Happy to submit the PR

If a unified PR (this comment + the original issue's target-side proposal, plus tests for both modes) would be useful, I'm happy to put one together.

cc @chenghao-mou in case this is helpful context.

[chenghao-mou]

Yeah, that sounds good.