BufferedTokenStream never yields the completed tokens (sentence or word) till the next token arrives causing delays in TTS Speech generation

[zaheerabbas-prodigal]

Bug Description

Issue 1:
Streaming tokenizers in livekit.agents.tokenize.token_stream.BufferedTokenStream never emit the first token unless a second token appears. Ref code

With real LLM output (multi‑second gaps between tokens in some cases) this means Cartesia/ElevenLabs TTS stays silent until another chunk arrives.

If we relax the guard to allow a single token, the loop spins forever because sentence/word tokenizers return inclusive end indices (tok[2]), so the buffer slice never shrinks.

Issue 2:
blingfire.SentenceTokenizer treats back‑to‑back sentences with no intervening space as a single token. So when the LLM outputs something like Sentence one.Sentence two - common when tool calls finish and new LLM request is made - BlingFire glues everything together, and the streaming path won’t yield a sentence until the entire turn finishes causing further delay in speech generation.

As an example run the tokenization script I have shared below and see the output as Could you please help me with your full name?What could go wrong. <break time="0.25s" />. even though there are 2 complete sentences here and 1 SSML tag but are sent to TTS as a single complete sentence. Please refer this

Expected Behavior

Issue 1:
As soon as a tokenizer detects a complete token - whether that’s a sentence, a word - it should be emitted immediately, even if it’s the only token so far and not depend on the look-ahead logic as the delay from LLM chunks can be in seconds in certain cases.

Issue 2:
Blingfire Sentence tokenizer should split between sentences instead of gluing the rest of the turn onto the first token if there are no intervening spaces

Reproduction Steps

1. pip install livekit-agents[elevenlabs,cartesia,openai,deepgram]==1.2.17
2. There are two scripts below as GH Gist

• tok.py script has LLM simulated responses with delays that have been observed in realistic cases of LLM when tool calls occur and the behavior of these tokenization implementation.
• tts_filler.py script that has the same above simulated responses but shows the experience with cartesia and elevelabs TTS - you manually have to update the script to use different TTS and run again.

3. Output of these scripts are added as comments in the Gist

• Execution of tok.py for all tokenizer behavior
• Execution of tts_filler.py with cartesia
• Execution of tts_filler.py with elevenlabs

Operating System

macOS Tahoe 26.0

Models Used

Deepgram nova-2-phonecall, elevenlabs, cartesia, custom llm simulation

Package Versions

livekit==1.0.16
livekit-agents==1.2.17
livekit-api==1.0.7
livekit-blingfire==1.0.0
livekit-plugins-anthropic==1.2.17
livekit-plugins-cartesia==1.2.17
livekit-plugins-deepgram==1.2.17
livekit-plugins-elevenlabs==1.2.17
livekit-plugins-google==1.2.17
livekit-plugins-noise-cancellation==0.2.5
livekit-plugins-openai==1.2.17
livekit-plugins-silero==1.2.17
livekit-plugins-turn-detector==1.2.17
livekit-protocol==1.0.8

Session/Room/Call IDs

No response

Proposed Solution

I have tried to relax the token length but that causes the buffer stream to go into an infinite state as mentioned in the bug description.

I am also unsure how we can implement this without introducing timers - hence created this bug to see if there could be other solutions as this is a core part of the code that hasn't been changed much.

Additional Context

Please go through this GH Gist

Screenshots and Recordings

No response

[zaheerabbas-prodigal]

Maybe we can't entirely call this as a bug as the core assumption might be that there won't be significant delays during LLM streaming. But in reality that does occur when using custom llm_node and custom llm_plugins and handling tool calls outside of the SDK.

One more thing that is missing here is that the latency being reported in ttfb of TTS doesn't take these buffering of text into account which causes someone to believe that the latency is lower but in fact in reality the latency is higher as there is buffering involved during streaming of text to TTS.

This causes the formula to be -> e2e latency = eou_delay + ttft + <unknown buffered amount> + ttfb

[zaheerabbas-prodigal]

cc: @theomonnom @longcw

[longcw]

Issue 1:
Streaming tokenizers in livekit.agents.tokenize.token_stream.BufferedTokenStream never emit the first token unless a second token appears.

break when len(tokens) <= 1 is to always keep the last token since we cannot know if it's an incomplete token without the followup context, e.g. "the price is $1.2" won't become "the price is $1." and "2".

the core assumption might be that there won't be significant delays during LLM streaming. But in reality that does occur when using custom llm_node and custom llm_plugins and handling tool calls outside of the SDK.

what do you think if there is a flush method or sentinel in custom llm_node that can flush all pushed text to TTS? ref #2417

[zaheerabbas-prodigal]

@longcw - thanks for the response

what do you think if there is a flush method or sentinel in custom llm_node that can flush all pushed text to TTS?

#2023 -> This PR might solve most of the challenges that we are facing, we essentially need a way to flush the generated text so far for an agent (and add that text as a message in chat_ctx). Any reason this PR isn't merged yet?

Is there a way for me to replicate the above PR in the current version of the SDK without using on_user_completed method, this is because our implementation is push based in nature (think websockets for LLM Agent) and the text can arrive at any point for a given turn - so the best way is to have the flush logic inside the llm_node. Appreciate your help here.

[longcw]

@zaheerabbas-prodigal

we haven't decide the API for the flush method.

for now a workaround is to customize both llm_node and tts_node, you can define a flush sentinel, yield if from the llm_node, and flush the tts stream when received it in the tts_node (https://github.com/livekit/agents/blob/livekit-agents@1.2.17/livekit-agents/livekit/agents/voice/agent.py#L426)

[zaheerabbas-prodigal]

e.g. "the price is $1.2" won't become "the price is $1." and "2".

@longcw One nuance though - if the LLM outputs a sentence that has a natural/full completion already (e.g. the price is $1.2. - notice the space after the .), I think the tokenizer should confidently emit that chunk even if the next token hasn’t arrived yet.

Right now, even fully terminated sentences (punctuation + trailing space) get held back waiting for the next token. I’m not sure if that is intentional.

On BlingFire: we’re seeing even stranger behavior when there is a natural completion and then a new (partial) token arrives—the already-complete chunk gets held back again.

Example stream:

1) "Let me look that up for you. <break time='0.2s'/>"
2) "Could you help"

At (1) we have a strong boundary (period + SSML self-closing tag). But when (2) arrives, BlingFire waits for the rest of the sentence before emitting anything - so the completed part becomes hostage to future tokens. See example output in my script

The only reason I am pushing for this is - it reduces latency in a lot of cases.

for now a workaround is to customize both llm_node and tts_node, you can define a flush sentinel, yield if from the llm_node, and flush the tts stream when received it in the tts_node (https://github.com/livekit/agents/blob/livekit-agents@1.2.17/livekit-agents/livekit/agents/voice/agent.py#L426)

Will implement and update here if this helps. Thanks!