ElevenLabs plugin not working for eleven_v3 model

[devb-enp]

Bug Description

The current implementation forces to use multi-stream-input websocket connection which doesn't even support eleven_v3 model.

https://elevenlabs.io/docs/cookbooks/multi-context-web-socket#:~:text=Multi%2Dcontext%20WebSockets%20are%20not%20available%20for%20the%20eleven_v3%20model.

Expected Behavior

There should be an option to use stream-input if we want to use eleven_v3 to get the latest high quality voice and features.

Reproduction Steps

Just try running eleven_v3 in the current version and it'll throw error

Operating System

macOS

Models Used

ElevenLabs TTS

Package Versions

1.2.17

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[devb-enp]

@longcw can you check this? we're stuck with older version of agent as long as this is not fixed

[davidzhao]

@devb-enp does it work well with the older API? what's the latency like?

[devb-enp]

@davidzhao we're on version 1.1.4 and wanted to upgrade it to latest, for 1.1.4 it's working when I pass eleven_v3.

i tried in ElevenLabs playground, the websocket connection were not working for eleven_v3 which is the sream url, but it's working with stream which is synthesize url

I tried running my latest upgraded code with 1.1.4 of 11labs but that also didn't work

[ndandanov]

Thanks for this discussion. I'd also like to use the eleven_v3 model with LiveKit.

However, according to the official ElevenLabs documentation here:

WebSockets are not available for the eleven_v3 model.

And here:

Multi-context WebSockets are not available for the eleven_v3 model.

[devb-enp]

@ndandanov That's exactly my point. It's not available as a websocket but it's available as a stream.
I see that we already have two functions for the URLs, I suggest we should have an option to use one or the other.
_synthesize_url OR _multi_stream_url

[ndandanov]

A very simple way to make eleven_v3 work in livekit-agents 1.3.1 is to use the synthesize method of the TTS node when overriding tts_node:

    async def tts_node(
            self, text: AsyncIterable[str], model_settings: ModelSettings
    ) -> AsyncIterable[rtc.AudioFrame]:
        buf = ""
        async for t in text:
            buf += t
        async for output in self.session.tts.synthesize(buf):
            yield output.frame

Set the TTS node as usual:

    session = AgentSession(
...
        tts=elevenlabs.TTS(
            model="eleven_v3",
        ),
...

One downside is that this is introduces significant delay, especially for long phrases.
Therefore, this shall not be used in production, but it's useful for testing eleven_v3 for your use case.

[devb-enp]

Oh I didn't think about overriding it, thanks this helps lot, I'll try and let you know

[devb-enp]

This worked so closing this issue

[nshelia]

I did try override option and it works but audio breaks sometimes with error message.

ARNING: failed to synthesize speech, retrying in 2.0s


Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 355, in _wait
await waiter
asyncio.exceptions.CancelledError


The above exception was the direct cause of the following exception:


Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 347, in _run
async for data, _ in resp.content.iter_chunks():
File "/app/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 71, in __anext__
rv = await self._stream.readchunk()
File "/app/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 484, in readchunk
await self._wait("readchunk")
File "/app/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 354, in _wait
with self._timer:
File "/app/.venv/lib/python3.11/site-packages/aiohttp/helpers.py", line 713, in __exit__
raise asyncio.TimeoutError from exc_val
TimeoutError


The above exception was the direct cause of the following exception:


Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/livekit/agents/tts/tts.py", line 280, in _main_task
await self._run(output_emitter)
File "/app/.venv/lib/python3.11/site-packages/livekit/plugins/elevenlabs/tts.py", line 353, in _run
raise APITimeoutError() from e
livekit.agents._exceptions.APITimeoutError: Request timed out. (body=None, retryable=True)```