Mobile Echo Issue with iphone's built in mics

[ClvHbl]

Bug Description

Issue Description

We're experiencing echo on mobile devices in our LiveKit voice agent setup when using the phone's built in microphones. The agent hears itself speaking back, creating an echo effect that causes performance degradation as it's considered as user speech. I have also tried on your playground and on mobile on iphone it's doing the same...

Setup Details

Client-side (Mobile Browser)

• React frontend using livekit-client
• Audio track created with getUserMedia with these constraints:

{
  channelCount: 1,
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: true,
  voiceIsolation: true, // iOS only
  // Google WebRTC constraints
  googDAEchoCancellation: true,
  googEchoCancellation: true,
  googEchoCancellation2: true,
  googNoiseSuppression: true,
  googNoiseSuppression2: true,
  googAutoGainControl: true,
  googAutoGainControl2: true,
  googHighpassFilter: true,
  googTypingNoiseDetection: true
}

• Audio track wrapped in LocalAudioTrack with userProvidedTrack=true
• Published to room with source: Track.Source.Microphone
• Agent audio playback volume set to 0.75 on mobile
• Playback element has playsinline and webkit-playsinline attributes

Server-side (LiveKit Agents - Python)

• Python agent using livekit-agents SDK
• For mobile devices, we force BVC noise cancellation (we've tried with standard and others and there is the same looping issue each time...):

from livekit.plugins import noise_cancellation

# Mobile detection
is_mobile = getattr(agent_config, 'is_mobile', False)

if is_mobile:
    room_input_options.noise_cancellation = noise_cancellation.BVC()

• Room connection:

await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

• Session started with:

room_input_options = RoomInputOptions(
    text_enabled=False,
    audio_enabled=True,
    close_on_disconnect=False
)

await session.start(
    room=ctx.room,
    agent=agent,
    room_input_options=room_input_options,
    room_output_options=room_output_options,
)

Environment

• LiveKit Server: Cloud (LiveKit Cloud with BVC enabled)
• Client SDK: livekit-client (latest)
• Agent SDK: livekit-agents (Python, latest)
• Noise Cancellation Plugin: livekit-plugins-noise-cancellation
• Mobile Browsers: iOS Safari, Chrome Android

Observed Behavior

Sometimes it's mostly fine but if I put my finger on top of the mic of if I suddenly move the iphone near my t-shirt or against a surface that creates reverberation the agnet's speech is detected as user's speech and it can create a loop interrupting the agent etc...

Questions

Is there a specific way to configure agent playout settings to prevent the agent from hearing itself? Should we be using specific RoomOutputOptions settings?

Are there known issues or best practices for using BVC on mobile browsers (iOS Safari, Chrome Android)?

I have tried elevenlabs widget that appears on a website to talk to it and whatever I try it's never doing such things, I also tried Grok in the web and it's working fine, so either I am doing something wrong, or they have some secret sauce that is preventing that...

Expected Behavior

On mobile even when using the iphone's microphone there should be no loop where the agent's speech is detected as user's speech, it should work like the Elevenlabs widget.

Reproduction Steps

1. Use Livekit SDK client side
2. Try the app on mobile
3. Just try to put a finger on the mic or move slightly or even sometimes don't do anything and at the beginning it can loop
...
- Sample code snippet, or a GitHub Gist link -



{
  channelCount: 1,
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: true,
  voiceIsolation: true, // iOS only
  // Google WebRTC constraints
  googDAEchoCancellation: true,
  googEchoCancellation: true,
  googEchoCancellation2: true,
  googNoiseSuppression: true,
  googNoiseSuppression2: true,
  googAutoGainControl: true,
  googAutoGainControl2: true,
  googHighpassFilter: true,
  googTypingNoiseDetection: true
}



### Server-side (LiveKit Agents - Python)
- Python agent using `livekit-agents` SDK
- For mobile devices, we force BVC noise cancellation (we've tried with standard and others and there is the same looping issue each time...):

from livekit.plugins import noise_cancellation

# Mobile detection
is_mobile = getattr(agent_config, 'is_mobile', False)

if is_mobile:
    room_input_options.noise_cancellation = noise_cancellation.BVC()


- Room connection:

await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)


- Session started with:

room_input_options = RoomInputOptions(
    text_enabled=False,
    audio_enabled=True,
    close_on_disconnect=False
)

await session.start(
    room=ctx.room,
    agent=agent,
    room_input_options=room_input_options,
    room_output_options=room_output_options,
)

Operating System

IOS 18.5, Iphone 14 Pro Max

Models Used

Deepgram Nova-3, GPT 4.1, Google CHIRP 3 HD (also tried with other models and I have the same issue)

Package Versions

# LiveKit Core
livekit==1.0.12
livekit-agents==1.2.14
livekit-api==1.0.5
livekit-blingfire==1.0.0
livekit-plugins-anthropic==1.2.14
livekit-plugins-assemblyai==1.2.14
livekit-plugins-cartesia==1.2.14
livekit-plugins-deepgram==1.2.14
livekit-plugins-elevenlabs==1.2.14
livekit-plugins-gladia==1.2.14
livekit-plugins-google==1.2.14
livekit-plugins-groq==1.2.14
livekit-plugins-noise-cancellation==0.2.5
livekit-plugins-openai==1.2.14
livekit-plugins-silero==1.2.14
livekit-plugins-speechmatics==1.2.14
livekit-plugins-turn-detector==1.2.14
livekit-protocol==1.0.4

# AI Provider SDKs
openai>=1.0.0
anthropic>=0.20.0
google-generativeai>=0.3.0
elevenlabs>=0.2.0

# Web and networking
fastapi>=0.100.0
uvicorn>=0.23.0
websockets>=11.0.0
httpx>=0.27.0
aiohttp>=3.8.0
urllib3==1.26.18  # Pin to v1.x for LibreSSL compatibility

# Utilities
python-dotenv>=1.0.0
pydantic>=2.0.0
asyncio>=3.4.0
psutil>=5.9.0
redis==6.4.0
axiom-py>=0.3.0

# ML Framework
# Torch is installed separately in Dockerfile (CPU version for cloud)
# For local: pip install torch==2.3.1 --index-url https://download.pytorch.org/whl/cpu

Session/Room/Call IDs

Every single mobile conversation not isolated to any specific. Here is one example of it:

Room session ID: RM_VvZGESXbhLAy
Room name: agent_1fc1c7c3-dcad-4fc8-8477-10a74fb18302_session_session_1761782751543_t9652st

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[tinalenguyen]

Sometimes it's mostly fine but if I put my finger on top of the mic of if I suddenly move the iphone near my t-shirt or against a surface that creates reverberation the agnet's speech is detected as user's speech and it can create a loop interrupting the agent etc...

I think using a VAD or setting min_interruption_words may help filter out any reverberation from such noises. Using a turn detector would also help prevent interruptions using context.

Are there known issues or best practices for using BVC on mobile browsers (iOS Safari, Chrome Android)?

For Android apps, it's best to ensure that the audio mode is always set to communication rather than media for better two-way communication.

Let me know if implementing these suggestions alleviates your issue!

[ClvHbl]

Hey thanks for the quick response, unfortunately I already have a pretty high VAD activation threshold, I already use min_interruption_words=1 and and for the turn detector you mean EOU? if so this only runs to terminate the turn and not really to decide if it shall interrupt or not I think, am I wrong?

[tinalenguyen]

You're right that it's more geared towards end-of-turn detection; my thought with that suggestion was that the turn wouldn't be committed as soon without a full transcript signaling so.

For iOS, it would be useful to switch the audio output to the earpiece rather than the speaker to prevent echoing. I don't think this is possible on the browser level unfortunately, but maybe there is a workaround?

I suspect that for the elevenlabs widget, they may temporarily disable STT/VAD inference while the agent is speaking. Cannot confirm if that is the secret sauce though.

[ClvHbl]

Yeah idk, I think they rather have a serve-side processing or something like that on top of the crappy hardware and browser's AEC. Cuz I can still interrupt the agent while he is speaking so they must keep the STT/VAD for that.

But yeah it's quite frustrating, not a lot of resources out there on this topic, and doesn't seem to be a robust public solution...

As Livekit is first of all a webrtc / conferencing framework I thought some of the lads working there would know how to prevent that, are there some experts on this specifically in the team?

[xianshijing-lk]

Do you have the same issues or behaviors across Safari and Chrome on IOS ?

Any chance that you can record the audio on the server before passing the data to LLM ? so that we will check the amount of echo ?

[ClvHbl]

Yeah it's almost the same in all browsers. It mainly happens during the greeting, idk if there is a hardware level model that takes time to calibrate itself or something but the greeting each time is really bad, the way I prevent looping here is just by not allowing interruptions during it. After it it seems to be a bit better, more stable, but if I unintentionally obstruct the mics somehow then it can loop pretty easily.

here are a few examples of echo during the first agent spoken message:

session_1761847475412_exnno4y.mp3

session_1761782751543_t9652st.mp3

then here for example it happened mid conversation with a sudden loud echo:

session_1761847304124_zv01vc0.mp3

and here is a full conversation where after 30s I mess up a bit the mic by obstructing it (putting it close to my sweat shirt, or putting my finger slightly over it etc...):

session_1761845087610_mydxhn9.mp3

But just to clarify what I share here are the outcome of dozens of iterations, so it's using BVC on mobile, otherwise it's even worse, I don't allow interruptions during the greeting message otherwise it's looping 60% of the time on this first message. I reduce the volume as well to put it at 0.75. I tested all the params when capturing the track client side. I have min_interruption_words set to 1 cause otherwise I don't even want to imagine how bad it would be with the VAD only... I have a pretty stable STT model (nova-3) I tried flux and it's a nightmare, not at all production ready as it's transcribing some stuff as soon as it gets audio even if this one is empty. And moreover I have a pretty good phone, iphone 14 pro max and I tried on iphone 15, 16 etc and it's the same issue even though they are supposed to the best hardware out there, so again it's probably even worse on older phones.

I managed to make it usable in production I think but it's far from being bulletproof...

[xianshijing-lk]

Hi again, thanks for the context.

The initial echo is a known problem that happens in quite a few audio chat applications.

AEC works in the way that the algorithm needs to cache some speaker audio as reference, so that it can detect the echo in the microphone audio by comparing mic audio with its cached speaker audio.

At the beginning of the call, as the speaker audio is still being cached, the initial echo will leak.

That says, no matter if it is software or hardware AEC, it will take a short time to stabilize.

Proposal : you probably can workaround it by playing some silence at the beginning of the call, like 1-2 seconds of silence. Note, you might not need to play any ringtone, but just inject some silence. But lots of chat apps will play a small greeting tone.

For other echo during the call, if I listen it carefully, I think the problem can be introduced by the AGC (automatic gain control). As context, AGC will automatically scale up the input volume when it detects some sound, but it will introduce echo problems as AEC assumes a mostly linear echo path, in some cases, if AGC is misbehaving, like cranking the far-end too loud introduce non-linear distortion, in that case, AEC can’t model this well, so users will hear distorted residual echo.

So please try setting the following things to false
googAutoGainControl: false,
googAutoGainControl2: false,

And let me know if these two things will solve or help the problems.

Thanks again,

[ClvHbl]

Hi, very insightful, it makes sense, thank you for this detailed answer, I will test both things and let you know if I can still hear it.

[ClvHbl]

Hi, I've been looking deeper and I am curious about the kind of setup that you would recommend to have client side, should I play this greeting tone in the same AudioContext as the agent's speech? Cause by default livekit is creating an html element, that is created only after so it's not really possible. So should I override that, is it even possible? Or I don't care and everything will be treated the same way as long as it's coming out of the speaker?

Also maybe it's my setup that is messed up but on mobile when wearing in-ear monitors it's consistently not playing the first few words as if the route was not created yet and has to catch up each time after 1-2s. Is that supposed to be solved with this tone as well, I also know that it sometimes require to have a user gesture in order to play audio but livekit's html element is probably bypassing that right?

[xianshijing-lk]

They are very good questions. This part is indeed very error prone and easy to mess up the system audio routing.

So the principle is to stick with one output route and keep it warm, that says, you can use WebAudio to play the greeting tone, as long as all the audio go to the same output device. If you want a single processing chain, route the remote track into WebAudio.

Regarding "Should I play the greeting tone in the same AudioContext as the agent’s speech?”
Assuming you are using WebAudio to play the greeting tone, it shouldn't matter if that is the same AudioContext as long as the audio goes to the same audio device. so it should be correct that "I don't care and everything will be treated the same way as long as it's coming out of the speaker?"

regarding “wearing in-ear monitors it's consistently not playing the first few words as if the route was not created yet and has to catch up each time after 1-2s”
I think the greeting tone via WebAudio will likely warm the media element, so the problem might be improved.
But the browsers will still satisfy the gesture requirement before audible playback is guaranteed, so livekit html element does not bypass it.

[ClvHbl]

Thanks, but even with all of that I can still notice echo... I see that there is a possibility to use singlePeerConnection when creating the room and I think that by default it's using dual connections. Do you recommend to use a single one perhaps?

[ClvHbl]

Another person told me that web based echo cancellation is pretty limited and recommend to have a native shell (built with something like capacitor or React Native) embed my existing web app inside a webview to gain finer native controls... What do you think about this?

[tarun7r]

Hi @ClvHbl , Could you please elaborate on how you resolved the echo issue where the LiveKit agent picks up its own voice as a query? This occurs during the welcome message and occasionally during user interactions on iOS devices. Have you found any solution that can be implemented at the client level?