Investigate Issues With Users Audio Not Working

[robertlong]

Umbrella issue for looking into recent reports of users not being able to hear others. This is intended to capture only bug reports made after we switched to the MediaSoup SFU.

• Add support for selecting audio input after joining a room (Support for Mic selection #3620)
• Add support for ICE retries (ICE retries and RTC fixes #3662)
• Better handle signaling disconnects in client/server to avoid a full reconnect (Suddenly lose audio until avatars briefly flash and reconnect to audio Mozilla Hubs #2584 (comment))
• Improve ICE reconnects in FF to independently reconnect the affected transport.
• Support for spatial and temporal layers.

┆Issue is synchronized with this Jira Task

[emclaren]

the hubs email address received this message related to audio issues on July 29

"Hubs is my absolute favorite video chat and conferencing tool. However, a couple of friends and I tested out his room where we had some audio issues. We are all very tech savvy, so none of believe it was a permissions or settings issue on our devices. I used the latest version of Firefox on a MacBook Pro while my friend used Chrome on his Windows PC. He could hear me, but I could not hear him nor could our other friend hear him. The other friend was using his iPhone and could hear me, but I could not hear him either.

Unless there is an obscure setting or permission we need to change that we haven’t thought of being related (e.g. accessibility features), I believe this is a programming bug."

[zach-vz]

@emclaren @robertlong We are also experiencing this issue in our Hubs Cloud deployment. It only happens in some instances, so does not appear to be driven by a setting or permission. We have found no way to fix yet except refreshing on the user's end. If there are additional logs we can provide, of course, we'll be in touch.

[JacobErvin]

Also encountered this. Was in a group, and one member (Person A) was unable to see or hear another member (person B). All other attendees could see and hear person B correctly. Person A re-loaded and it resolved the issue, but it caused significant confusion.

[vjFred]

Have this same issue, and also people sometimes don´t see each other, this on Hubs Cloud.

[zach-vz]

Additional context (@vjFred @JacobErvin -- would love for you to comment on whether this is your experience as well). One pattern we're seeing is that avatars are not loading up for users, and if they don't load up, that often seems to exclude them from AV in the space (not to mention they are invisible). I think these things could be very closely linked. Most of my tests have been in Hubs Cloud, but we've experienced this in Hubs as well, with both the default set of avatars as well as custom avatars we have imported.

[emclaren]

Hey folks - we have created a form to help us see if we can spot a pattern of when these issues are occurring

If you are experiencing issues with WebRTC (e.g. voice audio, webcam, and desktop streaming) could you complete this form

[vjFred]

@emclaren Great. i am closely follow this posts. Thnks

[blairmacintyre]

Just had this happen in a meeting in HMC with ~10 people. Two of the user complained they weren't hearing some of the others. Reloading fixed for one.

[blairmacintyre]

Related to this: I would strongly push for an option for Hubs Cloud installations for doing server-side audio mixing. This is a cost/performance tradeoff, and many people would (I think) choose to pay more for the option to get around all these audio issues (both this issue of some streams not working, and the various "low end device or crappy network can't handle the load of many streams").

Especially if this was something you could configure per room, or toggle on/off in the server admin panel.

For example, I'm using hubs to teach, and have a research budget that would allow me to jump from < $100/month to > $100/month without really causing much pain. (if it was > $1000/month, I'd need to reconsider).

[vjFred]

I think is time for a different voice server

[blairmacintyre]

The voice server was actually switched recently. It supports everything I described, it's just a matter of priorities and time. The Hubs team has a lot to do, so they will decide where this falls.

I wish I had the resources and time; it seems like something that could be done in a relatively short period, but that's probably true for the 1000 other things on the todo. ;)

[rawnsley]

I am having a similar problem: A could hear B, but B could not hear A. Also C could hear both A and B.

I actually connect to Hubs with my own native Android client but there may be some similarities in the sequence of connection events and opportunities for error.

During the setup of the DialogAdapter in naf-dialog-adapter.js there is a call (via protoo) to "join" the room. This returns a list of initial peers. Very soon after this call returns (sometimes before this list can be processed in my case) it is possible to receive "newConsumer" messages detailing audio streams that need to be consumed. In my code I was not accounting for this properly and rejected the newConsumer message because there was no registered peer to associate it with.

I am struggling to chase through the equivalent Hubs client javascript code, but it might be worth looking at this critical point in the event sequence in more detail. At the very least it would be a good warm up for investigating the problem and perhaps increasing the console logging in a bid to narrow down the cause.

I haven't done enough further testing to confirm if this was the whole story behind the failure for us, but I will post again if I find anything useful.

[keianhzo]

Some of the scenarios reported here (like #2643 (comment)) might fall in the #3093 case, if there is any model loading error on the client side, that avatar won't have an avatar-audio-source and won't be heard. That PR might help in those cases.

[vincentfretin]

I'm wondering if you're using the RTCPeerConnection iceConnectionState transition to "disconnected" the right way in
https://github.com/mozilla/hubs/blob/b9a8797277cacc7f31bb35cebb349817e3b1c79f/src/naf-dialog-adapter.js#L183-L186
maybe you should try a this.reconnect() to do an ICE restart? See https://stackoverflow.com/a/60232517

[robinkwilson]

This could also confuse people:
#3518

[vincentfretin]

I still use mozilla janus sfu plugin in my project hosted in a kubernetes cluster at Scaleway and I'm doing the audio setup almost like in Hubs. Lately the bug can be reproduced a lot, all is good and suddenly the mic from a participant seems to stop working and they need to refresh the page. All participants are using Chrome, almost all are on macOS, some on Windows. Maybe the audio track goes into the ended state for whatever reason?
I see you have a recreateAudioStream only for Oculus.
https://github.com/mozilla/hubs/blob/910923d66e84de79089ae87438b514c616b232f6/src/react-components/ui-root.js#L655-L672
So just sharing if you want to investigate for Hubs, maybe put a log on the ended event for the audioTrack.
I'm going to test doing the recreateAudioStream for all browsers so including chrome to see if things improve.

Note: doing the reconnect() for RTCPeerConnection disconnected state didn't resolve the issue, so it was not it.

[vincentfretin]

Oh this is actually a good lead, the audioTrack "ended" is definitely one of the origin of the issue. I can reproduce the issue on Chrome macOS with an external USB mic, just by disconnecting and reconnecting the USB mic. Apparently sometimes the USB can disconnect and reconnect just a second, maybe bad contact, or if you slightly touch the cable... When this occurs, the audioTrack will emit the ended event. In my case where I enabled the recreateAudioTrack when the "ended" event is emitted, it will fail because the USB mic is not reconnected yet, and will give simply an error DOMException "mic not found" when doing the getUserMedia with this specific mic.
What I will try I think is when the "ended" event occurs, try recreating the audio track for this mic, if I have an exception try to recreate the audio track 5 seconds later so the user can reconnect the mic or the system find the mic back again if it's the OS that had a micro cut on the USB. #3543 fixes this particular issue.

Note I have a similar issue on Hubs when I use the internal mic on Ubuntu 18.04 with my Dell XPS laptop and my laptop is connected to a dock via usb. If I touch the USB cable, the dock can be disconnected and reconnected in a second, and the mic will stop working (reproducible only on Chrome, Firefox doesn't have this issue). But for this case, there is no "ended" event. I have the same issue on Google Meet, to fix the issue I need to switch the mic via the select dropdown and select back again the original mic so it works again.
To detect this issue, if the mic volume was previously > 0.05 and if during like 10s the volume is around 0 (AFRAME.scenes[0].systems["local-audio-analyser"].volume returns 2.3899599043280413e-119 in my test) and you are un-muted, try recreating the audio track. What is working for me to get back the mic is first trackAudio.stop() then call recreateAudioTrack().

[vincentfretin]

@robinkwilson With the Room UI Redesign, do you provide a way to switch the mic during the meeting without refreshing the page? This is a must-have. I created a feature request for this: #3548

[vincentfretin]

I just reproduced another mic issue on Hubs with Chrome Ubuntu (the bug didn't happen with Firefox) on my Dell XPS, related to the RTCPeerConnection state to disconnected (as you can see in RTC Panel, Send Transport, state). If I switch off the wifi (I'm not wired connected either), wait a second or two, then switch back on the wifi, Chrome went to Send Transport state disconnected, and it didn't reconnect, the state kept in disconnected state, the mic was still working, I saw the volume in the icon.
In Firefox (I was connected with both browsers), the Sender Transport state switched from "disconnected" to "new" successfully. but I couldn't hear my voice from the Chrome session (normal because the Send Transport is disconnected on Chrome).
With my laptop connected to the dock station, the wifi off and on can happen sometimes if I touch the usb of my dock station that trigger a disconnection and reconnection to the dock station, RTCPeerConnection goes into disconnected state, I saw it.

Currently I identified 3 cases where the mic can stop working during the meeting on Chrome Ubuntu or macOS

[vincentfretin]

For the RTCPeerConnection that goes into the disconnected state, I fixed it in my project for naf-janus-adapter (see PR mozilla/naf-janus-adapter#107)
Chrome and Firefox behave very differently, see my comment in this commit.
For Hubs and naf-dialog-adapter, I thought we could do this.reconnect() in the this._protoo.on("disconnected" but this doesn't fix it, Send Transport stays disconnected. I don't know a thing about protoo so I'm not sure the on disconnected here corresponds to a RTCPeerConnection that changed its iceConnectionState to disconnected... So I'll leave someone else looking into it. Be careful to have some sort of debounce because all RTCPeerConnection (one for each participant) will go in iceConnectionState=disconnected (like we had with performDelayedReconnect in naf-janus-adapter).
If you want to work on it, enable the RTC Panel in Preferences/Misc, run hubs with npm run dev and open Firefox and Chrome in a room and switch off and on your Wi-Fi.

[vincentfretin]

From my logs with naf-janus-adapter, on Chrome, I got a RTCPeerConnection (I don't know if it's the publisher or subscriber) in state disconnected, the web socket didn't close, I was wired connected and didn't notice that network was down at all, for me it wasn't. So I guess it can happen that RTCPeerConnection goes into disconnected state, even if you still have the network... I know it can maybe be something on the server side with janus or my server and may not apply to new dialog medialog on AWS, but maybe it can.
So maybe an alternative to just doing disconnect and reconnect in this case may be just to recreate only the publisher or subscriber if the web socket didn't close.

[vincentfretin]

Nah, forget about the RTCPeerConnection disconnected state, @brianpeiris kindly gave me a link to this article https://blog.mozilla.org/webrtc/ice-disconnected-not/
The real issue I have with naf-janus-adapter (yes I know it's not used by Hubs, thus not maintained anymore) is with Chrome where the websocket time out but never reconnect mozilla/naf-janus-adapter#107 (comment)

But you should do some tests with Hubs and the RTC Panel open, you have similar issues when the network connection is off and on for a second.

[vincentfretin]

I reproduced an issue with Chrome that couldn't hear anyone anymore after the network switched from wired to Wi-Fi (I was connected with both, and just stopped wired network) and found the culprit. On Chrome
enableChromeAEC(sceneEl.audioListener.gain); is executed that create a local RTCPeerConnection.
I modified the code to log the ice state change for this local RTCPeerConnection with

  outboundPeerConnection.addEventListener("icecandidate", e => {
    inboundPeerConnection.addIceCandidate(e.candidate).catch(onError);
  });
  outboundPeerConnection.addEventListener("iceconnectionstatechange", e => {
    console.warn("outboundPeerConnection changed to", outboundPeerConnection.iceConnectionState);
  });

  inboundPeerConnection.addEventListener("icecandidate", e => {
    outboundPeerConnection.addIceCandidate(e.candidate).catch(onError);
  });
  inboundPeerConnection.addEventListener("iceconnectionstatechange", e => {
    console.warn("inboundPeerConnection changed to", inboundPeerConnection.iceConnectionState);
  });

When the network switched from wired to Wi-FI I got

inboundPeerConnection changed to disconnected
outboundPeerConnection changed to disconnected

and never changed back to connected after 30s.

I was still connected, but I couldn't hear anyone anymore.
Executing again in the console
enableChromeAEC(AFRAME.scenes[0].audioListener.gain)
fixed the audio.

[vincentfretin]

I'm wondering if the enableChromeAEC hack is still necessary on Chrome desktop. I commented it for now in my project.

[vincentfretin]

The enableChromeAEC hack seems to be always needed. I had echo like crazy without it.
I did a fix in PR #3604 for the case where the network changed from wired to Wi-Fi to fix the issue I explained above.

[vincentfretin]

On iOS Safari and iOS Chrome (mic is accessible since iOS 14.3, wasn't the case in iOS 14.2), when you unplug the headphone the AudioContext goes into the suspended state, see my comment
#338 (comment) if you want to fix it in Hubs.

[vincentfretin]

On Safari iOS 14.3, if you are in a room using the mic and you answer a Skype call, the mic audioTrack is muted, when the call ends the audioTrack is unmuted but is not working. This is a known issue of Safari iOS https://bugs.webkit.org/show_bug.cgi?id=213853
Twilio did a workaround for this issue twilio/twilio-video.js#1232 (https://github.com/twilio/twilio-video.js/blob/68091fa13417b60d436455cf3c2f4402620d30d9/lib/media/track/mediatrack.js#L282-L311) that recreates the audio track when detecting that the audio track is muted and then unmuted when the app comes back in foreground (document visibilitychange event).

Here is my current code in my app to fix the issue, you can adapt it for Hubs if you want (mainly replace "window.app" by "this" to use it in your React component)

window.app.mutedStateChanged = async () => {
  if (window.app.audioTrack && window.app.audioTrack.muted) {
    // On Safari iOS after answering Skype call muted becomes true,
    // when we end the call muted comes back to false, but the mic is
    // still not working, we don't see or hear the other participants
    // talking. So the workaround is to recreate the audio stream.
    if (!window.app.reconnectIfUnmuted) {
        window.app.addToPresenceLog({ type: "log", body: `Your mic is muted from the tab.` });
        // TODO add disabled attribute to the mic button and show it like it is muted
    }
    window.app.reconnectIfUnmuted = true;
  } else {
    if (window.app.reconnectIfUnmuted) {
      window.app.reconnectIfUnmuted = false;
//      window.app.addToPresenceLog({ type: "log", body: `recreating audio track` });
      await window.app.recreateAudioStream(); // be careful to put back the onmute and onunmute handler on the new audioTrack
        // TODO remove the disabled attribute to the mic button, show it like it is unmuted if it was enabled previously
    }
  }
//      window.app.addToPresenceLog({ type: "log", body: `audioTrack enabled ${window.app.audioTrack.enabled} muted ${window.app.audioTrack.muted}` });
};
window.app.audioTrack.onmute = window.app.mutedStateChanged;
window.app.audioTrack.onunmute = window.app.mutedStateChanged;

Be sure to communicate to the other participants that the user is muted if the audioTrack is muted=true (not only if the publisher RTCPeerConnection stream audio track is enabled=false)

[vincentfretin]

Btw, the AudioContext that goes into suspended state on Safari iOS and Chrome iOS is sometimes triggered when another tab use the mic, not just unplugging the headphones or bluetooth headset reconnection.

[vincentfretin]

See issue with coturn server with Firefox > 86 here #4187

[vjFred]

Last week i have seeing a different symptom, on HCEnterprise ( running C4large, 2x2) the audio that coming from avatars looks distorted, everybody on a room begin to hear sound drops and changes in time speed of the audio chat like noise, until finally just dropped off, this was tested on several devices and browser, on a room with only 5 users, low poly, and just one streaming running, ( that sometimes just get off ) 👎