Issue handling default audio device reconnection + solution

[fabio-garcia]

Currently BBB has an "instant reconnect" approach for handling audio devices that had become inactive, be it by physical disconnection or changing client OS settings. For context, when using removable devices such as USB microphone the OS (at least on Windows) serves them as the new default device. Upon disconnection, the OS fallbacks to another device as the default for any application running. We need to account for that change when handling switching physical device connections by properly changing the audio input stream.

In BBB HTML5 client we treat the default audio device with deviceId set exactly as default as reported by the browser. So when a disconnection from a removable microphone source happens, the audio should be immediatly switched to the now current reported default device, which could be any preferred option reported by the OS to the browser. This device change is invisible to the audio handler because when we enumerate the available devices the deviceId of the default source is always reported as the string default. This leads to three problems:

• Upon disconnection, another device is marked as default, and all incoming audio that comes through it is lost since we don't switch the media source. Upon reconnection, the reconnected device is seen by BBB UI as the default because we enumerated all of them again, but we also never reconnect the audio through it.
• Captions are generated through the reconnected device but, because we never properly set it as the input stream, BBB never captures its audio but keeps generating its captions.
• Ultimately, if there is only a single device reported by the browser and it disconnected and reconnected, the deviceId default never changes and we end up without audio. This is also true for faulty USB devices that sometimes reconnect on their own.

This can be observed by simulating a physical disconnection on the current (default) audio input device, then reconnecting it. Its audio never comes back and we still generate captions for it. See the following video:
https://github.com/user-attachments/assets/a590b48b-dc2c-4456-ba83-67450fa4d155

As soon as we disconnect the default device, the audio is lost until user intervention. Audio captions are generated through the newly set default device by the OS.

Steps to reproduce:

1. Join meeting
2. Join audio with a removable audio source as the OS default, such as an USB microphone.
3. Pull out the USB cable from it
4. Reconnect it but don't change the audio source from the BBB UI
5. Notice audio is lost and captions are still being generated through it

The problem lies with how the media stream is handled upon noticing a disconnection. This comes down to the following method in the HTML5 client at bigbluebutton-html5/imports/ui/services/audio-manager/index.js:

  handleMediaStreamInactive(stream) {
    logger.warn({
      logCode: 'audiomanager_stream_inactive',
      extraInfo: {
        currentStreamData: MediaStreamUtils.getMediaStreamLogData(this.inputStream),
        streamData: MediaStreamUtils.getMediaStreamLogData(stream),
      },
    }, 'Audio stream has become inactive');

    if (stream === this.inputStream) {
      this.inputStream = null;

      // Reset the input device (and consequently the stream) if it's inactive
      if (this.isUsingAudio()) {
        this.liveChangeInputDevice(DEFAULT_INPUT_DEVICE_ID).catch((error) => {
          logger.error({
            logCode: 'audiomanager_stream_inactive_device_reset_failed',
            extraInfo: {
              errorName: error.name,
              errorMessage: error.message,
            },
          }, `Failed to reset input device after stream became inactive: ${error.message}`);
        });
      }
    }
  }

Since inputDeviceId is always default, the sudden change of the default device isn't noticed by the handler, which leads to it reconnecting to the same default deviceId resulting in the observed behavior. After fiddling with it I've come to the following code snippet as a solution:

  handleMediaStreamInactive(stream) {
   logger.warn({
     logCode: 'audiomanager_stream_inactive',
     extraInfo: {
       currentStreamData: MediaStreamUtils.getMediaStreamLogData(this.inputStream),
       streamData: MediaStreamUtils.getMediaStreamLogData(stream),
     },
   }, `Audio stream has become inactive on ${getStoredAudioInputDeviceId()}`);

   if (stream === this.inputStream) {
     const currentDeviceId = this.inputDeviceId ?? 'none';
     const originalLabel = this.inputStream?.getAudioTracks()[0]?.label;
     this.inputStream = null;

     if (this.isUsingAudio() && originalLabel) {
       const checkDevice = async () => {
         try {
           await this.enumerateDevices();
           const found = this.inputDevices.find(d => d.label === originalLabel);
           // Abort if the device was manually changed
           if (this.inputStream && this.inputDeviceId !== currentDeviceId) {
             logger.info({ logCode: 'audiomanager_stream_changed' }, `Audio stream changed from ${currentDeviceId} to ${this.inputDeviceId}, abort waiting`);
             return;
           }
           if (found && this.inputStream) {
             // Reset the stream as the audio device ID is always 'default' between connections and the change won't be detected by the handler
             this.inputStream.getTracks().forEach(track => track.stop());
             this.inputStream = null;
             await this.liveChangeInputDevice(DEFAULT_INPUT_DEVICE_ID);
             logger.info({ logCode: 'audiomanager_stream_reconnected' }, `Reconnected to: ${originalLabel}, ${getStoredAudioInputDeviceId()}`);
           } else {
             // Change to the new reported default and try again because audio captions start generating on top of the new "default" named device
             if (!this.inputStream) {
               await this.liveChangeInputDevice(DEFAULT_INPUT_DEVICE_ID);
               logger.info({ logCode: 'audiomanager_stream_fallback' }, `Original device not present, using fallback`);
             }
             setTimeout(checkDevice, 2000);
           }
         } catch (error) {
           setTimeout(checkDevice, 2000);
         }
       };
       setTimeout(checkDevice, 2000);
     }
   }
  }

In this snippet we properly get rid of the stale audio track before falling back to any new default device reported by the OS for the browser. By implementing an async function and waiting, we keep track of the old default device in case it reconnects, which could lean towards a smoother experience switching devices without user intervention. If the user intervenes and stops using the default device, we abort waiting for the old device to reconnect.
This would allow a seamless transition between two different devices marked as the same "default" device ID and also solves the issue with captions as of recently they restart capturing for the current device every 15s.

The last concern was timing out the solution. Because we're waiting for stats from the browser, when changing devices we first get their full ID hash, then it gets reported as default again. These setTimeout are carefully placed and gives us enough time to guarantee for that to happen. I also added some info logs while debugging informing the status of the reconnection.

I've confirmed this behavior on some old 2.3.1 servers all the way up to a 3.0.10 fresh install. It also happens with LiveKit enabled.

[prlanzarin]

@fabio-garcia Can you please submit a pull request with this so it can be iterated upon?