Audio device switching

[padenot]

We need to define the behaviour of what happens when users change devices, either via the audio configuration menu, or by plugging or unplugging a physical device.

Here is a proposal that allows implementing something flexible:

First, we need to implement the device change notification callback on all supported platforms.

Next, I propose that inside the audio backends, we call the user callback before changing the stream configuration. We also need to make sure that no audio callback will be called during the time where the device change callback is called.

Then, we can modify the signature of the callback, and make it like so:

typedef enum {
  CUBEB_INPUT_DEVICE_CHANGED = 1 << 0;
  CUBEB_OUTPUT_DEVICE_CHANGED = 1 << 1;
} cubeb_device_change_information;

typedef enum {
  CUBEB_USE_DEFAULT_INPUT = 1 << 0;
  CUBEB_USE_DEFAULT_OUTPUT = 1 << 1;
  CUBEB_STOP_INPUT = 1 << 2;
  CUBEB_STOP_OUTPUT = 1 << 3;
} cubeb_device_change_action;

typedef cubeb_device_change_action (* cubeb_device_changed_callback)(void * user_ptr, cubeb_device_change_information);

If it's ok to switch to the default devices, the user can return CUBEB_USE_DEFAULT_INPUT|CUBEB_USE_DEFAULT_OUTPUT, and the switch will occur. If the user prefers to kill one side of the stream, passing the appropriate enum member turns the stream into an half-duplex stream, or stops the stream if both CUBEB_STOP_INPUT and CUBEB_STOP_OUTPUT are or-ed and returned. Users can then react to the device change, for example by destroying this stream and creating a new one.

It sounds a bit dangerous to allow calling cubeb APIs from the device change notification callback.

For gecko, this allows implementing the following:

• Having a virtual default device in getUserMedia that uses the default device for the system, and continues across default device changes.
• Being able to keep playing audio in case of audio output device changes (for HTMLMediaElement or Web Audio API)
• Being able to send the "ended" event on the MediaStream associated with the getUserMedia call, as specced, and stopping the capture. This informs the user that something has happened and that getUserMedia should probably be called again.
• Being able to react to the case where the last audio device has been removed.

[kinetiknz]

Quick notes:

• reporting input and output device disconnections at the same time may be difficult since the OS notifications can be async and possibly arrive at different time for each device
• I don't understand why switching to the default is treated as a special action but switching to another non-default device requires the caller to destroy and recreate the stream.

There are several types of changes we can signal:

• where we can no longer use the current device and some action is required
• where the configuration changes but the stream continues running
  (e.g. PA "seamlessly" moving a stream to a different device)

For the cases where we can no longer use the current device, there are several reasons. Using WASAPI as an example:

• device state changes (IMMNotificationClient): active, disabled, not present, unplugged
• device session disconnection (IAudioSessionEvents): removed, server shutdown, format changed, log off, disconnected, exclusive mode override
  (note that we don't listen for these right now, but we need to to handle some types of device loss, and I have a patch floating around for this)

The current enum values cover device disconnections, but don't have a way to signal that the same device may still be available but require reconfiguration to continue using.

It's also probably worth considering how to address the issues raised in https://bugzilla.mozilla.org/show_bug.cgi?id=1198591

[padenot]

reporting input and output device disconnections at the same time may be difficult since the OS notifications can be async and possibly arrive at different time for each device

True. It's however possible. Worst case we're re-configuring a bit too much.

I don't understand why switching to the default is treated as a special action but switching to another non-default device requires the caller to destroy and recreate the stream.

My thinking on this one was that you don't need to make any decision about which device to use if your policy is to let it switch to the new default device, so we can return from the callback and proceed. Also it's a rather common case (plugging in an USB headset on OSX changes the system default), and it let cubeb be faster about the switch.

Choosing a particular device (or device pair) often requires re-enumerating the available devices (because it's likely the list changed), which can be blocking and/or require user intervention to pick the right one. At this point, it's best to kill the stream and make the software do something.

There are several types of changes we can signal:

• where we can no longer use the current device and some action is required
• where the configuration changes but the stream continues running (e.g. PA "seamlessly" moving a stream to a different device)

Agreed, that's very useful and I missed it. Firefox uses the information that the device changed to reset the AEC for example.

• device state changes (IMMNotificationClient): active, disabled, not present, unplugged

I think we only care about the transition available -> unavailable here.

• device session disconnection (IAudioSessionEvents): removed, server shutdown, format changed, log off, disconnected, exclusive mode override (note that we don't listen for these right now, but we need to to handle some types of device loss, and I have a patch floating around for this)

Breaking the list down:

• Removed we can handle via IMMNotificationClient
• Server shutdown we can handle via the state error callback. This conveys the meaning that it's quite serious.
• Format change we can reconfigure the stream so that the client does not see the difference
• Disconnected/Log off we don't care too much about, maybe just kill everything, fire the state callback or do nothing
• Exclusive mode override we can just fire the state callback with an error

So, there is three reasons for getting the device notification:

1. Something changed, but no action is necessary (informative notification)
2. Something changed, you might tell cubeb to switch to the default devices, or you might close a side or both sides of the stream, and re-open (or not) a new stream.
3. Something changed, and audio is not available: no audio device available anymore, session closed, audio server crashed, etc. It's unnecessary to try again, put everything in error mode.

An updated proposal

typedef enum {
  CUBEB_INPUT_DEVICE_CHANGED = 1 << 0;
  CUBEB_OUTPUT_DEVICE_CHANGED = 1 << 1;
  CUBEB_DEVICE_SWITCHED = 1 << 2;
  CUBEB_DEVICE_RECONFIGURED = 1 << 3;
  CUBEB_DEVICE_UNAVAILABLE = 1 << 4;
  CUBEB_STREAM_ERROR = 1 << 5;
} cubeb_device_change_information;

typedef enum {
  CUBEB_USE_DEFAULT_INPUT = 1 << 0;
  CUBEB_USE_DEFAULT_OUTPUT = 1 << 1;
  CUBEB_STOP_INPUT = 1 << 2;
  CUBEB_STOP_OUTPUT = 1 << 3;
} cubeb_device_change_action;

typedef cubeb_device_change_action (* cubeb_device_changed_callback)(void * user_ptr, cubeb_device_change_information);