[Mac] Allow audio device selection #21

hiroshihorie · 2022-03-10T18:25:44Z

Goals

Audio device list & selection at runtime.
Detect and switch device when default device changes at runtime. (Connect AirPods etc)
Feed audio sample buffers directly.

bypassAudioProcessing is temporarily removed, but I will make it available again.
-> Now available again.

davidzhao

incredible work @hiroshihorie!

For the new files such as: RTCAudioDeviceModule.mm, are those from somewhere else in the webrtc tree?

davidzhao · 2022-03-14T05:27:48Z

modules/audio_device/mac/audio_device_mac.cc

@@ -867,12 +864,9 @@ int32_t AudioDeviceMac::RecordingDeviceName(uint16_t index,
  }

  memset(name, 0, kAdmMaxDeviceNameSize);
-
-  if (guid != NULL) {


can anything else call this function and pass in NULL for RecordingDeviceName ?

RecordingDeviceName is supposed to populate name and guid of the device,
but populating guid was not implemented in their code.

So I implemented to return the AudioDeviceID which is not technically guid but
it works for identifying the device.

hiroshihorie · 2022-03-14T07:40:27Z

Yes there is a webrtc::AudioDeviceModule class.
https://chromium.googlesource.com/external/webrtc/stable/webrtc/+/15a8a6e7edb2c8df7f5b21e759ace213a4ab7292/modules/audio_device/include/audio_device.h#19

davidzhao · 2022-03-14T18:02:00Z

Yes there is a webrtc::AudioDeviceModule class. https://chromium.googlesource.com/external/webrtc/stable/webrtc/+/15a8a6e7edb2c8df7f5b21e759ace213a4ab7292/modules/audio_device/include/audio_device.h#19

But is the ObjC wrapper class from the tree? or is that a new file you authored?

hiroshihorie · 2022-03-14T18:46:05Z

Yes there's a few new files I authored.

for RTCAudioDeviceModule, In the future,
I might want to add ability to create virtual devices.
RTCAudioDeviceModule.addVirtualDevice(...)

then user can call the existing (implemented in this PR) RTCAudioDeviceModule.setInputDevice(...),
to switch to their virtual device.

This virtual device class will conform to a new RTCVirtualAudioDevice protocol (or base class) so users can feed their own CMSampleBuffers through here.

This is just a thought for design 🙂

I want to achieve unified design that will work for both iOS / Mac platforms, but the underlying audio system is pretty different for each platform. So I'm not sure if it's possible.

Another concern is mixing the audio, for example game streaming, user would like to speak while having game audio. ReplayKit delivers audio buffer separately for app and mic.
https://developer.apple.com/documentation/replaykit/rpsamplebuffertype
So it needs to get mixed somehow... still investigating ...

davidzhao · 2022-03-14T18:47:32Z

Got it, for the new files, could you add a LiveKit copyright in it? This will help differentiate which files originated from upstream vs us.

davidzhao · 2022-03-14T18:48:14Z

This virtual device class will conform to a new RTCVirtualAudioDevice protocol (or base class) so users can feed their own CMSampleBuffers through here.

This is just a thought for design 🙂

I want to achieve unified design that will work for both iOS / Mac platforms, but the underlying audio system is pretty different for each platform. So I'm not sure if it's possible.

YES! This is going to be awesome!!

Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail.

hiroshihorie · 2022-05-27T10:32:41Z

RTC's thread checks were failing so improved thread handling.
All access to ADM methods are from worker thread now.

RTC_DCHECK_RUN_ON(&main_thread_checker_); checks whether its called from the same thread that the ADM was created on (which must be created on worker thread).
https://github.com/webrtc-sdk/webrtc/blob/main/modules/audio_device/audio_device_buffer.cc#L135
https://github.com/webrtc-sdk/webrtc/blob/main/modules/audio_device/audio_device_buffer.cc#L149

hiroshihorie · 2022-05-29T09:18:53Z

Additional required logic

Active device (lost) disconnected ✅

Current: Nothing happens (audio input/output stops)
Expected: Switch to default device and continue input/output.

OS's default device changed ✅

Current: Audio input/output continues on the same device even if default device is changed.
Expected: if default device is switched by the OS, change input/output to the new default device.

cloudwebrtc · 2022-07-27T10:59:30Z

hey @hiroshihorie, It seems that changing the PR from target m97 to m104 introduces additional diffs, should we still target m97 after merging we get the minimal diff for the patch?

hiroshihorie · 2022-07-27T11:01:26Z

hey @hiroshihorie, It seems that changing the PR from target m97 to m104 introduces additional diffs, should we still target m97 after merging we get the minimal diff for the patch?

Yup, I changed the target which wasn't a good idea..

cloudwebrtc · 2022-07-27T15:12:13Z

The code looks good.

davidzhao · 2022-07-28T16:36:01Z

@hiroshihorie shall we merge this as well as into 104?

* first attempt * remove unused dep * init playout / recording * use AudioDeviceID as guid * switch device method * equality * default device * `isDefault` property * dont format default device name * type param * bypass * refactor * fix * append Audio to thread labels * ref * lk headers * low level apis * fix thread checks Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail. * switch to default device when removed * close mixerManager if didn't switch to default device * default audio device switched * expose devices update handler * fix ios compile * fix bug: don't always recreate RTCAudioDeviceModule

* [Mac] Allow audio device selection (#21) * first attempt * remove unused dep * init playout / recording * use AudioDeviceID as guid * switch device method * equality * default device * `isDefault` property * dont format default device name * type param * bypass * refactor * fix * append Audio to thread labels * ref * lk headers * low level apis * fix thread checks Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail. * switch to default device when removed * close mixerManager if didn't switch to default device * default audio device switched * expose devices update handler * fix ios compile * fix bug: don't always recreate RTCAudioDeviceModule * handle guid. Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

allow listen-only mode in AudioUnit, adjust when category changes release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) * use `AVAudioSession` defaults * remove isRecordingEnabled feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) * feat: add audio device changes detect for windows. * Update audio_device_core_win.cc fix iOS/macOS/Android compile. fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * progress * tweak * clean * simplify audio unit restart call to SetupAudioBuffersForActiveAudioSession() might not be needed since sample rate won't change during restart. This might help reduce the unwanted noise when restarting audio unit. * clean Stop recording on mute (turn off mic indicator) (#55) * initial impl * more comments * more comment * adjust indent * comments Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * first attempt * remove unused dep * init playout / recording * use AudioDeviceID as guid * switch device method * equality * default device * `isDefault` property * dont format default device name * type param * bypass * refactor * fix * append Audio to thread labels * ref * lk headers * low level apis * fix thread checks Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail. * switch to default device when removed * close mixerManager if didn't switch to default device * default audio device switched * expose devices update handler * fix ios compile * fix bug: don't always recreate RTCAudioDeviceModule * handle guid. Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Fix external audio processor sample rate calculation (#108) Expose remote audio sample buffers on RTCAudioTrack (#84) Fix memory leak when creating audio CMSampleBuffer #86 Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net>

Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization 7454824 * allow listen-only mode in AudioUnit, adjust when category changes (#2) * release mic when category changes (#5) * Change defaults to iOS defaults (#7) * Sync audio session config (#8) * feat: support bypass voice processing for iOS. (#15) * Remove MacBookPro audio pan right code (#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (#29) * feat: add audio device changes detect for windows. (#41) * fix Linux compile (#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * Stop recording on mute (turn off mic indicator) (#55) * Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) * Allow custom audio processing by exposing AudioProcessingModule (#85) * Expose audio sample buffers for Android (#89) * feat: add external audio processor for android. (#103) * android: make audio output attributes modifiable (#118) * Fix external audio processor sample rate calculation (#108) * Expose remote audio sample buffers on RTCAudioTrack (#84) * Fix memory leak when creating audio CMSampleBuffer #86 ## 3. Simulcast/SVC support for iOS/Android. b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. 9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. 841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com>

allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk#2) release mic when category changes (webrtc-sdk#5) Change defaults to iOS defaults (webrtc-sdk#7) Sync audio session config (webrtc-sdk#8) feat: support bypass voice processing for iOS. (webrtc-sdk#15) Remove MacBookPro audio pan right code (webrtc-sdk#22) fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk#29) feat: add audio device changes detect for windows. (webrtc-sdk#41) fix Linux compile (webrtc-sdk#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk#52) Stop recording on mute (turn off mic indicator) (webrtc-sdk#55) Cherry pick audio selection from m97 release (webrtc-sdk#35) [Mac] Allow audio device selection (webrtc-sdk#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk#80) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io>

hiroshihorie added 8 commits March 11, 2022 03:24

first attempt

3c17eff

remove unused dep

42029a7

init playout / recording

4c929eb

use AudioDeviceID as guid

efc3c32

switch device method

a5042dc

equality

dfd4ace

default device

eda19e1

isDefault property

507913c

hiroshihorie changed the title ~~WIP: Expose audio device module~~ [Mac] Allow device selection Mar 12, 2022

hiroshihorie added 3 commits March 13, 2022 03:02

dont format default device name

c08e4cf

type param

e063d93

bypass

aba2227

hiroshihorie changed the title ~~[Mac] Allow device selection~~ [Mac] Allow audio device selection Mar 12, 2022

hiroshihorie requested a review from davidzhao March 13, 2022 09:18

refactor

149c99f

davidzhao approved these changes Mar 14, 2022

View reviewed changes

hiroshihorie added 3 commits March 14, 2022 17:39

fix

a00a74c

append Audio to thread labels

e5410c8

ref

c4f1305

hiroshihorie added 4 commits March 15, 2022 04:17

lk headers

a35ce56

low level apis

bdf5e7e

Merge branch 'm97_release' into expose-audio-device-module

9de0e4d

fix thread checks

bda204c

Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail.

hiroshihorie changed the base branch from m97_release to m104_release July 22, 2022 03:57

hiroshihorie changed the base branch from m104_release to m97_release July 27, 2022 13:14

cloudwebrtc mentioned this pull request Jul 29, 2022

[WIP] Add selectAudioOutput method. flutter-webrtc/flutter-webrtc#1030

Merged

13 tasks

hiroshihorie merged commit f48bd24 into m97_release Jul 31, 2022

hiroshihorie deleted the expose-audio-device-module branch July 31, 2022 21:39

cloudwebrtc mentioned this pull request Jun 12, 2023

update to m114 #76

Merged

cloudwebrtc mentioned this pull request Jul 7, 2023

Migrate to m114 livekit/client-sdk-swift#225

Merged

cloudwebrtc mentioned this pull request May 21, 2024

Update to m125. #119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mac] Allow audio device selection #21

[Mac] Allow audio device selection #21

hiroshihorie commented Mar 10, 2022 •

edited

Loading

davidzhao left a comment

davidzhao Mar 14, 2022

hiroshihorie Mar 14, 2022 •

edited

Loading

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented May 27, 2022

hiroshihorie commented May 29, 2022 •

edited

Loading

cloudwebrtc commented Jul 27, 2022 •

edited

Loading

hiroshihorie commented Jul 27, 2022

cloudwebrtc commented Jul 27, 2022

davidzhao commented Jul 28, 2022

[Mac] Allow audio device selection #21

[Mac] Allow audio device selection #21

Conversation

hiroshihorie commented Mar 10, 2022 • edited Loading

Goals

davidzhao left a comment

Choose a reason for hiding this comment

davidzhao Mar 14, 2022

Choose a reason for hiding this comment

hiroshihorie Mar 14, 2022 • edited Loading

Choose a reason for hiding this comment

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented May 27, 2022

hiroshihorie commented May 29, 2022 • edited Loading

Additional required logic

Active device (lost) disconnected ✅

OS's default device changed ✅

cloudwebrtc commented Jul 27, 2022 • edited Loading

hiroshihorie commented Jul 27, 2022

cloudwebrtc commented Jul 27, 2022

davidzhao commented Jul 28, 2022

hiroshihorie commented Mar 10, 2022 •

edited

Loading

hiroshihorie Mar 14, 2022 •

edited

Loading

hiroshihorie commented May 29, 2022 •

edited

Loading

cloudwebrtc commented Jul 27, 2022 •

edited

Loading