Audio sources

[pzeepzee]

I will like to use my headphones plugged or wired during a video call while also doing a screen recording, however, there's no frontend UI to select input options. Does the app programmatically default to external audio inputs/outputs (headphones wireless /non-wireless) during recording or does this have to be selected first of all in 'Sound' under Systems preferences?
Thanks

[Mnpn]

Hi! The app does not currently record any microphone/input, just system audio (output) -- is this something you'd like to see added?

[pzeepzee]

Yes, pretty pls. I would imagine a good number of people will have a headset/microphone on while using this app for screen recording. Thanks

[sihekuang]

I'd like to have mic recording too. I actually have started exploring the implementation but no luck so far. Having problem directing input(mic) audio stream to audio asset writer for output. I think the most difficult part was to convert AVAudioPCMBuffer to CMSampleBuffer correctly for the asset writer. Perhaps I am on a wrong path and there's better way to append AVAudioPCMBuffer to asset writer? I am using AVAudioEngine btw

[Mnpn]

Hmm, I think the most time-consuming thing would be figuring out how to merge both microphone and system audio into one audio track. Using AVAudioEngine sounds like the way to go, though. I haven't looked into it very thoroughly so forgive me if I am completely wrong, but I assume you'd need to create a mixer node which takes both the CMSampleBuffers coming from ScreenCaptureKit and mic data (having two AVAudioEngine input nodes?), and writes it to the AVAssetWriterInput already in Azayaka (awInput.append()). I can imagine it being difficult to pass the ScreenCaptureKit audio buffers into an AVAudioEngine input node.

Another solution could be to use two AVAssetWriterInputs, passing mic input in one and keeping the system audio one as-is, but that would create two audio tracks in the output file. It would be an easier solution, and is one I managed to get working in a short amount of time, though I assume that is not what you're after, right? (Perhaps we could make a checkbox titled "Write mic input to separate track" and have both as an option? I can see it being useful having them on separate tracks too.)

[sihekuang]

I think you are completely on the right track.
How you described the first approach was something I've tried. However, I was getting clear mic input but very choppy system output. (should've kept that branch but I deleted it 😝 ).

I am currently exploring the second method you are describing. The use of AudioEngine is only for mic input and nothing more at the moment. I am converting AVAudioPCMBuffer into CMSampleBuffer, but no luck so far.
But I did find another repo trying to do AVAudioPCMBuffer into CMSampleBuffer in this repo PLAudioMixer. Studying up that right now hoping to migrate that over, it is quite old. Last supported swift is 3.2

[Mnpn]

I've pushed up my implementation of writing microphone data to a branch called mic-track if you'd like to have a look! It seems that Quick Look in macOS plays both tracks at once, but a player like IINA requires selecting them separately.

I've not looked into merging these into one track thoroughly yet, but that seemed quite a bit more complicated, haha

[sihekuang]

@Mnpn I think your branch is very useful as-is! 😄
I opened a pull request to consolidate audio only and screen capture scenarios to use the same AVAssetWriter. Let me know what you think

[Mnpn]

Hi again! I've just pushed an update with the microphone recording to a separate audio track. It uses the currently selected input device in macOS. Let's keep this issue open to track discussions about recording both system audio and microphone input to the same track! (and let me know if you have any issues with the implementation :-) )

[ankurdahama1997]

Awesome stuff folks.

When recording mic audio along with the video, the change in mic input device stops the mic audio recording. Any recommended solutions?