Unable to play audio while recording a video in iOS

[mask477]

Bug Report

To Do First

• [ YES ] Did you try latest release?
• [ YES ] Did you try master?
• [ YES ] Did you look for existing matching issues?

Platforms

No issue faced in Android

It happens only in iOS.

Versions

• Android: Used different versions of android, no issues faced
• iOS:
• react-native-camera: 3.44.3
• react-native: 0.64.2
• react: 17.0.1

Description/Current Behaviour

This happens with iOS only, it's working fine in Android. I want to play an audio while recording a video so that the audio playing on the phone's speaker can be captured through the microphone that is being used by the camera. I play the audio and as soon as I call the recordAsync() function, the audio stops and the recording starts, at first I though that it's impossible to do this in iOS due to different permission policies in iOS as compared to Android, but I was provided with an native iOS version of the app by the client that is doing the same thing, so permissions is not an issue.

Expected Behavior
Expected to allow the audio to keep playing while I record the video

Steps to Reproduce

Try playing any audio from a file or a URL using react-native-sound and call recordAsync() from react-native-camera,

Additionals

I'm using react-native-sound for playing the audio, as the audio is played through a URL, This is my RNCamera Component with props:

    <RNCamera
          style={styles.preview}
          type={RNCamera.Constants.Type.back}
          flashMode={RNCamera.Constants.FlashMode.on}
          captureAudio={true}
          androidCameraPermissionOptions={{
            title: 'Permission to use camera',
            message: 'We need your permission to use your camera',
            buttonPositive: 'Ok',
            buttonNegative: 'Cancel',
          }}
          androidRecordAudioPermissionOptions={{
            title: 'Permission to use audio recording',
            message: 'We need your permission to use your audio',
            buttonPositive: 'Ok',
            buttonNegative: 'Cancel',
          }}>
          {({camera, status, recordAudioPermissionStatus}) => {
            if (showFilenameModal) {
             // show a modal to save the video file with a user entered name
            }
            if (status !== 'READY')
              return (
                <View
                  style={{
                    flex: 1,
                    width: globalStyles.WIDTH,
                    backgroundColor: 'black',
                    justifyContent: 'center',
                    alignItems: 'center',
                  }}>
                  <Text>Waiting</Text>
                </View>
              );
            return recording || processing ? (
              <IconButton
                title={'Stop'}
                icon={'stop'}
                disabled={processing}
                onPress={() => {
                  console.log('Stop Recording');
                  camera.stopRecording();
                }}
              />
            ) : (
              <View
                style={{
                  display: 'flex',
                  flexDirection: 'row',
                  justifyContent: 'space-around',
                  marginBottom: 10,
                }}>
                  {/* Button to record video without playing the audio */}
                <IconButton
                  title={'Record'}
                  icon="record"
                  onPress={() => this.startRecording(camera, false)}
                />
                  {/* Button to record video while playing the audio */}
                <IconButton
                  title={'Record with Audio'}
                  icon="record"
                  showSound={true}
                  onPress={() => this.startRecording(camera)}
                /> 
              </View>
            );
          }}
        </RNCamera>

This is how I start the recording:

startRecording = async function (camera, withAudio = true) {
    const {audioPlayer, setVideoStatus} = this.props;
    audioPlayer.stopAudio();
    setVideoStatus({
      recording: true,
      processing: false,
      uri: null,
    });

    if (withAudio) audioPlayer.playAudio();

    const video = await camera.recordAsync({
      quality: RNCamera.Constants.VideoQuality['720p'],
      // I've tried this with different flag combination i.e:
      // captureAudio: true / false,
      // keepAudioSession: true / false,
    });

    console.log(video.uri.substr(7, video.uri.length), video.codec);

    audioPlayer.stopAudio();
    setVideoStatus({
      recording: false,
      processing: true,
    });

    this.setState({
      showFilenameModal: true,
      uri: video.uri,
    });
  };

This is my info.plist

  <key>NSCameraUsageDescription</key>
  <string>Allow app to open camera to record video</string>
  <key>NSPhotoLibraryAddUsageDescription</key>
  <string>Allow app to save videos to library</string>
  <key>NSAppleMusicUsageDescription</key>
  <string>Allow app to use media library for audio</string>
  <key>NSLocationAlwaysUsageDescription</key>
  <string>Allow app to use location for getting current state</string>
  <key>NSLocationWhenInUseUsageDescription</key>
  <string>Allow app to use location for getting current state</string>
  <key>NSMicrophoneUsageDescription</key>
  <string>Allow app to use microphone to record video with audio</string>

Just in case these are the permissions in my AndoroidManifest.xml:

    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-permission android:name="android.permission.RECORD_AUDIO"/>
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

The other app that I'm talking about, is not based on react native, I think it's native iOS.

Love react-native-camera? Please consider supporting our collective: 👉 https://opencollective.com/react-native-camera/donate
Want this issue to be resolved faster? Please consider adding a bounty to it https://issuehunt.io/repos/33218414

[cristianoccazinsp]

Not possible on iOS I'm afraid. When you setup the camera to record video, it takes exclusive access of the microphone and of the audio session, so everything else is stopped.

I believe it may be possible doing some changes on the native iOS code that requests the audio session. Basically, you need to use the AVAudioSessionCategoryPlayAndRecord here https://github.com/react-native-camera/react-native-camera/blob/master/ios/RN/RNCamera.m#L1424

Keep in mind that doing the above results in very low quality audio recording (I recall testing this a while ago), reason it was never implemented. For your use case, you can change the above line in XCode to test what you want and see if it works. For a full implementation, it would need to be added to a new property and setting in the library.

[mask477]

Not possible on iOS I'm afraid. When you setup the camera to record video, it takes exclusive access of the microphone and of the audio session, so everything else is stopped.

I believe it may be possible doing some changes on the native iOS code that requests the audio session. Basically, you need to use the AVAudioSessionCategoryPlayAndRecord here https://github.com/react-native-camera/react-native-camera/blob/master/ios/RN/RNCamera.m#L1424

Keep in mind that doing the above results in very low quality audio recording (I recall testing this a while ago), reason it was never implemented. For your use case, you can change the above line in XCode to test what you want and see if it works. For a full implementation, it would need to be added to a new property and setting in the library.

Thank you so much, I'll give it a try and let you know :)

[mask477]

Not possible on iOS I'm afraid. When you setup the camera to record video, it takes exclusive access of the microphone and of the audio session, so everything else is stopped.

I believe it may be possible doing some changes on the native iOS code that requests the audio session. Basically, you need to use the AVAudioSessionCategoryPlayAndRecord here https://github.com/react-native-camera/react-native-camera/blob/master/ios/RN/RNCamera.m#L1424

Keep in mind that doing the above results in very low quality audio recording (I recall testing this a while ago), reason it was never implemented. For your use case, you can change the above line in XCode to test what you want and see if it works. For a full implementation, it would need to be added to a new property and setting in the library.

I tried changing this:
BOOL setActive = [[AVAudioSession sharedInstance] setActive:YES error:&error];

to this:
BOOL setActive = [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionMixWithOthers error:&error];

but no luck, am I doing something wrong? :|

[cristianoccazinsp]

The audio library may also be taking exclusive access of the audio session, but your code above should definitely work.

Have you tried just with RNCamera and using the music app to play audio? I did something very similar to your code in the past and it let me play music in background (with the music app) while recording with audio.

Take a look at the other library's code: https://github.com/johnsonsu/react-native-sound-player/blob/master/ios/RNSoundPlayer.m#L95 , chances are, you also need to tweak settings there too so both libraries are compatible.

Also check the various settings and caveats here: https://developer.apple.com/documentation/avfaudio/avaudiosessioncategoryoptions/avaudiosessioncategoryoptionmixwithothers

Truth is, it is very tricky to mix sound, but testing all combinations with both libraries should give you the expected results.

[mask477]

The audio library may also be taking exclusive access of the audio session, but your code above should definitely work.

Have you tried just with RNCamera and using the music app to play audio? I did something very similar to your code in the past and it let me play music in background (with the music app) while recording with audio.

Take a look at the other library's code: https://github.com/johnsonsu/react-native-sound-player/blob/master/ios/RNSoundPlayer.m#L95 , chances are, you also need to tweak settings there too so both libraries are compatible.

Also check the various settings and caveats here: https://developer.apple.com/documentation/avfaudio/avaudiosessioncategoryoptions/avaudiosessioncategoryoptionmixwithothers

Truth is, it is very tricky to mix sound, but testing all combinations with both libraries should give you the expected results.

Thanks a bunch, I got it working :)

These are the things that I did:

in RNCamera.m line #1425:

            BOOL setActive = [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord
                     withOptions:AVAudioSessionCategoryOptionMixWithOthers
                           error:nil];

with react-native-sound, I set the category to Playback and OptionMixWithOthers to true like this:

Sound.setCategory('Playback', true);

[cristianoccazinsp]

Could this be a RNCamera setting somehow? I recall more people looking for what you're doing, Changing these at runtime is tricky though as it impacts the camera, fires sound interrupted events and so on, so it may break stuff.

How do you plan on keeping this change without updating RNCamera itself?

[mask477]

Could this be a RNCamera setting somehow? I recall more people looking for what you're doing, Changing these at runtime is tricky though as it impacts the camera, fires sound interrupted events and so on, so it may break stuff.

How do you plan on keeping this change without updating RNCamera itself?

well my use case is very very specific, because the application that I'm working on is not that complex is a very small app in itself, so this will work for me, even in the long run.

but just in case in the future if I would be facing such situation, I'll go for native iOS and Android development, as I'll have more control over the native features.

[renatobenks]

Thank you for the contribution @mask477! I'll be checking for a way to maybe include it on the API, but honestly, I'm not sure how possible it would be, once it has been deeply unstable.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You may also mark this issue as a "discussion" and i will leave this open.

[stale]

Closing this issue after a prolonged period of inactivity. Fell free to reopen this issue, if this still affecting you.