Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

using .defaultToSpeaker causes audio session initialization problems on iPhone. #14

Open
DaveHineman opened this issue Oct 4, 2022 · 22 comments
Open

using .defaultToSpeaker causes audio session initialization problems on iPhone. #14

DaveHineman opened this issue Oct 4, 2022 · 22 comments

Comments

@DaveHineman
Copy link

To make an iPhone play sound from the loudspeakers, not the receiver, it is necessary to add .defaultToSpeaker to the AVAudioSession options. This is causing the audio_capture session to not initialize.

Step 1:
Run the flutter_audio_capture Example without modification. Observe that the first press of Start correctly causes data to stream to the Console.

Step 2:
Modify line 21 of AudioCapture.swift as such:
options: [.defaultToSpeaker, .allowBluetoothA2DP, .allowAirPlay, .allowBluetooth])

Or change line 21 to read:
options: [.defaultToSpeaker])

Run the flutter_audio_capture Example again. Observe that nothing happens when Start is pressed.
Press Stop, then press Start again and the stream is finally seen in the Console.

Other observations:
When line 21 does not include .defaultToSpeaker, the sample rate is 44100. Adding .defaultToSpeaker changes this to 48000.
To see this, add a print statement before 'audioEngine.prepare()' .
print("Buffer: ", buffer.format, ")

I have not been able to reveal any error messages using Xcode or VScode.

Clue: The AudioKit Cookbook is able to correctly implement .defaultToSpeaker without causing iPhone problems. I am looking for differences in the code, but have not found anything that makes a difference when applied to the audio_capture plugin.
https://github.com/AudioKit/Cookbook

Tested using Xcode 14.0.1 with iPhone 7S on iOS 15.7 and iPhone XR on iOS 16.0.2.
@DaveHineman
Copy link
Author

Update:
If lines 15 and 16 are commented out, then capture will occur the first time Start is pressed.

Afterward, pressing stop, then start, will cause a crash because the Tap() is not nil.

This appears to be a problem with initialization. Adding a check for Tap() status, before resetting the engine and removing a non-existent tap is probably needed. If I come up with something that works, I will post it.

@ysak-y
Copy link
Owner

ysak-y commented Oct 8, 2022

Hi @DaveHineman , thank you for your feedback!! I tried to run example app by adding defaultToSpeaker option on line 21 as you explained but seems working on my simulator.

Xcode version: 13.4.1
device: iPhone 13 with iOS 15.5 (Simulator)

run-example-with-defaultToSpeaker.mp4

I'll try to update Xcode version to 14.x and run it on actual device later.

@DaveHineman
Copy link
Author

DaveHineman commented Oct 8, 2022 via email

@DaveHineman
Copy link
Author

DaveHineman commented Oct 8, 2022 •
edited

Adding these code changes is allowing capture the first time I press Start, with .defaultToSpeaker as an option.

  • initialize a counter
  • if statement that does not remove the Tap unless count > 0
  • increment count after Tap installed
  • I also relocated prepare and start above the installTap, so that line-by-line execution occurs. This does not have any real effect on the code execution.

I will now test the code changes in my complete app and confirm that it functions as expected. I think a smarter solution is possible, but I was unable to find a useful query that would indicate if the Tap was in use and satisfy my IF statement.

import Foundation
import AVFoundation

class AudioCapture {
private let audioEngine: AVAudioEngine = AVAudioEngine()
private var count = 0
init() {
}

deinit{
audioEngine.inputNode.removeTap(onBus: 0)
audioEngine.reset()
}

public func startSession(bufferSize: UInt32, sampleRate: Double, cb: @escaping (_ buffer: Array<Float>) -> Void) throws {
    
    do {

        if (count != 0) {
            audioEngine.inputNode.removeTap(onBus: 0)
        }
        let audioSession: AVAudioSession = AVAudioSession.sharedInstance()
        try audioSession.setCategory(.playAndRecord, options: [.defaultToSpeaker, .mixWithOthers]) // init line from AudioKit adds .defaultToSpeaker
        try audioSession.setActive(true)
        let inputNode = audioEngine.inputNode
        let inputFormat  = inputNode.inputFormat(forBus: 0)

        audioEngine.prepare()
        try audioEngine.start()
        
        inputNode.installTap(onBus: 0,
                             bufferSize: bufferSize,
                             format: inputFormat) { (buffer: AVAudioPCMBuffer, when: AVAudioTime) in
            // Convert audio format from 44100Hz to passed sampleRate
            // https://medium.com/@prianka.kariat/changing-the-format-of-ios-avaudioengine-mic-input-c183459cab63
            let formatToConvert = AVAudioFormat(commonFormat: .pcmFormatFloat32,
                                                sampleRate:   sampleRate,
                                                channels:     1,
                                                interleaved:  true)!
            if buffer.format != formatToConvert {
                var convertedBuffer: AVAudioPCMBuffer? = buffer
                if let converter = AVAudioConverter(from: inputFormat, to: formatToConvert) {
                    convertedBuffer = AVAudioPCMBuffer(pcmFormat:  formatToConvert,
                                                       frameCapacity: AVAudioFrameCount( formatToConvert.sampleRate * 0.4))
                    let inputBlock : AVAudioConverterInputBlock = { (inNumPackets, outStatus) -> AVAudioBuffer? in
                        outStatus.pointee = AVAudioConverterInputStatus.haveData
                        let audioBuffer : AVAudioBuffer = buffer
                        return audioBuffer
                    }
                    var error : NSError?
                    if let uwConvertedBuffer = convertedBuffer {
                        converter.convert(to: uwConvertedBuffer, error: &error, withInputFrom: inputBlock)
                        cb(Array(UnsafeBufferPointer(start: uwConvertedBuffer.floatChannelData![0], count:Int(uwConvertedBuffer.frameLength))))
                    }
                }
            } else {
                cb(Array(UnsafeBufferPointer(start: buffer.floatChannelData![0], count:Int(buffer.frameLength))))
                }
        }
        count += 1

    } catch let err {
        print(err)
    }
    
    }

public func stopSession() throws {
let audioSession: AVAudioSession = AVAudioSession.sharedInstance()
audioEngine.stop()
try audioSession.setActive(false, options: .notifyOthersOnDeactivation)
}
}

@DaveHineman
Copy link
Author

DaveHineman commented Oct 9, 2022 •
edited

Update:
Running my full app (which uses AudioCapture and Flutter_Midi plugins) after making these changes is causing some kind of initialization problem on iPhoneXR and iPhone7S. The UI is extremely slow, CPU is consumed and the device gets warm. The same code on an iPad has no problems at all. Again, if I remove .defaultToSpeaker from the init string, both iPhones load the program normally (with sound defaulting to the receiver speaker). I'm not sure where to look, but I will try to examine the device logs for information.

In conclusion, my changes weren't a solution. Hopefully the changes will reveal something useful.
Previously, the mic would not produce any data. After my changes, the mic produces output for a few seconds and then stops. The 'slow loading graphics' occurs when using 'flutter run.' If I use 'flutter run --release', then the UI loads faster, but is very sluggish.

@ysak-y
Copy link
Owner

ysak-y commented Oct 19, 2022

In my environment, I can't reproduce it with my actual iPhone 12...Could you please try again with pulling #13 ?

@DaveHineman
Copy link
Author

the '.defaultToSpeaker' option is not in your newest code. Update line 21 to include this option.

@DaveHineman
Copy link
Author

I retested with #13 and the issue still exists when '.defaultToSpeaker' is added to line 21 of AudioCapture.swift
Something breaks when the StartSession executes 'audioEngine.inputNode.removeTap(onBus: 0)' when a tap doesn't exist. That is why I had to add a counter and check if count > 0 condition before removing the tap.
However, if you don't reset the tap, and it exists, then that also causes a crash, so the Tap on Bus 0 must be removed if it exists before adding a new one.

@DaveHineman
Copy link
Author

Another update - I found the issue is present in the AudioKit Cookbook, so I will also open a problem report there.

@DaveHineman
Copy link
Author

The AudioKit issue was different - the speaker routing was changing, but the mic continued to function.

@DaveHineman
Copy link
Author

I have successfully resolved the .defaultToSpeaker problem. I have attached the improved AudioCapture.swift file that you can include with the next release.
It is not necessary, or desirable, to reset the audio every time you start a new stream. I also added some code from the flutter audio_streams package that keeps the stream from dying when changing pages in an app. (https://pub.dev/packages/audio_streams).

AudioCapture.swift.zip

You may add the .allowBluetoothA2DP, .allowAirPlay, .allowBluetooth back to the options line, but I don't use them.

Thanks!
Dave.

@DaveHineman
Copy link
Author

feel free to remove the print statements 'startSession' and 'stopSession' that I used for debugging.

@kijka
Copy link

kijka commented Nov 30, 2022

@DaveHineman nice work! BTW, do you have issues with the latest 1.1.1 version on iOS? I found that it did not work correctly, and I needed to fall back to 1.1.0:
#16

@DaveHineman
Copy link
Author

DaveHineman commented Nov 30, 2022 via email

@kijka
Copy link

kijka commented Dec 1, 2022 •
edited

@DaveHineman, you see 'start-stop-start' because, in 1.1.1, we now have a temporary listener to check for sampleRate (I'm not sure why), so it's good.

The issue is that 1.1.1 don't work in iOS with the example in this library. I'll try to add your fix and test #16 again.

@ysak-y
Copy link
Owner

ysak-y commented Dec 3, 2022

Hi, @DaveHineman nice work!! Thank you for your work!! btw, can you make Pull Request for it? I can implement also but I want you to send me the Pull Request because it leave your contribution!

@DaveHineman
Copy link
Author

I've added my changes to 1.1.2 and the example runs properly. Will begin the pull request now.

@DaveHineman
Copy link
Author

Pull request submitted.

@ysak-y ysak-y mentioned this issue Dec 18, 2022
Merged
@ysak-y
Copy link
Owner

ysak-y commented Dec 18, 2022

Hi @DaveHineman , @vladsonkin !!
I published new version 1.1.3 with #17 that is mentioned in here. Could you please try to update package and resolve this issue?
#18

@DaveHineman
Copy link
Author

DaveHineman commented Dec 18, 2022 via email

@MaskaMaciej
Copy link

Hello.
I can confirm, that this is working now (v 1.1.3). Thank you for this fix :)
iOS version 16.1.1
iPhone XR

@kijka
Copy link

kijka commented Dec 19, 2022 •
edited

Looks good indeed, but there is still an iOS issue #16 (tested with 1.1.1, 1.1.2, 1.1.3)
I can use this library only on 1.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kijka @ysak-y @DaveHineman @MaskaMaciej