iOS AVAudioPcmBuffer.Format.StreamDescription returns garbage values

[richardgarnier]

Description

In order to record audio, one can use the following code

private AVAudioEngine audioEngine;
private void StartAudioEngine()
{
    AVAudioEngine audioEngine = new AVAudioEngine();
    AVAudioFormat recordingFormat = audioEngine.InputNode.GetBusOutputFormat(0);
    audioEngine.InputNode.InstallTapOnBus(0, 1024, recordingFormat, OnBusTap);
}

private void OnBusTap(AVAudioPcmBuffer buffer, AVAudioTime when)
{
    // do something with buffer
    // processBuffer(buffer);
    AudioStreamBasicDescription desc = buffer.Format.StreamDescription;
    LogBasicDescription(desc); // will contain garbage
}

private void LogBasicDescription(AudioStreamBasicDescription desc)
{
    StringBuilder sb = new StringBuilder();

    sb.Append($"Reserved = {desc.Reserved}, ");
    sb.Append($"BitsPerChannel = {desc.BitsPerChannel}, ");
    sb.Append($"SampleRate = {desc.SampleRate}, ");
    sb.Append($"BytesPerFrame = {desc.BytesPerFrame}, ");
    sb.Append($"FramesPerPacket = {desc.FramesPerPacket}, ");
    sb.Append($"BytesPerPacket = {desc.BytesPerPacket}, ");
    sb.Append($"FormatFlags = {desc.FormatFlags}, ");
    sb.Append($"Format = {desc.Format}, ");
    sb.Append($"ChannelsPerFrame = {desc.ChannelsPerFrame}, ");
    sb.Append($"IsEncrypted = {desc.IsEncrypted}, ");
    sb.Append($"FormatName = {desc.FormatName}, ");
    sb.Append($"IsVariableBitrate = {desc.IsVariableBitrate}, ");
    sb.Append($"IsExternallyFramed = {desc.IsExternallyFramed}");

    Console.WriteLine(sb.ToString());
}

However the values contained in AudioStreamBasicDescription are complete garbage.
It also has some interesting characteristics:

• During an execution, the data contains two different values. During the first few seconds of recording, the sample rate is extremely low (2.1E-314 in my case), but not 0. Then the values change, gets a sample rate of 0 and will stay constant from now on.
• The values are always different between two executions
• The audio data itself looks correct. A SFSpeechAudioBufferRecognitionRequest can at least make sense of it and perform audio recognition on the data.

Some sample output:

Reserved = 0, BitsPerChannel = 0, SampleRate = 3.0467068875153E-314, BytesPerFrame = -2111226112, FramesPerPacket = 1, BytesPerPacket = 137997088, FormatFlags = LinearPCMIsFloat, Format = 22512464, ChannelsPerFrame = 2, IsEncrypted = False, FormatName = , IsVariableBitrate = False, IsExternallyFramed = False
Reserved = 1, BitsPerChannel = 137997088, SampleRate = 0, BytesPerFrame = 137997088, FramesPerPacket = 1, BytesPerPacket = 137997088, FormatFlags = LinearPCMIsFloat, Format = 137948384, ChannelsPerFrame = 1, IsEncrypted = False, FormatName = , IsVariableBitrate = False, IsExternallyFramed = False

Steps to Reproduce

1. Setup an iOS project with a button to execute the above code
2. Take a look at the output.

Expected Behavior

AudioStreamBasicDescription contains valid data.

Actual Behavior

AudioStreamBasicDescription contains garbage data.

Basic Information

• Version with issue: 4.3.0.991221, 4.4.0.001640
• Last known good version: None
• IDE: VS2019
• Platform Target Frameworks:
  • iOS: I am not sure where to find this info. The project is set to 'Default'.
• Affected Devices: iPhone 6 running iOS 12.4.5, maybe others

Workaround

No workaround available for the moment. However using an AVCaptureSession like below provide correct values in DidOutputSampleBuffer.

private void StartAudioCapture()
{
    session = new AVCaptureSession();
    session.SessionPreset = AVCaptureSession.PresetMedium;

    AVCaptureDevice[] mic = AVCaptureDevice.DevicesWithMediaType(AVMediaType.Audio);
    AVCaptureDeviceInput micInput = new AVCaptureDeviceInput(mic[0], out NSError _);

    AVCaptureAudioDataOutput audioOutput = new AVCaptureAudioDataOutput();
    audioOutput.SetSampleBufferDelegateQueue(this, new DispatchQueue("sample audio buffer"));

    session.AddInput(micInput);
    session.AddOutput(audioOutput);
    session.StartRunning();
}

[Export("captureOutput:didOutputSampleBuffer:fromConnection:")]
public void DidOutputSampleBuffer(AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
{
    AudioStreamBasicDescription? desc = sampleBuffer.GetAudioFormatDescription()?.AudioStreamBasicDescription; // contains correct data
    sampleBuffer.Dispose();
}

[richardgarnier]

I confirmed that the audio data was correct. It is possible to extract it with the following code, should someone need it:

private void OnBusTap(AVAudioPcmBuffer buffer, AVAudioTime when)
{
    AudioBuffer audioBuffer = buffer.AudioBufferList[0];
    byte[] data = new byte[audioBuffer.DataByteSize];
    Marshal.Copy(audioBuffer.Data, data, 0, audioBuffer.DataByteSize);
    // do something with data. However, note that you are not on the main thread.
}

buffer.Format.StreamDescription is however definitely reading junk data.

[richardgarnier]

I forgot to mention, but the format in StartAudioEngine obtained through

AVAudioFormat recordingFormat = audioEngine.InputNode.GetBusOutputFormat(0);

is also valid.

[chamons]

Here is an obj-c test app:
SoundTest.zip

And a full C# app:
iOSTest12.zip

I didn't bother printing every single field but the first N suggest this might be a real bug:

2020-02-20 15:23:46.545349-0600 SoundTest[5022:1603027] 0 32 48000.000000 4 4 4
vs
2020-02-20 15:07:24.053 iOSTest12[4978:1593199] Reserved = 1, BitsPerChannel = 159510328, SampleRate = 3.5720197047524E-314, BytesPerFrame = -1672435723, FramesPerPacket = 1, BytesPerPacket = 92750480, FormatFlags = LinearPCMIsFloat, Format = 92750336, ChannelsPerFrame = 792, IsEncrypted = False, FormatName = , IsVariableBitrate = False, IsExternallyFramed = False

[chamons]

FYI @dalexsoto

[eshvatskyi]

any news? 2 years old bug

[rolfbjarne]

It looks like this has been fixed actually (it's a duplicate of #8892).