New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App is using hardcoded 2 channels for input which will result in increased latency #5

Open
dturner opened this Issue Oct 19, 2017 · 21 comments

Comments

Projects
None yet
2 participants
@dturner

dturner commented Oct 19, 2017

There is a bug in Android O (internal id 66967812) which means that requesting a stereo input stream may result in a non-FAST track, resulting in increased latency on Android O devices.

Checking the logcat output on an Android O device should verify this.

Solution/workaround: Request only a single (mono) channel input stream.

Thanks to philburk@ for pointing this out.

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 20, 2017

Thank you for the info. Is there any news about the grouped input/output callbacks?

@dturner

This comment has been minimized.

dturner commented Oct 20, 2017

I just fixed this issue in the AAudio echo sample here: googlesamples/android-audio-high-performance@d269d62

This is almost certainly the cause of the high round-trip latency being reported by this app on Android O.

Not sure what you mean by grouped input/output callbacks? Are you referring to synchronous I/O? If so, this is not supported yet. Best practice is to perform input stream reads during the output stream callback (see the echo sample for code).

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 20, 2017

No, the grouped input/output callbacks are a different thing. I already explained this multiple times, over Twitter, in email and also here in GitHub. But let's do this again:

Previously, input and output callbacks monotonically followed each other: input, output, input, output, ...

Currently, input and output callbacks are coming in this order: input, input, input, input, output, output, output, output, input, input, input, input, ...

The size of the input/output groups changes between 3 and 5.

@dturner dturner changed the title from [Needs verifying] App is using hardcoded 2 channels for input which may result in non-FAST track to App is using hardcoded 2 channels for input which will result in increased latency Oct 20, 2017

@dturner

This comment has been minimized.

dturner commented Oct 20, 2017

Sorry I have no idea why that behaviour has changed, you are welcome to look at the Android source to investigate: http://androidxref.com/.

My advice is rather than having an input stream callback, perform a non-blocking read from the output stream callback, then you won't have this issue.

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 20, 2017

Yes, we are doing non-blocking read just like you describe, with AAudio.
But I'm talking about OpenSL ES here. It's latency is significantly higher, negatively impacting all pro audio apps.

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 20, 2017

Using mono input for AAudio reduced the round-trip audio latency to 38 ms (built-in mic/speaker) and 21 ms (loopback) on the Nexus 6P. These values are just slightly higher than OpenSL ES on Android 7. Meaning that AAudio provides no latency benefit yet, on this device.

@dturner

This comment has been minimized.

dturner commented Oct 20, 2017

Please raise a separate issue for the "i/o callback grouping", talking about two different issues on the same thread is confusing.

For the 21ms value - is that lower than it was without using the mono input?

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 20, 2017

Yes, much better, 57/40 became 38/21 (built-in/loopback).
Still not 33/17 though.

I'm not sure opening an issue here. Is this the appropriate place for it? The issue is not in the latency measurement app, but in Android.

@dturner

This comment has been minimized.

dturner commented Oct 20, 2017

OK I strongly suggest the following course of action if you want this fixed:

FYI it's only those apps which rely on real-time audio monitoring which would be affected by this issue. That's still a big deal, just saying it's not all pro audio apps.

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 20, 2017

Hi Don,

As you state, "it's a big deal" and big deal is within Android, not Superpowered.

As such, this is clearly an issue that the Android audio team should want to fix -- since you're on the team, and know about the issue, by all means.

Given how our earlier issues were handled, we're not very motivated to create an issue and also a test application. We already spent a lot of time and energy with this.

We've also sent our findings on multiple channels, via GitHub, email and Twitter DM. Unfortunately, we're not able to do significantly more at this moment -- we are small, Google is big.

@dturner

This comment has been minimized.

dturner commented Oct 23, 2017

"...and know about the issue" - The reason for asking you to submit that information in a bug report is because I don't know enough about the issue.

Here's some immediate questions I have:

  • What is the impact of having 4 input buffer callbacks followed by 4 output buffer callbacks? Presumably increased latency. If so, by how much?
  • Is there a way of avoiding this problem by using non-blocking reads inside the output buffer callback, rather than having 2 separate callbacks?
@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 30, 2017

I already mentioned a couple of times that non-blocking reads inside the output buffer callback is not possible with OpenSL ES. It's only possible with AAudio, and we are doing exactly that.

Please find all of our findings here: http://superpowered.com/android-audio-latency-problem-just-got-worse

@dturner dturner closed this Oct 31, 2017

@ghost

This comment has been minimized.

Collaborator

ghost commented Oct 31, 2017

We provided tons of information and analysis via several channels in the past few weeks. We hope that turning them to the public will help your team getting the resources and attention for a proper fix, just like it happened in the past with the "10 ms problem" article.

@dturner

This comment has been minimized.

dturner commented Nov 1, 2017

I shouldn't have closed this. From the article I read "So we fixed our latency measurement app and everybody was curious..." and assumed that you'd fixed the input stream from stereo to mono, but it hasn't been fixed.

@dturner dturner reopened this Nov 1, 2017

@dturner

This comment has been minimized.

dturner commented Nov 1, 2017

I have provided a fix/workaround to this issue here: #6

I should have picked up on this sooner but you thought this bug only applied to AAudio, when in fact it applies to OpenSL ES as well.

Now for some explanations:

  1. The app was requesting a stereo input stream which because of bug 66967812 results in obtaining a non-FAST path
  2. This non-FAST stream has burst sizes which are significantly larger than a FAST stream
  3. The app assumes it has a FAST stream and in each callback write a single burst of data which contains only FRAMES_PER_BUFFER audio frames
  4. The non-FAST stream still has more data to write so makes immediate successive callbacks until its buffer is empty
  5. Meanwhile the output stream is FAST and is ticking along making regular callbacks every ~4ms
  6. The different callback periods of input and output streams causes the irregular pattern of callbacks IN-IN-IN-IN-OUT-OUT-OUT-IN-IN etc
@ghost

This comment has been minimized.

Collaborator

ghost commented Nov 1, 2017

Thank you for the explanation, it's helpful and makes sense.

However, we think that switching to mono just to get nicer numbers is a cosmetic patch. A white lie.

Since there is no way for user apps to detect if they need to switch to mono input for the lowest latency or not.

There are also apps that can not switch to mono input at all. Think about a DJ app with DVS input. A single DVS signal requires 2 channels. Or multi-channel recording with live feedback.

@dturner

This comment has been minimized.

dturner commented Nov 1, 2017

If you have an audio device with mono input and a stereo output it makes sense to create streams with those properties.

For determining what audio devices are attached and their properties you can use the AudioManager.getDevices() and AudioDeviceInfo APIs

@ghost

This comment has been minimized.

Collaborator

ghost commented Nov 1, 2017

  1. That API is available from 23 only.
  2. But the biggest problem is: it returns 1, 2, and 3 channel options on the Nexus 6P for audio input TYPE_BUILTIN_MIC.
@dturner

This comment has been minimized.

dturner commented Nov 1, 2017

For now the suggested workaround is to only request a mono input on Android 8.0. I have updated my pull request to this effect: c6a7471

Tested on Pixel running Android 7.1 and Nexus 6P running Android 8.0

We'll need to see whether the OEM releases of Android 8.0 include the AOSP fix before deciding if a more complex workaround is required (e.g. only enabling this workaround if the version of Android doesn't include the AOSP fix).

@ghost

This comment has been minimized.

Collaborator

ghost commented Nov 2, 2017

We hope it's not too late for most of them. Fingers crossed!

@dturner

This comment has been minimized.

dturner commented Nov 8, 2017

Been giving this some thought. Respecting the fact that you want to provide a completely accurate picture of latency to developers and users, perhaps a good solution would be to add the ability to select the channel count for both input and output streams. There is at least one other scenario where latency might be different between stereo and mono:

When AAudio sharing mode is set to EXCLUSIVE, it will (for now) only be granted if the output stream is in stereo.

I am also removing my comment about the article being "misleading" - it was said in the heat of the moment and wasn't constructive, my apologies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment