Suspending audio context for battery usage reasons

[cwilso]

We've run in to a significant request to reduce battery consumption. In short, we need to be able to suspend the audio hardware device usage.

Unfortunately, given the current design, we have no way of doing this; the best we can tell users is "drop all references to AudioContexts", but even then, until garbage collection happens there's no guarantee the hardware device context will be freed up (enabling it to suspend and stop consuming power).

Raymond and I have discussed this in some detail, and I think our preferred option is that we provide two methods on the AudioContext:

void suspend();
Promise resume();

...that will put the system into suspension (meaning, the AudioContext currentTime stops progressing, all sound output is paused, and the audio hardware can power down), and asynchronously wake it up again and resuming processing (which is an asynchronous operation, and may take some amount of [real] time until the audio system has resumed processing data).

This takes away the unpredictability of when the audio system shuts down, but also lets us account for the latency hiccup in restarting.

[padenot]

fwiw, Gecko does that at the moment (pausing the audio hardware) for the same reasons (for example, some parts of FirefoxOS use the Web Audio API to output system sounds, battery drain is real concern in this context).

Our solution works, but has the drawback of not taking latency into account (it's something like pausing the underlying audio stream when we see that only the destination node is present in the graph).

I'd rather, as you say, have an explicit API than relying on the behavior of the engine.

Random things that need to be specced (maybe there are more):

• MediaStreams should see their output ignored by the graph (as they are real time and we don't want to buffer everything)
• HTMLMediaElement can either be paused or see their output ignored by the graph, we need to decide, as we can do both. Then again, that's tricky, because HTMLMediaElements can have a source that is a real time MediaStream.
• ScriptProcessorNode events: should they drain the last bits of audio to the consumer, or just not fire their events if suspend() has been called ? What would the buffers look like, a shortened buffer for input, or a full buffer filled with silence? If the events are not fired, should they be queued for when the context is resume()ed ?
  • Should the context stop immediately and throw away the hardware buffers, or drain the remaining audio ? Basically, which guarantees to we put on the suspend() call.
• If we don't drain, would calling suspend() play back queued buffers from last time, or are they lost ?
• Do we assume, for AnalyserNode window functions, that the period before the resume() is silence, or the last audio data played back?

[cwilso]

Yeah, the introduced latency or clock-that-doesn't-progress was the problem with trying to do this automatically.

You've correctly found the one piece I didn't elaborate on. :) Real-time source nodes - MediaStreams and HTMLMediaElements - should have their output ignored while the AudioContext is paused.

ScriptProcessors shouldn't be a problem - they should pause just like the destination node, and events should not be fired. They would not get "queued", per se - but once resume() is called, the events would start firing again as time progresses.

I don't think we should drain the buffers, and yes, the playback would continue from the last played frame. And yes, for windowing purposes, the last audio data played back would be retained.

[padenot]

Well, I think we agree, then. Any other thoughts from actual users of the API instead of implementers :-) ?

[cwilso]

:) (Note there's an implicit users-of-API ask here from us, since this came
from an internal team using Web Audio.)

On Mon, May 5, 2014 at 3:42 PM, Paul ADENOT notifications@github.comwrote:

Well, I think we agree, then. Any other thoughts from actual users of the
API instead of implementers :-) ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/317#issuecomment-42249571
.

[jernoble]

This looks very similar to issue #72, which I opened for similar reasons.

[cwilso]

Hmm. There are three related but somewhat orthogonal things wrapped up here:

1. Exclusive access to sound hardware.
2. Pausing audio graph/processing for code scenario reasons (e.g. pausing while answering a call)
3. Avoiding battery drain by enabling the audio hardware to power down, while accounting for the latency inherent in reviving the audio hardware.

Jer, you wrapped the first and second together; I actually would like to explicitly enable audio apps to ask for exclusive access on systems that need exclusive access for better latency characteristics (e.g. Windows!), and I see how that would tie together with #2. I only list #2 and #3 separately because I think #2 can quickly imply pausing PARTS of the audio graph - for example, if a gamer hits Pause, the game sound may pause but the menu popped up frequently has UI feedback audio still active. That's much harder to explain (separate timelines for each audio pathway), unless we just say that's a multiple-AudioContext scenario. I'd like to explicitly avoid doing this, and keep one "currentTime" for each AudioContext.

Perhaps what we need here is

1. suspend and resume methods, so the developer can explicitly suspend/resume. If the system has suspended you (e.g. a phone call), resume()'s promise may not return for a while.
2. an event that fires when your AudioContext is suspended (since this may happen in your scenario when the system
3. a way to request "exclusive mode" on systems that have one. My preference would be to only do this on AudioContext startup, but at the very least, you should only be able to do this while the audio is suspended.
4. a running/suspended state exposure, as you suggested.

WDYT?

[padenot]

(1) is necessary.
(2) it might be nice to have an event for when the stream is restarted as well, because it can be super long. We put in probes in Firefox to measure this, and we now have numbers. 1 is the first time an audio stream is created for the process, 2 is after the first creation (we were pretty sure that the initial stream creation for a given process would set up a bunch of IPC stuff between the mixer process and the Firefox process and so would be longer). Number are in milliseconds, and if you look at the histogram at the bottom, the tail is depressingly long. This is across all platforms (someone on the WebRTC team at Mozilla has a Mac Book Pro where he regularly see a 8.5 second stream creation time, that's what made us actually measure it in the field).
(3) has already been requested by some people I met. This would allow apps to use ASIO drivers on Windows, for example, or WASAPI exclusive mode (although the same people have told me that the support is pretty buggy most of the time)
(4) I agree, it's cheap to do, and it's likely that the developer would implement something like that anyways, so it makes sense to directly have it on the context.

[jernoble]

On Jun 16, 2014, at 6:31 AM, Paul ADENOT notifications@github.com wrote:

(1) is necessary.

Agreed.

(2) it might be nice to have an event for when the stream is restarted as well, because it can be super long. We put in probes in Firefox to measure this, and we now have numbers. 1 is the first time an audio stream is created for the process, 2 is after the first creation (we were pretty sure that the initial stream creation for a given process would set up a bunch of IPC stuff between the mixer process and the Firefox process and so would be longer). Number are in milliseconds, and if you look at the histogram at the bottom, the tail is depressingly long. This is across all platforms (someone on the WebRTC team at Mozilla has a Mac Book Pro where he regularly see a 8.5 second stream creation time, that's what made us actually measure it in the field).

So a “suspended” and a “resumed” event (or some equivalent name)? Sounds fine to me.

(3) has already been requested by some people I met. This would allow apps to use ASIO drivers on Windows, for example, or WASAPI exclusive mode (although the same people have told me that the support is pretty buggy most of the time)

Why should this particular use case be exposed through API? Would a web page try “regular mode”, and if that fails retry with “exclusive mode”? Or would web developers start to always request “exclusive mode” because that “works better on windows”?

Maybe I’m missing something about what an “exclusive mode” to an AudioContext would allow UAs on Windows to provide.

(4) I agree, it's cheap to do, and it's likely that the developer would implement something like that anyways, so it makes sense to directly have it on the context.

Agreed.

-Jer

[padenot]

On 16/06/2014 19:38, Jer Noble wrote:

On Jun 16, 2014, at 6:31 AM, Paul ADENOT notifications@github.com wrote:
(3) has already been requested by some people I met. This would allow apps to use ASIO drivers on Windows, for example, or WASAPI exclusive mode (although the same people have told me that the support is pretty buggy most of the time)

Why should this particular use case be exposed through API? Would a web page try “regular mode”, and if that fails retry with “exclusive mode”? Or would web developers start to always request “exclusive mode” because that “works better on windows”?

Maybe I’m missing something about what an “exclusive mode” to an AudioContext would allow UAs on Windows to provide.

It's a mode where an application takes exclusive control of the audio
hardware, to get lower latency (by bypassing the mixing stage, for
example). It's not really clear to me that the people that are
requesting it have actually measured the latency: we can get between 10
to 30ms on desktop OS using the built-in audio stacks, these days, and
the Android situation is improving, so I think this can wait until there
is a clear need.

[cwilso]

Should additionally provide "release" to forcibly release unneeded AudioContexts.

[bjornm]

Will these methods be available for OfflineAudioContexts, too? Please see my comment on #21