context.resume() behavior when not allowed to start?

[rtoy]

https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-resume describes how resume() should behave and step 3 says

If the BaseAudioContext is not allowed to start, append promise to pendingResumePromises and abort these steps, returning promise.

Since the following steps are aborted, no control message is queued, so what happens to the pendingResumePromises? As currently written I think nothing happens; we neither resolve nor reject the promise.

It's seems useful for the promise to be rejected if the context is not allowed to start.

[padenot]

Closely related, this issue over on HTMLMediaElement: whatwg/html#3617

[padenot]

Current state of things for Web Audio, and problem statement: whatwg/html#3617 (comment)

tl; dr: For the Web Audio API, there is a single gap to fill: knowing that the transition to "running" will no happen automatically after construction of the AudioContext, because the UA is blocking auto-play.

There are a number of things to consider:

• Simply creating an AudioContext does not make sound in itself, but sound can happen in a variety of ways after its instantiation. Instantiating an AudioContext, and doing nothing with it is bad practice (it can have non-trivial battery impact, for example, waking up the physical audio hardware on a mobile device, this can however be mitigated by UAs, with latency implications but is worth noting). We (Mozilla, in Firefox Nightly, as an opt-in measurement, probably with a heavy bias) started gathering data [0] to try to characterize the patterns here with an histogram with a few buckets trying to understand what documents are doing. Creating an AudioContext and doing nothing with it is not super common, but it's certainly not something to ignore. It's also completely pointless and to be avoided by authors.
• Knowing that a certain audio graph topology will output sound ahead of time is non-trivial, if at all possible (some processing nodes can act as sources, some sources can be silent, etc.). It seems a bit complicated to determine that, from a certain calling patterns of methods, sound will or will not happen as a result.
• There is the guarantee that a "empty" graph (a graph with just the DestinationNode) does not make audible sound.
• It is useful to have the promise from resume be queued up internally and resolved when audio output is allowed, I find resulting code nicer to read and write. I'd rather not spec what @rtoy proposes here.
• Documents don't often use multiple AudioContexts, but they certainly can

On a different note, some of the things that UAs are currently exploring:

• Some UAs are considering having a prompt as a last resort, for when documents don't provide their own controls to allow starting the AudioContext. Having this prompt at AudioContext creation can be disturbing. On the other hand, it's bad form to create an AudioContext and to not use it immediately, as noted above. This could encourage writing applications in a better style. The reason of existence of this prompt is to be a last resort solution that avoids breaking the large amount of content that has been written in the past and will not be modified, but still allow blocking any audio output by default, allowing the user to decide whether audio output is acceptable for them. Applications that are being written or can be modified will be encouraged to move towards a solution that does not show this prompt (which means, creating the AudioContext at the right time).
• Some UAs are considering allowing the initial "suspended" -> "running" transitions when the start method from an AudioScheduledSource is called from a user gesture. This solves a lot of the cases above, but I find it inelegant, fragile and too implicit. As an example, applications that only make use of a ScriptProcessorNode or AudioWorklet because, for example, they consist of cross-compiled native code and don't make use of any native AudioNodes are going to still be broken by this. This accounts for a non-trivial amount of pages that make use of the Web Audio API.

The current things that I don't know and that are important to go forward confidently:

• @jernoble, what is the current behaviour for Safari on iOS: is the permission to play sound tied to the AudioContext or the document (unclear if it's the same as an HTMLMediaElement)

I see two possible solutions right now:

1. Adding a fourth member to the AudioContextState enum, in the style of what Safari does. This fourth state would indicate that playback is currently not possible because it's blocked by policy. It would go like this:

   • Initially, an AudioContext's "state" is "suspended". Immediately, one of two things can happen
     • If the UA allows playback, the AudioContext starts a transitions to "running" (it can take some time for this transition to happen, starting an audio output stream can be a long process)
     • If the UA blocks playback, the AudioContext immediately transitions to "placeholder enum value" (intentional placeholder name, we can chose the name later). This is edge-signaled by the onstatechange handler, and the state property allows querying the current value.
   • On the first AudioNode creation, UAs can decide to show a prompt (asynchronously, the nodes are constructed and everything works as it does today). Three possibilities here:
     - The user consents: the AudioContext starts a transition to "running"
     - The users does not consent: the AudioContext stays at the current state
     - The user does something on the document (user gesture, etc.) that the UA interprets as consent: the AudioContext starts a transition to "running"

2. Adding a member to the document which is a boolean, and states whether or not an AudioContext created right now would be allowed to output audio.

Some implications of those two directions:

1. Means that it is possible to create an AudioContext and to use it to determine whether or not audio output will be possible. Depending on the outcome of this, maybe the document can show its own UI (i.e. its own prompt), or proceed directly. This allow a granularity of auto-play per AudioContext
2. Means that it possible to do the above without creating an AudioContext, but that the granularity of auto-play is per document. Note however that it does not make a huge difference for AudioContexts (unlike HTMLMediaElement), because you can do whatever you'd do with two contexts with only a single one.

[0]: Link to Firefox's telemetry: a measurement of the time it takes, in seconds, to have non-silent output reaching the AudioDestinationNode (= making sound), after it's instantiation.

[padenot]

cc-ing the usual suspects: @mounirlamouri @cpearce @alastor0325 @foolip

[goldfire]

• It is useful to have the promise from resume be queued up internally and resolved when audio output is allowed, I find resulting code nicer to read and write. I'd rather not spec what @rtoy proposes here.

Would you mind expanding on this? The logical result of a promise is to either resolve or reject. What is the point of resume being a promise if it is possible for it to do neither?

We are trying to get this working in howler.js (goldfire/howler.js#939), and so far we've found no workable solution that works in a general use-case (which means thousands of sites/web apps are going to break when Chrome 70 is released). Maybe there is another solution that we are overlooking?

[padenot]

Would you mind expanding on this? The logical result of a promise is to either resolve or reject. What is the point of resume being a promise if it is possible for it to do neither?

My sentence explicitly says that the promise is resolved when the AudioContext switches to " running", I'm not sure what you're asking?

[goldfire]

@padenot My point is that it can't switch to running without doing another call to resume() inside a click handler or similar. It just doesn't make any sense the way it is setup and I don't understand how it can be argued that it does. If the audio is being blocked than the developer needs a way to know that. The only outcome of this is a greatly degraded experience for users of web apps trying to do the right thing.

[padenot]

The second sentence of my post says that the current setup does not work, and the rest of the post explains in details various bits of context, current solutions, and makes two different proposals to solve the issue.

[goldfire]

I read every word of your post and think these could be workable solutions. I (and many others) are simply frustrated because these changes are being rolled out to Chrome in a matter of weeks and who knows how long it will take to get an actual solution introduced into the spec and then implemented by the browsers. It just makes no sense why this wasn't addressed first.

[rtoy]

The launch of the auto-play policy for WebAudio in Chrome was done without knowledge of the Chrome WebAudio team. (We actually found out from people on the WebAudio slack channel). If we had known, we would have tried to do something. Which is not to say that the final outcome would have been any different, but we could have at least started discussions in the spec, at least before launch.

[padenot]

This was worked on during the TPAC 2018 in Lyon. Here is a summary:

No API change on the AudioContext. A new property on the document allows knowing whether or not an AudioContext will be "allowed to start". Details here. A proper spec with more details with cover all this, to be written.

Starting an AudioScheduledSource can now resume an AudioContext (#1771). AudioContext.resume() can also resume it.

I'm closing this, in light of the above.

[goldfire]

@padenot Great news, thanks for the update!

[Justinyu1618]

It seems like this is still an issue in Chrome. Calling resume() on an audio context that's been blocked by autoplay policies still never rejects or resolves.

The only resource I could find on detecting "allowed to start" is https://developer.mozilla.org/en-US/docs/Web/API/Navigator/getAutoplayPolicy but it's not implemented in any Chrome version.

Are there any updates to this, or potential solutions?

[padenot]

Justin, getAutoplayPolicy() is indeed what you're supposed to use to solve this and if for now only implemented in Firefox.

Maybe @hoch can comment on whether this is planned or if there is a ticket to follow on https://bugs.chromium.org/p/chromium/issues/list.

I'm not sure about WebKit, maybe @jyavenard or @Youenn have an idea.

In any case, Chrome is correct to not reject those promises, per spec, I expect that other implementation behave similarly (please let us know if this is not the case!).