Playout Statistics API for WebAudio

[palak8669]

Introduction

When playing audio through WebAudio, we want to be able to measure the delay of that audio and the glitchiness of the audio. This document contains a proposal of an API that would allow WebAudio users to do this.

Use Cases

There is currently no way to detect whether WebAudio playout has glitches (gaps in the played audio, which typically happens due to underperformance in the audio pipeline). There is an existing way to measure the instantaneous playout latency using AudioContext.outputLatency, but no simple way to measure average/minimum/maximum latency over time.

Glitches and high latency are bad for the user experience, so if any of these occur it can be useful for the application to be able to detect this and possibly take some action to improve the playout.

Goals

We want to be able to calculate the following things:

• The fraction of audio that was made up from fallback frames (which in chromium is silence) due to audio glitches over a certain time interval
• The number of audio glitches over a certain time interval
• The average audio playout delay over a certain time interval, and the interval that the delay moves in

Proposed Solution

We can do this by adding a new attribute to AudioContext interface, AudioPlayoutStats. This new interface AudioPlayoutStats, defines multiple metrics to measure the glitches and latency in the audio playout.

partial interface AudioContext {
[SameObject] readonly attribute AudioPlayoutStats playoutStats;
}

/**
Fallback frames: Frames played by the output device that were not provided
by the playout path. This happens when the playout path fails to provide audio frames
to the output device on time, in which case fallback frames will be played. This typically
only happens if the pipeline is underperforming. In the Chromium implementation, fallback
frames are always silence. This includes underrun situations that happen for reasons unrelated to WebAudio/AudioWorklets.
*/

[Exposed=Window, SecureContext]
interface AudioPlayoutStats {
readonly attribute DOMHighResTimeStamp fallback[FramesDuration](https://w3c.github.io/webrtc-stats/#dom-rtcaudioplayoutstats-synthesizedsamplesduration)
readonly attribute unsigned long fallback[FramesEvents](https://w3c.github.io/webrtc-stats/#dom-rtcaudioplayoutstats-synthesizedsamplesevents)
readonly attribute DOMHighResTimeStamp [totalFramesDuration](https://w3c.github.io/webrtc-stats/#dom-rtcaudioplayoutstats-totalsamplesduration) 
readonly attribute DOMHighResTimeStamp averageLatency;
readonly attribute DOMHighResTimeStamp minimumLatency;
readonly attribute DOMHighResTimeStamp maximumLatency;
undefined resetLatency();
[Default] object toJSON();
}

Examples

var oldTotalFramesDuration = audioContext.playoutStats.totalFramesDuration
var oldFallbackFramesDuration = audioContext.playoutStats.fallbackFramesDuration
var oldFallbackFramesEvents = audioContext.playoutStats.fallbackFramesEvents
audioContext.playoutStats.resetLatency();

// Wait while playing audio
...

// the number of seconds that were covered by the frames played by the output device between the two executions.
let deltaTotalFramesDuration = (audioContext.playoutStats.totalFramesDuration - oldTotalFramesDuration) / 1000;
let deltaFallbackFramesDuration = (audioContext.playoutStats.fallbackFramesDuration - oldFallbackFramesDuration) / 1000;
let deltaFallbackFramesEvents = audioContext.playoutStats.fallbackFramesEvents - oldFallbackFramesEvents;

// fallback frames fraction stat over the last deltaTotalFramesDuration seconds
let fallbackFramesFraction = deltaFallbackFramesDuration / deltaTotalFramesDuration;
// fallback event frequency stat over the last deltaTotalFramesDuration seconds
let fallbackEventFrequency = deltaFallbackFramesEvents / deltaTotalFramesDuration;
// Average playout delay stat during the last deltaTotalFramesDuration seconds
let playoutDelay = audioContext.playoutStats.averageLatency / 1000;

Privacy & Security Considerations

Exposed information
This API includes stats about dropped frames and latency.

Dropped frame counters
This concerns the following attributes:

• AudioPlayoutStats.fallbackFramesDuration
• AudioPlayoutStats.fallbackFramesEvents
• AudioPlayoutStats.totalFramesDuration

fallbackFramesEvents is needed to calculate the number of glitches and fallbackFramesDuration calculates the duration of lost audio due to such glitches. This information is needed to evaluate the quality of audio streams played through WebAudio.

Latency
This concerns the following attributes:

• AudioPlayoutStats.minimumLatency
• AudioPlayoutStats.maximumLatency
• AudioPlayoutStats.averageLatency
• AudioPlayoutStats.resetLatency()

The instantaneous latency is already available through the AudioContext.outputLatency property. This new API adds the ability to get the minimum, maximum and average outputLatency over a time interval. It is theoretically already possible to calculate these stats through JavaScript, but it would be less efficient and user friendly.

Fingerprinting

• The latency information does not contribute to the fingerprinting surface at all because all the information is already available through AudioContext.outputLatency
• Glitches occur rarely and when they do, they do not follow any consistent frequency or pattern, so the dropped frame counters in the API are unlikely to contribute to fingerprinting.
• If two different origins are cooperating to compare glitches occurring simultaneously, and a distinct pattern of glitches occurs on both of these renderers, it might be possible to use them to figure out that they are running on the same device. This would be a very unreliable method of fingerprinting though as glitches only occur rarely and even if they exhibit a similar pattern it is unlikely to be identical. It is therefore difficult to reliably conclude that they come from the same system.

rarely = Based on measurements in Chrome, less than 0.5% of 10-second audio intervals contain glitches.

Let’s Discuss

We are looking for feedback on our proposal and improving it by looking for other possible solution alternatives for the use case. We would also like to hear from people who would be interested in using this API to gauge interest.

[tomayac]

CC: @hoch.

[hoch]

Can we consider renaming this subgroup of API to: Playout Statistics API for WebAudio

[palak8669]

Draft spec: https://palak8669.github.io/web_audio_playout/

[mehagar]

Zoom is interested in this API for the purpose of detecting audio glitches 👍

[cwilso]

Sounds good. @palak8669 , if you want to transfer the repo to me I can transfer it into the WICG org.

[cwilso]

Transferred to https://github.com/WICG/web_audio_playout.