Add MediaStreamTrackAudioStats interface #117

henbos · 2023-09-29T08:25:37Z

Only partially fixes #96, but is missing latency

The RTCAudioSourceStats contains capture related metrics that are better suited for a MediaStreamTrack method than RTCPeerConnection.getStats() since you may be capturing for a non-WebRTC context. Moving the metrics was covered by the April 18, 2023 Virtual Interim and the Working Group input was supportive of the move.

Since then I've updated the language to talk about audio frames rather than audio samples and to use the same language about "delivered" vs "dropped" that we used in the video stats case.

This is the new interface (edited to match latest patch set):

[Exposed=Window]
interface MediaStreamTrackAudioStats {
  readonly attribute unsigned long long deliveredFrames;
  readonly attribute DOMHighResTimeStamp deliveredFramesDuration;
  // TODO: Moved to #119
  // readonly attribute DOMHighResTimeStamp deliveredFramesLatency;

  readonly attribute unsigned long long totalFrames;
  readonly attribute DOMHighResTimeStamp totalFramesDuration;

  [Default] object toJSON();
};

Edit: From the October interim minutes, conclusion: "Overall approach makes sense, flesh out the details in the editors meeting"

Preview | Diff

henbos · 2023-09-29T08:31:40Z

History: Already covered in the April Virtual Interim and a PR like this "kind-of" already got approved in #97, but then that started the whole async vs sync debate so we never merged that

henbos · 2023-10-11T15:05:40Z

Thought: In order to be consistent with video stats, where we don't have droppedFrames - instead we have totalFrames, we should consider changing this PR to have totalFrames and totalFramesDuration instead of "dropped".

The app could simply calculate "droppedFramesDuration" as totalFramesDuration - deliveredFramesDuration.

So the glitch% would be (totalFramesDuration - deliveredFramesDuration) / totalFramesDuration

index.html

henbos · 2023-10-18T11:38:54Z

Based on the Virtual Interim, I should:

Replace droppedFrames with totalFrames (dropped = total - delivered so the same information is exposed it's just a different format). The reason for this is to make the audio stats consistent with video stats, which has totalFrames instead of droppedFrames already.
Rename "delay" to "latency".
Some clarifications if necessary, e.g. make sure everyone is on the same page on what delivered means and spell out the fact that latency and deliveredFrames need to increment simultaneously.

henbos · 2023-10-19T09:46:46Z

PR updated!

henbos · 2023-10-19T11:13:27Z

There's a section in the webrtc stats spec (here) that motivates total counters versus instantaneous values or pre-computed averages. Part of the API shape discussions that the WG has had was to do totals in track.stats as well for the same reasons.

henbos · 2023-10-19T14:10:07Z

Spoke with Jan-Ivar, I think it makes sense to also have the latest latency sample for use cases like A/V sync, in addition to average latency over app defined interval

jan-ivar

This LGTM but I'd like @padenot to sign off on it.

henbos · 2023-10-19T14:39:23Z

I also filed #119 to add latency of the latest audio frame for use of A/V sync. The avg latency is still useful for "what was the quality during interval X"

jan-ivar · 2023-10-19T14:39:57Z

@padenot does #119 satisfy your concern about obtaining the most current value?

henbos · 2023-10-20T08:35:21Z

I added the current latency in follow up PR #120 but because I can't figure out how to create PR chains, this PR needs to land first so I can rebase the next PR. In the meantime its diff contains both PRs

henbos · 2023-10-24T13:41:15Z

Friendly ping

jan-ivar · 2023-10-26T14:09:51Z

The editors can integrate label is so @henbos can merge this after @padenot looks at it tomorrow.

index.html

…pture-extensions into 230928_AudioStats

henbos · 2023-10-27T13:15:19Z

Unless anyone objects, I will merge this on Monday.

index.html

padenot · 2023-10-27T15:29:30Z

index.html

+            <li>
+              <p>The <dfn data-lt="current audio latency">current audio
+              latency</dfn> is the input latency of the last audio frame that
+              was delivered. If no audio frames have been delivered yet, this


what's the unit?

Milliseconds like the other metrics. But this should be in #119, sorry.

padenot · 2023-10-27T15:31:15Z

index.html

+              manner.</p>
+              <p>The percentage of audio that was dropped can be calculated as
+              [= dropped audio frames duration =] / ([= delivered audio frames
+              duration =] + [= dropped audio frames duration =]).</p>


This is incorrect, this is a ratio, not a percentage.

Rephrased as delivered/total to match the JS exposed attributes rather than the internal slots and moved the note. But if 9 out of 10 seconds of audio was delivered, that's a 9/10=90% percentage of the total duration delivered?

@henbos "percentage" implies a number between 0 and 100, when the division produces a number between 0 and 1 i.e. "ratio".

Maybe do an editorial PR to s/percentage/ratio/?

Lol, sorry I didn't catch that. Yes: #126

padenot · 2023-10-27T15:37:20Z

index.html

+              the same time as [= delivered audio frames =] and is measured in
+              milliseconds. Note that once a frame has been delivered to a sink,
+              that sink may internally have its own latency (such as playout
+              delay) which is not included in this measurement.</p>


Summing latencies of audio buffer (if that's what you mean) does not make any sense. You want a series of values here if you want to understand what's going on.

You want a series of values here if you want to understand what's going on.

Is that perhaps covered by #119?

@henbos would one way to progress here be to split out the latency metric in a separate PR (maybe for #119) so this PR with the audio stats model can be merged?

For the sake of progress I'll remove all latency metrics and follow up on them in #119.

It seems to me that the "controversial" part is the latency so let's merge the frame counters and follow up. The second part is about how to phrase the algorithm ("stable state" or otherwise) but that's #108 and editorial so let's do that for both audio and video when we have decided what phrase to use.

The remaining counters are very close to what we already have for video.

padenot · 2023-10-27T15:38:08Z

index.html

+              {{MediaStreamTrackAudioStats/[[DroppedFrames]]}} is set to
+              [= dropped audio frames =] and
+              {{MediaStreamTrackAudioStats/[[DroppedFramesDuration]]}} is set to
+              [= dropped audio frames duration =].</p>


Please rewrite this in terms of an algorithm running in parallel and stable state: https://html.spec.whatwg.org/multipage/webappapis.html#await-a-stable-state so it's clear what happen. Alternatively you can post a task from your algorithm running in parallel, that's more or less equivalent here.

No need to invent something else.

Note whatwg/html#2882.

I already filed #108 but did not have a reference for the language requested. Now I have a reference, but I also have a bug that questions this wording (Jan-Ivar's link). So I'm not sure what to make of that, but let's follow up in that issue as a separate PR because we should fix both the audio and video algorithm at the same time, they use identical language and this is editorial.

I also have a bug that questions this wording (Jan-Ivar's link)

To clarify, Paul said:

"please rewrite this in terms of an algorithm running in parallel and stable state"

"Alternatively you can post a task from your algorithm running in parallel, that's more or less equivalent here.

My reason for mentioning whatwg/html#2882 was to suggest going with 2.

They're just different ways to write algorithms in the spec that involve synchronous steps AND async steps.

To apply a JS analogy to spec writing: 1 is like async-await, and 2 is like promise-then. 1 is out of favor, because 2 is closer to how you would need to implement things e.g. in c++.

Ack, let's continue over at #108 where I quoted this last response.

Add MediaStreamTrackAudioStats interface

a1dfe0e

henbos requested a review from jan-ivar September 29, 2023 08:26

henbos mentioned this pull request Sep 29, 2023

Migrate capture metrics from RTCAudioSourceStats to MediaStreamTrack method #96

Closed

henbos requested a review from alvestrand September 29, 2023 08:32

padenot reviewed Oct 17, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

index.html Outdated Show resolved Hide resolved

Latency, totalFrames and clarifications

2546a9b

time time => time

08721a9

henbos mentioned this pull request Oct 19, 2023

[Audio Stats] Add current latency #119

Closed

jan-ivar reviewed Oct 19, 2023

View reviewed changes

Add current audio latency metric to track.stats.

8c45ba1

henbos mentioned this pull request Oct 20, 2023

Add current latency to audio track.stats (this has not actually merged, but my commits got entangled, do ignore) #120

Merged

mehagar mentioned this pull request Oct 25, 2023

Audio quality metrics similar to RTCAudioPlayoutStats WebAudio/web-audio-api#2542

Closed

aboba added the Editors can integrate label Oct 26, 2023

jan-ivar reviewed Oct 26, 2023

View reviewed changes

index.html Show resolved Hide resolved

henbos added 3 commits October 26, 2023 17:20

Rebase

e8511f4

Merge branch '230928_AudioStats' of https://github.com/henbos/mediaca…

109c82c

…pture-extensions into 230928_AudioStats

Add paragraph about microphone stats

4180da6

padenot suggested changes Oct 27, 2023

View reviewed changes

henbos added 3 commits October 28, 2023 08:44

Remove latency metrics

148fd9f

Remove latency reference

961b459

Move and rephrase note

66e96a4

henbos merged commit c4c6320 into w3c:main Oct 28, 2023
1 check passed

henbos mentioned this pull request Oct 28, 2023

Add current audio latency metric #124

Merged

jan-ivar mentioned this pull request Nov 30, 2023

Move MediaStreamTrack stats in its own spec? #132

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MediaStreamTrackAudioStats interface #117

Add MediaStreamTrackAudioStats interface #117

henbos commented Sep 29, 2023 •

edited by pr-preview bot

Loading

henbos commented Sep 29, 2023

henbos commented Oct 11, 2023 •

edited

Loading

henbos commented Oct 18, 2023

henbos commented Oct 19, 2023

henbos commented Oct 19, 2023

henbos commented Oct 19, 2023

jan-ivar left a comment •

edited

Loading

henbos commented Oct 19, 2023

jan-ivar commented Oct 19, 2023

henbos commented Oct 20, 2023 •

edited

Loading

henbos commented Oct 24, 2023

jan-ivar commented Oct 26, 2023

henbos commented Oct 27, 2023

padenot Oct 27, 2023

henbos Oct 28, 2023 •

edited

Loading

padenot Oct 27, 2023

henbos Oct 28, 2023

jan-ivar Oct 30, 2023

henbos Oct 31, 2023

padenot Oct 27, 2023

jan-ivar Oct 27, 2023

henbos Oct 28, 2023

padenot Oct 27, 2023

jan-ivar Oct 27, 2023

henbos Oct 28, 2023

jan-ivar Oct 30, 2023 •

edited

Loading

henbos Oct 30, 2023

Add MediaStreamTrackAudioStats interface #117

Add MediaStreamTrackAudioStats interface #117

Conversation

henbos commented Sep 29, 2023 • edited by pr-preview bot Loading

henbos commented Sep 29, 2023

henbos commented Oct 11, 2023 • edited Loading

henbos commented Oct 18, 2023

henbos commented Oct 19, 2023

henbos commented Oct 19, 2023

henbos commented Oct 19, 2023

jan-ivar left a comment • edited Loading

Choose a reason for hiding this comment

henbos commented Oct 19, 2023

jan-ivar commented Oct 19, 2023

henbos commented Oct 20, 2023 • edited Loading

henbos commented Oct 24, 2023

jan-ivar commented Oct 26, 2023

henbos commented Oct 27, 2023

Choose a reason for hiding this comment

henbos Oct 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-ivar Oct 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henbos commented Sep 29, 2023 •

edited by pr-preview bot

Loading

henbos commented Oct 11, 2023 •

edited

Loading

jan-ivar left a comment •

edited

Loading

henbos commented Oct 20, 2023 •

edited

Loading

henbos Oct 28, 2023 •

edited

Loading

jan-ivar Oct 30, 2023 •

edited

Loading