MediaStreamTrack: actual frame rates captured

[henbos]

Sub-issue of w3c/mediacapture-main#826.

track.getSettings() already gives us the setting (assuming w3c/mediacapture-main#906 is fixed), but it's still useful to know what the actual frame rate is as it could be lower if...

• Camera is unable to produce the FPS of the setting (e.g. poor lighting).
• Frames are produced but dropped (e.g. frame decimation or pipeline not consuming frames fast enough).

Edit: New proposal:

• Just add framesCaptured. The "frames emitted" or similar can be observed from other APIs later in the pipeline.

Previous proposal: track.framesEmitted + track.framesDropped = framesCaptured

[jan-ivar]

Don't we already have this in stats for peer connection-related use cases? Also works outside of peer-connection in a pinch.

For playback, frame-rates appear measurable on the video element in Chrome and Firefox (dunno what Safari has here). What other use cases are there?

[jan-ivar]

track.framesEmitted: Total counter of frames emitted by the source.

This explanation confused me. In the PR it appears to instead be frames emitted by the track to its consumer, not the total frame count of the (camera) source. Which is it?

[henbos]

@jan-ivar "media-source" frame rates need to be clarified... historically they are the standardized version of "googFrameRateInput". In contrast to the send FPS or encoder FPS, this is the "input FPS", i.e. the rate of frames being input to WebRTC. So with zero frame drops within WebRTC, the media-source's FPS was the maximum possible FPS you could achieve as the send FPS. But frame drops happening prior to WebRTC is not visible in getStats().

Frame producer (e.g. Camera) --> Media Source --> Frame consumer
                              |                |
                              |          framesEmitted
                              |
            framesDropped = delta between framesProduced and framesEmitted (not in getStats)

Arguably you could add framesDropped to getStats(), but there are several use cases where we want to know about framesDropped or framesProduced where WebRTC isn't involved. What would be the argument for adding metrics useful outside of WebRTC to WebRTC specific APIs? Why not just add them to the track?

[henbos]

track.framesEmitted: Total counter of frames emitted by the source.

This explanation confused me. In the PR it appears to instead be frames emitted by the track to its consumer, not the total frame count of the (camera) source. Which is it?

I was avoiding saying "source FPS" because that begs the question if we're talking about FPS that was produced for the source or FPS emitted from the source (i.e. that was not dropped on the way between producer and source).

There are three interesting frame rates, and none of them are specific to WebRTC:

• Camera configured FPS: Should be getSettings().frameRate IMO.
• Camera actual FPS: There is no API for this, but in my proposal this is framesEmitted + framesDropped. Could alternatively be expose as framesProduced.
• Media source FPS: The frames that have not been dropped on the way to the source, i.e. that consumers could make use of. This is framesEmitted.

[henbos]

Changed this to only include framesCaptured