MSE with variable frame rate videos

[NatalyMagluy]

I didn't find any documentation stating if variable frame rate (VFR) is supported by HTML5 video+MSE.
I see problems in VFR stream playback in Chrome and IE.

Can anyone tell me if VFR should be supported?

Thanks,
Nataly.

[jyavenard]

define frame.
There's nothing in the spec that specifies a particular frame rate or define it per say. The spec is all about samples having a start time and a duration

In Firefox, frame rate (aka sampling rate) for audio track must be constant between init segments. It won't play nice otherwise.

For video however, only the start time of the video frame matters. A video frame is displayed until a new one starts. The duration of a frame is as such nextFrame.startTime - currentFrame.startTime

[NatalyMagluy]

We have stream with PTS values as following:
133411486861242 to 133411486864245 = 3003
133411486864245 to 133411486868749 = 4504
133411486868749 to 133411486871752 = 3003
133411486871752 to 133411486876257 = 4505
133411486876257 to 133411486879260 = 3003
133411486879260 to 133411486883764 = 4504
133411486883764 to 133411486886767 = 3003
133411486886767 to 133411486891272 = 4505
133411486891272 to 133411486894275 = 3003
...

And sourceBuffer.buffered shows gaps every ~ 2 secs (which is duration of segments)

[wolenetz]

For Chrome, please file a bug at https://crbug.com with media that reproduces the problem, and we'll investigate. Feel free to send email to me directly with a link to repro media, if sharing it widely is a concern. In general VFR video should not be a problem in Chrome. Does chrome://media-internals give any warnings or errors? Note especially that MSE is quite strict about when a gap (discontinuity in buffered ranges) might be detected in the appended media. Especially, if there is a jump forward more than 2*last frame duration in the sequence of DTS (not PTS), then the spec indicates discontinuity has occurred. This is the most likely source of gaps, and isn't necessarily rated to VFR.

[jyavenard]

AFAIK, Chrome and Edge have currently a bug open in that they incorrectly use the DTS to calculate the buffered range. This is probably what you're seeing
See #54 (comment) (chrome tracking bug is https://bugs.chromium.org/p/chromium/issues/list?q=label:MSEptsdtsCleanup)

Firefox with those timestamps, shouldn't show gaps in the buffered range.
Regardless, this wouldn't affect playback.

[wolenetz]

@ #176 (comment) : You're correct about the Chrome bugs (I'm working on those now :)). However, the MSE spec describes the discontinuity detection within a sequence of appended frames to be based upon DTS. The other half of the problem is overlap-detection in the already-buffered media, which is known in Chrome to be based incorrectly on DTS at the moment as noted in those bugs.

[jdsmith3000]

Did a crbug get entered on this? I'm not sure of IE behavior, but we expect that Edge would play variable frame rate with no special limitations.  I'd be interested in getting content if anyone observes otherwise.

[NatalyMagluy]

@jdsmith3000 No bugs got entered yet, we're investigating how our PTS/DTS are populated to eliminate encoding issue.
It's currently a Widevine only stream, so couldn't check in Edge.

[wolenetz]

@NatalyMagluy Please retry with Chrome Canary or Dev (M-66) with command-line option "--enable-features=MseBufferByPts". That switch engages new logic for compliant buffering by PTS intervals (not DTS intervals) in Chrome MSE. We'll be starting experiments in preparation for launch of that logic soon, too.

[wolenetz]

Please re-open this issue if there is still some concern around lack of variable frame rate support. The previously referenced MSE implementation issues around PTS/DTS have been fixed in Chromium for well over a year now.