What happens if the callback takes too long?

[Yay295]

If the function passed to video.requestAnimationFrame doesn't complete before expectedPresentationTime, will the frame still be presented, or would it be delayed until the function completes?

[tguilbert-google]

This is based on the current state of the implementation, rather than from a formal spec POV, but it's a good question that should be addressed in the spec.

The frame would still be presented, and video.rAF callbacks shouldn't hold anything up. In practice, it's because the presenting is happening on the compositor thread, and the callbacks are queued on the main thread immediately after the frame has been updated.
From some empirical testing, even if there is a heavy load on the main thread (e.g. flooding it with JS tasks), when the video.rAF callbacks are fired, they are always fired in with an "unexpired" expectedPresentationTime, with enough time to queue a useful window.rAF call. In other words, the following should always hold true:

video.requestAnimationFrame((x, metadata) => {
window.requestAnimationFrame((now) => {
console.log("Enough time?: " + (now <= metadata.expectedPresentationTime));
});
});

[WesselKroos]

Since this issue asks about the current implementation of the callbacks I want to pitch in. I have a use case in which you would want to block the immediate display of the next video frame.

I'm currently developing a Chrome extension that surrounds ambilight effect around YouTube video's with a canvas element. With the current implementation of requestAnimationFrame in the Chrome Beta channel the video and canvas frames are perfectly in sync. But that's only for video's up to 1080p and depends on the power of the graphics card. The canvas is a frame too late for video's larger or faster than that (1440p/24fps and up), but the canvas can still maintain the same framerate as the video on a 144hz monitor.

Currently, to workaround this issue I position another canvas on top of the video element. Then in the same requestAnimationFrame callback I execute drawImage on the canvas and the canvas on top of the video. This works pretty well. But when other things are happening on the main thread the video & ambilight canvasses are dropping frames, which is undesirable.

To be clear, I don't know a perfect solution for this. But the goal for my application is to keep the canvas and video in sync with each other.
(PS: Let me know if this should be a separate issue.)

Possible solution
Ideally I would like to be able to buffer the video frames by specifying a frameBufferMaximum option, so that we have enough headroom to process the video frame and draw it on a canvas in sync with the video element. This could be an option which defaults to 0 but can be specified between for example 0 and 3 frames. Then the new buffered video frame is displayed anyway when the requestAnimationFrame callback takes longer than the specified maxFrameBufferSize.

It could look like this in code:

var canvasElem = document.querySelector('canvas');
var videoElem = document.querySelector('video');
videoElem.requestAnimationFrame(
  () => {
    canvasElem.drawImage(videoElem, 0, 0)
  },
  {
    frameBufferMaximum: 1
  }
)

[tguilbert-google]

@Yay295, here's the latest version of the spec, and you can raise any other issues on that github.

The callbacks now always fire right before window.requestAnimationFrame(), after a new frame has been presented. We also pull the info for the last frame presented, right before running the callbacks. This means that if there were two frames presented between window.rAF calls, only the second would have its metadata surfaced via callbacks.

If the code in the callback takes a long time to run, it should not delay frames from being displayed. It would be the same behavior as when a call to window.rAF takes a long time to execute (which can be tested via this demo. Chrome has a different composition path for videos, which allow them to play smoothly even if the page blocks up. It seems like Firefox has a similar mechanism too.

@WesselKroos, part of the complexity of perfectly syncing a canvas and a video element in Chrome comes from the fact that videos have their own composition path. Blocking or buffering frames has a performance cost in cases where HW decoders have a limited frame buffer pool. Video.requestAnimationFrame won't be adding a way to synchronize paints on the main thread with video frames being presented on the compositor thread.

I think the WebCodecs API will give you the opportunity to buffer frames yourself, but that also involves decoding the video yourself, instead of using HTMLVideoElement.