Fast frame-perfect seeking [Help needed]

[JonnyBurger]

Problem: During render, we want to precisely capture each frame of a <Video>. Meaning frame 10 in the source video is frame 10 in the rendered video as well. This works, but the browser might not immediately show the correct frame after seeking to that position and emitting the seeked event. There is always some delay until the frame is actually displayed. This led to some frame jumping in Remotion and is currently fixed in the Audio PR by just setting a 300ms timeout after the seeked event was emitted. This seems to work, but of course slows down render time.

This is my command with which I tested it: npx remotion render src/index.tsx video-testing video-testing.mp4 && ffmpeg -i video-testing.mp4 -f image2 out%02d.jpg

The screenshot below shows the desired output: Frames numbered 0 - n in the correct order.
The wrong output is if some frame numbers are duplicated. Sometimes one has to run a command multiple times to get a buggy output.

What I've tried:

• videoRef.current.requestVideoFrameCallback: This is a new Web API (doesn't have TypeScript support yet) which should tell you when the frame is expected to be rendered. Unfortunately it wasn't accurate enough to be perfect. https://web.dev/requestvideoframecallback-rvfc/

• Web Codec API: Allows you to decode a video in Javascript. Needs a flag enabled but we can do that in Puppeteer. Unfortunately there is very little information on the internet out yet on how to do decode for example the example framer.mp4 video. I am struggling to pass the correct description parameter and don't know how to figure it out:

  const canvas = document.getElementById("id");
    const video = document.getElementById("video");
    var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';

    var assetURL = "./packages/example/src/resources/framer-24fps.mp4";

    const mediaSource = new MediaSource();

    video.src = URL.createObjectURL(mediaSource);
    mediaSource.addEventListener("sourceopen", () => {
      const sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
      fetch(assetURL).then(async function (res) {
        sourceBuffer.addEventListener("updateend", function (_) {
          if (!sourceBuffer.updating && mediaSource.readyState === "open") {
            mediaSource.endOfStream();
            video.play();
          }
        });
        function handleFrame(frame) {
          console.log(frame);
        }
        const init = {
          output: () => console.log("hi"),
          error: (e) => {
            console.log(e.message);
          },
        };

        const description = new Uint8Array(4);
        const config = {
          codec: "avc1.64000a",
          codedWidth: 1080,
          codedHeight: 1080,
          description: new UInt8Array([]) // What to pass in here?
        };

        let decoder = new VideoDecoder(init);
        decoder.configure(config);
        const buf = await res.arrayBuffer();
        let chunk = new EncodedVideoChunk({
          type: "key",
          timestamp: Date.now(),
          data: new Uint8Array(buf),
        });
        decoder.decode(chunk);
      });

Of course it could also be done using FFMPEG commands, but it would be more complicated. Remote videos would have to be downloaded first.

Another technique is rendering the video on a canvas and using some algorithm to determine if the video has already jumped to the right place. But also for remote videos, they need to have CORS enabled which would introduce another requirement.

Anybody has knowledge in any of these areas and can help out with code or advice?

---

IssueHunt Summary

Backers (Total: $0.00)

Become a backer now!

Or submit a pull request to get the deposits!

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[issuehunt-oss]

@jonnyburger has funded $150.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[JonnyBurger]

Added a bounty tho whoever can provide a solution for this problem!

Requirements: Remove the need for a timeout in VideoForRendering.tsx and still make the tests pass (npx jest in root).

But as a warning, this is very hard and might even be impossible.

[yuyuyzl]

Using canvas is a great idea in browser scenario.
But it might be better to simply render large videos(eg. a background video from screen recorder) with ffmpeg...

[JonnyBurger]

Created a Chrome issue to see what's up:
https://bugs.chromium.org/p/chromium/issues/detail?id=1255923

also the bug is fixed on linux and windows, only remains on macos (!!). See #624

[issuehunt-oss]

@jonnyburger has cancelled funding for this issue.(Cancelled amount: $150.00) See it on IssueHunt

[JonnyBurger]

Fixed by introducing a new component: <OffthreadVideo>: https://remotion.dev/docs/offthreadvideo