[web-animations-2] [css-animations-2] Allow animations to be synchronized to the playback of audio or video elements

[bramus]

(This was originally presented during the lightning round at the 2023 F2F in Cupertino)

With Scroll-Driven Animations we have defined some new types of timelines which authors can use to run an animation on. Besides the existing DocumentTimeline, authors can now use a ScrollTimeline to animate based on a scroller’s scroll offset or ViewTimeline to animate as an element crosses its scrollport.

One thing that authors also want to do is synchronise their animations to the playback of an <audio> or <video> element; see list of links enclosed below

List of links

• https://stackoverflow.com/questions/74212169/how-can-i-perfectly-synchronize-a-css-animation-to-the-start-of-a-looping-html-v
• https://stackoverflow.com/questions/71802931/sync-motion-of-graphics-with-video-in-javascript
• https://stackoverflow.com/questions/39489464/canvas-animation-synchronized-with-video-time
• https://stackoverflow.com/questions/6520772/how-to-link-synchronize-animation-to-a-video-recording-in-real-time
• http://www.elusien.co.uk/shotcut/OBS-animation/index.html
• https://fenomas.com/2014/03/animate-video-sync/
• https://www.smashingmagazine.com/2011/03/syncing-content-with-html5-video/
• https://www.geeksforgeeks.org/how-to-sync-css-animations-with-html5-audio/#
• https://www.sitepoint.com/syncing-css-animations-with-html5-audio/

The proposal is to add a MediaPlaybackTimeline type of timeline which would allow that. Authors can use it tosync up animations to a media element – <audio> or <video> – play position.

Reusing the existing animation-timeline and animation-range syntax, the CSS code for it would look like this. A new media-playback-timeline to create a MediaPlaybackTimeline would also be introduced.

#video {
  media-playback-timeline: --video;
}

.animated {
  animation: slide linear forwards;
  animation-timeline: --video;
  animation-range: 1s 5s;
}

[Que-tin]

I'm not sure if this is possible as of now, but will there also be a possibility to sync the <audio> or <video> – play position to the scroll position syncing it the other way around? I mean this is quite a common use case that currently often gets build around by having an image sequence because syncing videos to scroll is janky most of the time. I mean this is probably the best known example for what I'm speaking of.

[ydaniv]

@Que-tin yes, this is what I wanted to propose here, that a non-less interesting use-case is the opposite, of synching a media element to a timeline, like scroll/view/hover/etc.
But then this has nothing to do with animations.
Instead, you just want to be able to say: "sync this media's timing to another declared timeline".

Something perhaps like the following:

.timeline {
  view-timeline: --scrubber y;
}

video {
  playback-timeline: --scrubber;
}

So need to differentiate between declaring a timeline of a media for usage in an animation vs. specifying a timeline for a media element of a declared timeline.

[Que-tin]

So need to differentiate between declaring a timeline of a media for usage in an animation vs. specifying a timeline for a media element of a declared timeline.

Definitely, so I guess this this is another issue right?

Instead, you just want to be able to say: "sync this media's timing to another declared timeline".

I think we have to be careful with the word syncing here or at least how we define it. I think this is a unidirectional relation where only media adapts the timing of the other timeline not the other way around, right?
Because I don't see where syncing a view-timeline to a media-playback-timeline would make sense. Would this mean that the pages automatically scrolls as the video progresses?

[bramus]

I'm not sure if this is possible as of now, but will there also be a possibility to sync the <audio> or <video> – play position to the scroll position syncing it the other way around?

This is possible using a tad of JS

// Get Animation Progress
let progress = animation.effect.getComputedTiming().progress * 1;
if (animation.playState === "finished") progress = 1;
progress = Math.max(0.0, Math.min(1.0, progress)).toFixed(2);

// Sync Video to Animation Progress
document.querySelector("video").currentTime = (document.querySelector("video").duration * progress).toFixed(5);

See https://www.bram.us/2023/06/21/synchronize-videos-3d-models-to-scroll-driven-animations/ for a full write-up and an easy to use utility function to simplify things for you.

Note that thanks to the resolution in #8799 (comment), you can simplify the code and simply rely on animation.progress, but that’s not implemented in any browser just yet.

[ydaniv]

@bramus

This is possible using a tad of JS

Yes, but then you run inside a rAF in main thread. Hopefully this can stay off it.

@Que-tin

I think this is a unidirectional relation where only media adapts the timing of the other timeline not the other way around, right?

This is no different then what we have today with animations, you can get the animation and set its currentTime and of course it won't set the corresponding scroll position (:

---

One thing that would be missing in the above is the ability to set the ranges, which are part of the animation model.
Perhaps this should be part of the playback property on the target media element?