Sync animation to video

[SpecularTheoryBen]

I am trying to sync an animation or timeline to a video, but it is jittering due to inconsistent frame reads.

I need a way to ensure an animation stays frame locked to the video that is playing. Playable director has allows DSP control, so that could be one way to keep it locked.

We have attempted to drive an animator by getting the current milliseconds and converting it to a frame number, but that seems to be inconsistent. It will be off by a frame for long enough to see it jump.

[AndrewRH]

Hi,

Currently we don't really have a system for this. There might be a few ways you can try though.

On some platforms (like Windows) you can retrieve the timestamp of the texture. This is more accurate than querying the mediaPlayer for it's time, as the mediaPlayer time is the time is decoding but not the frame you're about to display.

long time = mediaPlayer.TextureProducer.GetTextureTimeStamp();

This 64-bit value is returned in HNS format (hundreds of nanoseconds), so you might want to convert it to seconds or frames:

double seconds = time / 10000000.0;

float frameRate = mediaPlayer.Info.GetVideoFrameRate();
int frame = Helper.ConvertTimeSecondsToFrame((float)seconds, frameRate);

You then have an accurate way to update your animation to the correct frame.

Also a good idea when you need to sync a video to something is for the video to be encoded in a way that is very fast to seek randomly. For this you need a video with lots of key-frames. Ideally each frame should be a key-frame. See the PDF documentation where we have a whole section on seeking and key-frames.

[jsd-within]

The technique works perfectly on Windows with GetTextureTimeStamp(). Unfortunately, on Android the results remain too inconsistent to be useable.

Part of the problem is that on Android the timestamp does not start at 0 for frame 0. Instead it seems to be time of day or some such. I'm not choosing the exactly correct moment to store the timestamp. It's not entirely clear how to go about it; I tried storing the first reported number above zero, but that was off by 35-37 frames. Then I tried storing it based on when GetCurrentTimeMs() is zero, which ends up being off by 0-2 frames. I also tried using MediaPlayerEvent.EventType.FirstFrameReady to set the initial timestep, which again was off by 35-37 frames. And that range changes as the video plays so I go out of sync. Generally it does not seem to be more reliable timing than just using GetCurrentTimeMs().

On Android you can change the video API, but results were the same with MediaPlayer and ExoPlayer, except that with MediaPlayer the 35-37 frame gap became 4-5 frames.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

[jsd-within]

I found a workaround by storing frame information in the video and reading it out of a shader, but it's inconvenient and leaves artifacts in the video. So I'm hoping the new Android version will work better for this use case.

[AndrewRH]

In version 2.x we have improved the time precision and frames have an accurate timestamp. There is always latency though when asking for a video frame, so to keep them in sync you could either buffer decoded frames and then select the one with the matching timestamp, or you could wait/block while the decoder is processing the requested frame and poll for it in a loop with some sane timeout value.

Closing this as there's nothing we can do to improve this