[Request/Suggestion] Support unpredictable frame drops and unmatching speed/pitch (drift correction)

[alopatindev]

If anyone considers implementing this feature in AudioAlign/Aurio and/or suggests possible ideas for implementation, please share them here. Thanks!

[protyposis]

What you are describing is generally referred to as drift correction. Drift is usually, but not necessarily always, related to pitch change.

unpredictable frame drops in both tracks

This can be interpreted as a case of non-linear drift (w/o pitch change).

unmatching overall average speed (often with higher pitch for faster audio)

This can be interpreted as linear drift, which is the simplest case (e.g., PAL speed-up).

AudioAlign already includes tools to handle drift semi-automatically, but I'll need to validate. Could you provide example files?

[alopatindev]

Could you provide example files?

Sure, here are some examples of problematic extracted audio tracks (password is SyncWithDriftCorrection, size is ≈11 GiB, sha256sum is dff403ac3d1958eff2636b3806af8a58cc21a37bb64e36dae2cef5e7813fbaf2).

Please note that some of these files may contain multiple audio tracks, sometimes with wrong language labels. Thanks!

[protyposis]

Thanks, here's an example video with the The_OC set. I'm not sure what dubbed vs. restored means, but I used the dubbed_1 and restored files as they seem to be two different languages with a different runtime. I transcoded them to .wav because direct FFmpeg decoding in AudioAlign isn't super stable yet. The aligned tracks are perfectly in sync:

Recording.2023-12-29.211558.mp4

Looking at the alignment graph, we can immediately see that there's

• linear drift (diagonal line instead of horizontal)
• a few obvious "frame drops" seen as glitches in the diagonal line (circled in red):

[alopatindev]

Wow, thanks for checking it; that's really great that AudioAlign can handle such cases!