video tracks are not aligned

[gsabran]

I'm trying to run mp4dash with two inputs representing the same video in different size:

mp4dash video-small.mp4 video-large-2.mp4 --no-split --no-media --verbose --use-segment-list

It fails with ERROR: video tracks are not aligned I've tracked down that error that is raised here Apparently, some sample counts differ. Sample counts are generated here and I have honestly no idea what they correspond to.

The large and small videos have been previously fragmented with mp4fragment (no particular settings) and the original versions have been created with ffmeg using the same original video as input and simply changing size settings.

[barbibulle]

When you package multi-bitrate video, you need to ensure that all the variants of your video streams have key frames at the same location, so that the fragments in the variant streams are aligned.
The typical encoding workflow is to configure an encoder/transcoder to produce one file per bitrate, in a way that guarantees the alignment of GOPs (Group Of Picture), so that a valid, aligned, set of DASH segments can be made. Some encoders can produce files that are already in fragmented-mp4 form, but some don't (in which case mp4fragment is used to fragment the encoder output).
It isn't always easy to constrain encoders/transcoders to produce outputs with fixed GOP sizes (encoders try to be smart, with features like automatic scene-cut detection, etc.), but it is necessary. Some encoders are sophisticated enough to know about multi-bitrate encoding right from the start, which is more efficient than doing multiple single-bitrate encodings.
The simple utility mp4-dash-encode.py included in the Bento4 distribution shows how this can be done with ffmpeg (both for h.264 and hevc).

[awakenedguy]

Just got the same problem. I actually use scene detection on purpose, it's simply way better with than without (better for visual quality for fast moving dynamic scenes), but always with fixed 2 seconds GOP alignment, which corresponds to 2 seconds segment size that I want to use in fmp4 hls with mp4dash.

Here is how I encode samples that cause "video tracks are not aligned" error (ffmpeg 3.4.1, constrained VBR 110% setup):

1. ffmpeg -i sourcevideo.mkv -an -movflags +faststart -vsync 1 -c:v libx264 -r 23.976 -preset slower -vf 640:-2 -b:v 800k -bufsize 800k -maxrate 880k -bf 3 -refs 3 -force_key_frames "expr:gte(t,n_forced*2)" -keyint_min 24 -sc_threshold 40 -pass 1 -f mp4 NUL

2. ffmpeg -i sourcevideo.mkv -an -movflags +faststart -vsync 1 -c:v libx264 -r 23.976 -preset slower -vf 640:-2 -b:v 800k -bufsize 800k -maxrate 880k -bf 3 -refs 3 -force_key_frames "expr:gte(t,n_forced*2)" -keyint_min 24 -sc_threshold 40 -pass 2 out360.mp4

3. mp4fragment out360.mp4 --fragment-duration 2000 out360_fragmented.mp4

Same 3 commands for 480p, 720p and 1080p, all parameters are the same except bitrate, bufsize, maxrate and obviously "-vf" (each variant is produced with its own separate 1st pass).

So all four have a fixed GOP alignment, even with scene detection enabled. This is known to not cause any problems with ffmpeg HLS TS muxer (when used with TS mezzanine files) and with Shaka packager HLS fMP4 (when used with MP4 mezzanine files). That probably means that those two packagers somehow successfully handle this problem internally, since testing does not uncover any problems whatsoever (multi-bitrate setup is throughly tested with Flowplayer, which is based on hls.js).

I even studied the output (those four mezzanine files) with Telestream Switch Pro, it allows to beautifully visualize all GOP positions. All I-frames' positions in 360, 480, 720 and 1080 variants seem to correspond one to another, again, even with scene detection enabled, which means IDR I-frames at same fixed intervals each 2 seconds + additional non IDR I-frames at scene changes. Positions of those latter seem to be the same too, in all four variants, at least at first sight. I was unable to find any differences in those non IDR I-frames' positions.

So does that mean that scene detection is a show stopper to using bento to correctly produce multi-bitrate hls fmp4?

P.S.: I also tried to add --index, --force-i-frame-sync auto, --force-i-frame-sync all, none of that helps:

mp4fragment --fragment-duration 2000 --index --force-i-frame-sync all audio.m4a audio_fragmented.mp4
mp4fragment --fragment-duration 2000 --index --force-i-frame-sync all out360.mp4 out360_fragmented.mp4
mp4fragment --fragment-duration 2000 --index --force-i-frame-sync all out480.mp4 out480_fragmented.mp4
mp4fragment --fragment-duration 2000 --index --force-i-frame-sync all out720.mp4 out720_fragmented.mp4
mp4fragment --fragment-duration 2000 --index --force-i-frame-sync all out1080.mp4 out1080_fragmented.mp4

mp4dash --hls audio_fragmented.mp4 out360_fragmented.mp4 out480_fragmented.mp4 out720_fragmented.mp4 out1080_fragmented.mp4

I still get error:

Parsing media file 1: audio_fragmented.mp4
Parsing media file 2: out360_fragmented.mp4
Parsing media file 3: out480_fragmented.mp4
Parsing media file 4: out720_fragmented.mp4
Parsing media file 5: out1080_fragmented.mp4
ERROR: video tracks are not aligned ("File 3#1" differs from File 2#1)

[awakenedguy]

I slighty modified ffmpeg call and that completely solved the problem. Seems like it really was not properly aligned and now I constrained ffmpeg to do a real fixed GOP alignment, while keeping scene detection enabled.

For those who are interested, instead of:

ffmpeg -i sourcevideo.mkv -an -movflags +faststart -vsync 1 -c:v libx264 -r 23.976 -preset slower -vf 640:-2 -b:v 800k -bufsize 800k -maxrate 880k -bf 3 -refs 3 -force_key_frames "expr:gte(t,n_forced*2)" -keyint_min 24 -sc_threshold 40 -pass 2 out360.mp4

I did

ffmpeg -i sourcevideo.mkv -an -movflags +faststart -vsync 1 -c:v libx264 -r 23.976 -preset slower -vf 640:-2 -b:v 800k -bufsize 800k -maxrate 880k -bf 3 -refs 3 -force_key_frames "expr:eq(mod(n,$keyframe_interval),0)" -x264-params rc-lookahead=${keyframe_interval}:keyint=${keyint}:min-keyint=${keyframe_interval} --pass 2 out360.mp4

Where:

$keyframe_interval = $roundedFrameRate * $segsize
$keyint = $keyframe_interval * 2
$segsize = 2 (seconds)
$roundedFrameRate = 24 (in this case)

Doubing keyint is required to trick ffmpeg, since otherwise it does not allow to assign the same value to both keyint_min and keyint, it tends to divide keyint by 2 internally. This trick allows to compensate that and to have the same keyint_min and keyint, even if in metadata it would appear as not the same values, in reality it is.

More info here:
https://superuser.com/a/1223359

Then it works:

mp4fragment --fragment-duration 2000 --index audio.m4a audio_fragmented.mp4
mp4fragment --fragment-duration 2000 --index out360.mp4 out360_fragmented.mp4
mp4fragment --fragment-duration 2000 --index out480.mp4 out480_fragmented.mp4
mp4fragment --fragment-duration 2000 --index out720.mp4 out720_fragmented.mp4
mp4fragment --fragment-duration 2000 --index out1080.mp4 out1080_fragmented.mp4

mp4dash --hls audio_fragmented.mp4 out360_fragmented.mp4 out480_fragmented.mp4 out720_fragmented.mp4 out1080_fragmented.mp4

[rodrigo-brito]

I solved with a different combination fix the GOP size:
ffmpeg -i sourcevideo.mp4 -c:v libx264 'keyint=48:min-keyint=48:no-scenecut' -strict -2 -r 24 -vf 'scale=480:-2' ouput-480.mp4

[Makio64]

Recently encounter the same problem and I solved it by updating ffmpeg to the latest ( from 4.3.x to 6.0.x )

[samcurran22]

what bento4 command must be used if the file is HEVC ?
Please suggest
@awakenedguy @barbibulle ?