Common Time base normalization support(?)

[bryc]

Using LosslessCut I was trying to merge videos which had the same resolution and codecs, and comparable bit rates. But it failed as the second video became 'frozen' with the audio still playing. VLC could play further into the video, but some others could not. It seems that 'time base' was to blame.

The only major difference between the files are the timebase (tbn) and/or frame rate:

29.97 fps, 29.97 tbr, 492900 tbn, 59.94 tbc
29.97 fps, 29.97 tbr, 14625  tbn, 59.94 tbc
30    fps, 30    tbr, 468k   tbn, 60    tbc
30    fps, 30    tbr, 515700 tbn, 60    tbc
30    fps, 30    tbr, 853200 tbn, 60    tbc
26.10 fps, 26.10 tbr, 314700 tbn, 52.19 tbc
28.21 fps, 28.21 tbr, 133680 tbn, 56.42 tbc
30    fps, 30    tbr, 613200 tbn, 60    tbc
30    fps, 30    tbr, 277400 tbn, 60    tbc
30    fps, 30    tbr, 272600 tbn, 60    tbc
29.97 fps, 29.97 tbr, 21008  tbn, 59.94 tbc
29.97 fps, 29.97 tbr, 650400 tbn, 59.94 tbc

I got around this issue by converting them to .ts (ffmpeg -i $1 -c copy $1.ts), which then changed all tbn values to 90k:

29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
30    fps, 30    tbr, 90k tbn, 60    tbc
30    fps, 30    tbr, 90k tbn, 60    tbc
30    fps, 30    tbr, 90k tbn, 60    tbc
26.08 fps, 26.08 tbr, 90k tbn, 52.19 tbc
28.25 fps, 28.25 tbr, 90k tbn, 56.42 tbc
30    fps, 30    tbr, 90k tbn, 60    tbc
30    fps, 30    tbr, 90k tbn, 60    tbc
30    fps, 30    tbr, 90k tbn, 60    tbc
29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc

As far as I can tell, it created a fully valid video, and is still lossless and audio is in sync.

So LosslessCut can't do this itself - could this process be automated (using common tbn value) and implemented into the app?

Might be a considerable reliability improvement.

[mifi]

TBH I'm not sure about how tbn works, but I believe that we shouldn't automatically change the tbn for all files that are being processed in losslesscut. I think this may be caused by a flag like -map_metadata 0 or -movflags use_metadata_tags. could you try each of those?

[mifi]

Could you try the new smart cut option? It will apply the same timebase to all segments.

[bryc]

This issue is about merging mp4 files with different timebase values. Videos merged in this way break in VLC and other players when transitioning into the next video stream.

I'm only seeing 'smart cut' when in cut mode during the export options screen. This doesn't seem to popup during a merge operation.

[mifi]

Ah, right. I misunderstood and thought you were cutting+merging which you are obviously not. Maybe losslessCut in the future can set a common timebase also when merging multiple files. However I'm not sure if simply setting -video_track_timescale on the ffmpeg concat command will solve the problem. Also AFAIK -video_track_timescale is only for MP4, and does not work for matroska.

[bryc]

However I'm not sure if simply setting -video_track_timescale on the ffmpeg concat command will solve the problem

Applying video_track_timescale to each segment individually should work. So it'd have to create temporary files similarly to how cut and merge works normally.

does not work for matroska

Does Matroska even have a concept of tbn in the format? FFmpeg seems to always output MKV files with a 1k tbn value, converting back to MP4 results in 16k tbn. No clue if there are ways to create mkv with a different tbn. I could be wrong, but I imagine mkv would not be affected, only MOV/MKV/3GP or any others that can have a different tbn.

[mifi]

I see. I can imagine a merging with first converting tbn for all segments would require double the disk space and double the processing time.
I wonder if merging mp4 files in losslesscut but simply outputting as mkv instead of mp4 will solve the problem, maybe the tbn value will be ignored and the output will be playable

[bryc]

I wonder if merging mp4 files in losslesscut but simply outputting as mkv instead of mp4 will solve the problem, maybe the tbn value will be ignored and the output will be playable

Unfortunately that doesn't work. The concat demuxer treats input the same regardless if the output is mp4 or mkv. Basically, LosslessCut would have to ask the user if they want to modify (preprocess) their input files to have a consistent timescale (which might even be against the project's goals, though a timescale like 90k is quite lossless), or create temp files with altered timescale, there's no other way that I know of.

Alternatively you can detect and alert the user when merging might result in an incompatible output file. This could also be extended to check for mismatching codecs, sample rates or AVC profiles (main vs high vs baseline), all of which can cause issues in a lossless merge, and would require re-encoding to allow proper merging.

[mifi]

There are all good ideas. Definitely at the very least we should warn the user if there is a mismatch.

[mifi]

Someone else reported the same issue #1215

For reference, smart cut now does something like this before merging cut segments:

lossless-cut/src/smartcut.js

Line 60 in ecef901

const timebaseMatch = videoStream.time_base && videoStream.time_base.split('/');

Then it will append the argument:

lossless-cut/src/ffmpeg.js

Line 801 in ecef901

export const getVideoTimescaleArgs = (videoTimebase) => (videoTimebase != null ? ['-video_track_timescale', videoTimebase] : []);

So we could do this when concat/merging with temp files like you described. I will probably implement this in a way so that if timebase differs in two or more files, it will automatically do the intermediary temp file step.

[mifi]

In the next version I'm including a "file check" functionality, where it will show warnings if some parameters are different:

lossless-cut/src/components/ConcatDialog.jsx

Line 84 in fe897b5

['codec_name', 'width', 'height', 'fps', 'pix_fmt', 'level', 'profile', 'sample_fmt', 'r_frame_rate', 'time_base'].some((key) => {

[bryc]

Is it supposed to list all mismatches, or does it only show the first warning?

I tried it with two files that cannot be merged because of a mismatched timebase, but it does not warn about a timebase mismatch, only a framerate mismatch:

Track 0 mismatch: r_frame_rate 128300/4663 != 815921/33918

After fixing the timebase (and creating a playable file), the framerate warning remains, but with different numbers:

Track 0 mismatch: r_frame_rate 55/2 != 289/12

In my experience, framerates can be variable across keyframes, as is the case with adaptive livestreams, but a timebase mismatch will always break players such as VLC.

I think both warnings should be listed.

[mifi]

Thanks for this. I agree we should list all the errors. The reasoning behind showing only the first is that if the files are very different, the list of differing parameters would be huge, so I had decided to just show the first. I will change it so that it instead shows a popup with all the errors when clicking the exclamation icon next to one file.

[cannondale0815]

I know this issue is closed, but I want to mention that the conversion to *.ts, then concatenating those ts files in losslesscut (as mp4 output) got around the issue of the time_base mismatch losslesscut was reporting before.

So @bryc is onto something that ts conversion aligns the time_base. Would be a nice enhancement if losslesscut could internally manage all that, making time_base mismatches easier to deal with :)

[mifi]

this issue is not closed. I intend to implement some fix for this. Not sure why converting to mpeg ts will always(?) convert tbn to 90k (where does that number come from?) I think however preprocessing all videos first and appending the same -video_track_timescale to all the files would be the way to go. However not sure which value to use for video_track_timescale, maybe the value from the first file?

[cannondale0815]

Not sure to be honest. I don't know anything about time_base, but when I encountered the message in llc, I googled for it and came upon this thread here. And just converting it to ts, as @bryc mentioned above, allowed me to proceed in merging several files that otherwise llc couldn't deal with.
The command used was ffmpeg -i input.mp4 -c copy output.ts, exactly as @bryc mentioned.
This interim step works great and alleviated the need for me to perform the concatenation through other means (i.e. DaVinci Resolve), which would have taken hours to transcode.
I'm happy to test any other ffmpeg commands to try out to align the time_base without converting the file to ts, if you tell me which ones :)

P.S.: Reading up a bit more, ts files always appear to have a fixed timescale of 90k as per the top answer in this thread.

[cannondale0815]

I tested further with the video_track_timescale option and have run into issues.
ffmpeg -i input.mp4 -c copy -video_track_timescale 30 output.mp4
I chose 30 for no specific reason (unsure what value I should try with). One of the two files I treated like that is now out of sync with the audio, and I see lots of these messages:
[mp4 @ 0x55d1e00476c0] Non-monotonous DTS in output stream 0:0; previous: 154572, current: 92747; changing to 154573. This may result in incorrect timestamps in the output file.

[mifi]

Correct me if i’m wrong but I think you’re supposed to use 30000 (not 30)

[cannondale0815]

Indeed 30000 worked well! Both individual files are playable after the conversion, as is the llc-concatenated output file. Great! For reference, the original mismatch warning was Track 0 mismatch: time_base 1/1000 != 1/16000.

The only other remaining issue I see is that losslesscut is still giving me a different warning Track 0 mismatch: level 52 != 50. This was also previously remedied by converting to ts (don't know why or how). Is there another flag in ffmpeg to align those as well? Either way, the concatenated file still plays well, so not sure about the actual impact of a "level" mismatch.

[bryc]

@mifi

Not sure why converting to mpeg ts will always(?) convert tbn to 90k (where does that number come from?)

From a comment here: https://stackoverflow.com/questions/43333542/what-is-video-timescale-timebase-or-timestamp-in-ffmpeg

The reason for the typical use of 90,000 as a common base of calculation is that it is a number which is divisible by 24, by 25, and by 30 (in each case the result is an integer - there is no remainder, decimal or fraction), thus the maths is equally suitable for handling 24 frames per second, 25 fps, and 30 fps.

Though in practice, it can be as low as 14625 for 30 fps and still be fine (I think). The only thing that matters is that the videos all have the same value, and that its 'big enough'.

@cannondale0815 As far as I remember, merging works fine with different codec profiles (Baseline / Main / High). Though not 100% sure about different levels.

[cannondale0815]

@bryc Sounds like 90k is the winner then :) I have a bunch of video files from PAL & NTSC recordings, so using a base that works well with all common fps seems like a good idea.

[cannondale0815]

I have to say, the detour via ts conversion is ingenious -- I have a few cases where losslesscut is unable to cut/merge elements from a single large mkv or mp4 file, but when I convert the entire source file to ts beforehand, those same cuts/merges work just fine. I may have to fall back to this more often now :)

[mifi]

updated troubleshooting to include this trick