Difficulty in cutting h264 recordings

[Jpilk]

• Platform:Fedora 35 x86_64

• MythTV version:

• Package version:mythtv-33.Pre.497.g8e36123478-100.fc35.x86_64.rpm

• Component:

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behaviour?

What do you see instead?

Additional information

I have returned to my scripts using 'ffmpeg' to cut h264 recordings. I have one that usually gives results acceptable to me, but only after a preliminary run in which no cuts are applied. Without that step the cutting is wildly inaccurate. It isn't clear to me exactly what makes the difference; I think it's probably to do with the timestamps.

FWIW the same script behaves similarly with mpeg2 recordings, but for those I know about, and use, alternatives.

Here are brief stream analyses for the final, intermediate and initial files:

Final, after cutting:

ionice -c 3 mythffmpeg -hide_banner -i /home/john/SGs/RecsSG2/20055_20220629205800.ts
[h264 @ 0x55b4ef7adf00] mmco: unref short failure
Last message repeated 1 times
Input #0, mpegts, from '/home/john/SGs/RecsSG2/20055_20220629205800.ts':
Duration: 00:40:19.24, start: 1.400000, bitrate: 1172 kb/s
Stream #0:0[0x100]: Video: h264 (High), yuv420p(tv, bt470bg, top first), 544x576 [SAR 64:33 DAR 544:297], 25 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:10x101: Audio: aac_latm (LC), 48000 Hz, stereo, fltp
At least one output file must be specified
#############

Intermediate., uncut:

ionice -c 3 mythffmpeg -hide_banner -i /home/john/SGs/RecsSG2/20055_20220629205800.ts_370262in.ts
[h264 @ 0x555f5c68cf00] mmco: unref short failure
Last message repeated 1 times
Input #0, mpegts, from '/home/john/SGs/RecsSG2/20055_20220629205800.ts_370262in.ts':
Duration: 01:05:57.82, start: 1.400000, bitrate: 1179 kb/s
Stream #0:0[0x100]: Video: h264 (High), yuv420p(tv, bt470bg, top first), 544x576 [SAR 64:33 DAR 544:297], 25 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:10x101: Audio: aac_latm (LC), 48000 Hz, stereo, fltp
At least one output file must be specified

##################

As recorded:

ionice -c 3 mythffmpeg -hide_banner -i /home/john/SGs/RecsSG2/20055_20220629205800.ts_369556in.ts
[h264 @ 0x56079d5e4f00] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0x56079d5e4f00] decode_slice_header error
[h264 @ 0x56079d5e4f00] no frame!
[h264 @ 0x56079d5e4f00] non-existing PPS 0 referenced
Last message repeated 1 times

There's a full page of this, then

[h264 @ 0x56079d5e4f00] decode_slice_header error
[h264 @ 0x56079d5e4f00] no frame!
[h264 @ 0x56079d5e4f00] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0x56079d5e4f00] decode_slice_header error
[h264 @ 0x56079d5e4f00] no frame!
[h264 @ 0x56079d5e4f00] mmco: unref short failure
Last message repeated 1 times
[mpegts @ 0x56079d5e0e00] start time for stream 2 is not set in estimate_timings_from_pts
[mpegts @ 0x56079d5e0e00] stream 2 : no TS found at start of file, duration not set
[mpegts @ 0x56079d5e0e00] Could not find codec parameters for stream 2 (Audio: aac_latm, 0 channels, fltp): unspecified sample rate
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, mpegts, from '/home/john/SGs/RecsSG2/20055_20220629205800.ts_369556in.ts':
Duration: 01:05:57.82, start: 59402.916156, bitrate: 1192 kb/s
Stream #0:0[0x1d4d]: Video: h264 (High), yuv420p(tv, bt470bg, top first), 544x576 [SAR 64:33 DAR 544:297], 24.92 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:10x1d4e: Audio: aac_latm (LC), 48000 Hz, stereo, fltp
Stream #0:20x1d52: Audio: aac_latm, 0 channels, fltp
Stream #0:30x1d51: Subtitle: dvb_subtitle
At least one output file must be specified

###########

[Jpilk]

The problem occurs when 'ffmpeg' is cutting the wanted segments from the raw recording. They are much longer than intended.

[john@HPFed Bashtests]$ ls -Llh /home/john/SGs/RecsSG3/20055_20220702074100raw.ts
-rw-r--r-- 1 john john 202M Jul 2 09:51 /home/john/SGs/RecsSG3/20055_20220702074100raw.ts
[john@HPFed Bashtests]$ ls -lh /home/john/SGs/RecsSG3/20055_20220702074100cutfromraw.ts
-rw-rw-r-- 1 john john 550M Jul 2 09:40 /home/john/SGs/RecsSG3/20055_20220702074100cutfromraw.ts

ionice -c3 mythffmpeg -hide_banner -ignore_unknown -noaccurate_seek -ss 4.9000 -t 54.8400 -i /home/john/SGs/RecsSG3/20055_20220702074100.ts -vcodec copy -acodec copy -avoid_negative_ts make_zero -f mpegts /var/RSG4/TScut_work/20055_20220702074100_184524_001.ts

Working from the raw recording:
[john@HPFed Bashtests]$ ls -lh /var/RSG4/TScut_work/20055_20220702074100_182878_00*.ts
-rw-rw-r-- 1 john john 199M Jul 2 09:40 /var/RSG4/TScut_work/20055_20220702074100_182878_001.ts
-rw-rw-r-- 1 john john 185M Jul 2 09:40 /var/RSG4/TScut_work/20055_20220702074100_182878_002.ts
-rw-rw-r-- 1 john john 167M Jul 2 09:40 /var/RSG4/TScut_work/20055_20220702074100_182878_003.ts
[john@HPFed Bashtests]$
[john@HPFed Bashtests]$
After preliminary run with no cutlist, mythcommflag--rebuild, reset the same cutlist...
[john@HPFed Bashtests]$ ls -lh /var/RSG4/TScut_work/20055_20220702074100_184524_00*.ts
-rw-rw-r-- 1 john john 9.4M Jul 2 10:00 /var/RSG4/TScut_work/20055_20220702074100_184524_001.ts
-rw-rw-r-- 1 john john 9.3M Jul 2 10:00 /var/RSG4/TScut_work/20055_20220702074100_184524_002.ts
-rw-rw-r-- 1 john john 30M Jul 2 10:00 /var/RSG4/TScut_work/20055_20220702074100_184524_003.ts
[john@HPFed Bashtests]$

#########
----Segments are

file /var/RSG4/TScut_work/20055_20220702074100_184524_001.ts
file /var/RSG4/TScut_work/20055_20220702074100_184524_002.ts
file /var/RSG4/TScut_work/20055_20220702074100_184524_003.ts

File segments have been written, Concatenating.

ionice -c3 mythffmpeg -hide_banner -ignore_unknown -fflags +genpts -f concat -safe 0 -i seglist_184524 -c copy /mnt/sdb1/Vids/20055_20220702074100_184524.ts
Output file is /home/john/SGs/RecsSG3/20055_20220702074100.ts

I think the essential difference from the raw recording format is ' fflags +genpts '

[rcrdnalor]

Some thoughts came into my mind when reading this issue:
Did you mean: "I've downloaded a program or recorded an aired DVB program
and FFMPEG provided by MythTV master can not cope with this recording in
respect to cut and transcode this recording?"

In order to eliminate any impacts which might not be under control of MythTV,
please consider the following items:

• Output data rely on the quality of the input.
  In order to check the input quality, you can use a script like
  https://forum.mythtv.org/viewtopic.php?f=2&t=4013&p=19636&hilit=check+recording+quality#p19688
  or, if recorded via DVB, check for reception, e.g.: by
  https://www.linuxtv.org/wiki/index.php/Dvbv5-zap
  https://www.linuxtv.org/wiki/index.php/Dvb-fe-tool

• In case of a DVB recording, you can try to eliminate MythTV completely, like
  saving a DVB stream to disk. See programs in
  https://www.linuxtv.org/wiki/index.php/TV_Related_Software

• If you think that FFMPEG is part of your issue, then try static builds
  of FFMPEG, like
  https://johnvansickle.com/ffmpeg/

• I successfully ran 'avidemux 2.7' on recordings in order to cut and trancode them.
  Please download and install from trusted source this version and report the results.

• The issue mentions an output of 'ffmpeg -i' on a pristine recording:

  As recorded:
  ionice -c 3 mythffmpeg -hide_banner -i /home/john/SGs/RecsSG2/20055_20220629205800.ts_369556in.ts
  

  Please confirm that this recording named with a non MythTV conpliant name is actually
  the recording 'as recorded by MythTV'.

Additional tweaks within MythTV which are worth to try:

• MythTV provides the ability to remove unwanted streams during recording:
  What does mythtv-setup tell you in
  Recording Profiles -> Hardware DVB Encoders -> Default -> Recording Type
  If set to 'TV only', try 'all' instead.

• Try 'mythtranscode' with the '--fifodir' option and compare results.
  See
  https://www.mythtv.org/wiki/Example_Script_using_mythtranscode_in_fifodir_mode
  https://www.mythtv.org/wiki/Mythtranscode

In general, this issue does not provide detailed information for me about the
reason and reproducibility (is it a regression?).
Please provide your scripts and the source file, i.e.: the pristine recording file.

[Jpilk]

Thank you for this very full response. I don't expect to try using many of the tools that you suggest soon, because my script now usually does what I want, but it's good to have the references.

I created this Issue because it appears that the underlying cause has existed for years. In particular, my script logs for cutting h264 recordings often complain, as below. I had found no useful suggestions about how to "Fix my code". I'm not suggesting that there has been a reversion.

• "Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly"

You asked about non-MythTV-compliant filenames. I rename input files so that the script output can replace the original.

Thanks again.

[Jpilk]

Another reference that might be relevant here:

https://forum.mythtv.org/viewtopic.php?t=4400#p21724

Almost all broadcast mpeg-ts has an offset in AV packets (with same PTS) & never uses IDR I frames..
AIUI Mythtv uses the first video packet as time zero, almost all other AV software (mkvtoolnix, vlc etc) uses the first (earliest PTS).
I've struggled to find a ffmpeg/ffprobe/mediainfo cmd to extract that offset.

[Jpilk]

A new aspect of this has come to light after a recent thread on the users list.

Using a command line like

mythffmpeg -fflags +genpts -i inputfile -acodec copy -vcodec copy -scodec copy -f mpegts outputfile

apparently allowed a suggested cutting script to work when the 'outfile' had replaced the original recording. This has been reported earlier in several other contexts.

It seemed worth trying the same procedure on the output of my old 'pycut.py' script, which simply copies and concatenates segments of the original recording, where the cuts are made at video keyframes. Direct playback had been mostly good, but usually with transients at the joins, where the timestamps jump. More recently I have used ffmpeg concatenation as outlined above; better but not perfect.

So far my experience with this new scheme is limited, but the transients seem much less noticeable during playback with both mythfrontend and leanfront. As I have it at present both the 'pycutting' and the remuxing are followed by seektable rebuilds and other actions needed to create a replacement recording, but it ought to be possible to drop the first set, and possibly to use pipes instead of multiple large working files.

My old scripts, in various states of revision, are still accessible through https://www.mythtv.org/wiki/MythDVBcut