Thoughts on finding broken videos.

[Farmadupe]

in response to feature request in issue #704

It is actually possible to detect broken videos using the cmdline version of ffmpeg (already included in some distributions of Czkawka) by asking it to decode a video into a null output, with options to only write to stderr if an error is encountered:
ffmpeg -v error -i $FILENAME -f null -

The level of error detection can be configured with the -err_detect commandline option.

  -err_detect        <flags>      ED.VAS..... set error detection flags (default 0)
     crccheck                     ED.VAS..... verify embedded CRCs
     bitstream                    ED.VAS..... detect bitstream specification deviations
     buffer                       ED.VAS..... detect improper bitstream length
     explode                      ED.VAS..... abort decoding on minor error detection
     ignore_err                   ED.VAS..... ignore errors
     careful                      ED.VAS..... consider things that violate the spec, are fast to check and have not been seen in the wild as errors
     compliant                    ED.VAS..... consider all spec non compliancies as errors
     aggressive                   ED.VAS..... consider things that a sane encoder should not do as an error

Detecting broken videos will be slow (maybe even slower than creating a video hash) because the entire video must be scanned.

[chapmanjacobd]

I didn't notice a difference when using -err_detect explode but I found a few ways that can help detect corrupt videos.

1. Counting packets is likely the fastest way but not all formats use 1 frame per packet and so the resulting number is difficult to compare against.

2. The next fastest way is probably to copy the audio into a new container (like .mkv). Often, if the file is really messed up, this will provide an accurate way to calculate how much of the file is missing. Alternatively, you could decode the audio into PCM but this takes about twice as long and I haven't found many cases where this actually makes a difference.

3. The next slowest way is to count frames with ffprobe, similar to counting packets

4. Less accurate and slower than options 1 and 2 but faster than option 3 is to decode some frames. Even with a small number of samples you can get a good idea of whether the file is corrupt or not.

5. The slowest but most accurate way is to decode the full file and read the report--but it is quite verbose and difficult to interpret for some codecs:

   ffmpeg -nostats -report -i input.mkv -f null /dev/null

But it's difficult to automate parsing the resulting log because it is full of specific details.

I wrapped the above 1-4 options in the media-check subcommand here:

$ pip install xklb
$ library media-check -h
usage: library media-check [--chunk-size SECONDS] [--gap SECONDS OR 0.0-1.0*DURATION] [--delete-corrupt >0-100] [--full-scan] [--audio-scan] PATH ...

Defaults to decode 0.5 second per 10% of each file

    library media-check ./video.mp4

Decode all the frames of each file to evaluate how corrupt it is (very slow; about 150 seconds for an hour-long file)

    library media-check --full-scan ./video.mp4

Decode all the packets of each file to evaluate how corrupt it is (about one second of each file but only accurate for formats where 1 packet == 1 frame)

    library media-check --full-scan --gap 0 ./video.mp4

Decode all audio of each file to evaluate how corrupt it is (about four seconds per file)

    library media-check --full-scan --audio ./video.mp4

Decode at least one frame at the start and end of each file to evaluate how corrupt it is (takes about one second per file)

    library media-check --chunk-size 0.05 --gap 0.999 ./video.mp4

Decode 3s every 5% of a file to evaluate how corrupt it is (takes about three seconds per file)

    library media-check --chunk-size 3 --gap 0.05 ./video.mp4

Delete the file if 20 percent or more of checks fail

    library media-check --delete-corrupt 20 ./video.mp4

https://github.com/chapmanjacobd/library/blob/main/xklb/media/media_check.py
https://github.com/chapmanjacobd/library#file-subcommands