ffmpeg hangs when processing some kinds of unsupported/corrupted media

[jrobble]

Update 5/30/2023:

Related to #1687.

The ffprobe command mentioned below doesn't hang in the OpenMPF 7.1.12 and 7.2.0 Docker images because it uses a more recent version of ffprobe/ffmpeg than what I had installed on my dev. VM when I created this issue.

Now what's happening is that the Workflow Manager in 7.1.12 is locking up in FrameCounter.countNative():

# DEBUG: cpu time is 4.29 hours?
"Camel (camelContext) thread #142 - JmsConsumer[MPF.MEDIA_INSPECTION_WORK_QUEUE]" #296 daemon prio=5 os_prio=0 cpu=15439603.43ms elapsed=15614.40s tid=0x00007efe78013eb0 nid=0x5ea runnable  [0x00007efd724ee000]
   java.lang.Thread.State: RUNNABLE
    at org.mitre.mpf.framecounter.FrameCounter.countNative(Native Method)
    at org.mitre.mpf.framecounter.FrameCounter.count(FrameCounter.java:52)

Even if the media were to get past media inspection it will get locked up in the component when it uses OpenCV to read the video frames. I verified this using the CLI Runner. Thus, this issue still affects 7.2.0 even through we no longer use FrameCounter.countNative() in that version of OpenMPF. The reason this happens is because OpenCV is compiled with ffmpeg dependencies than are older than the dependencies used to install the command-line version of ffprobe/ffmpeg we use during media inspection.

To address this issue, we should run the ffprobe command with -loglevel error. If it generates log messages on the err stream we then generate an OpenMPF media error so that the job fails. We should limit the number of error log lines we capture when generating the OpenMPF error message. We should consider 5 lines and make it a configurable system property.

---

The issue appears to be with the Tika parser. Specifically, this ffmpeg command being used by Tika hangs:

ffmpeg -nostats -i b1xxx.mp4 -map 0:v:0 -c copy -f null -
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[libaom-av1 @ 0x562251cf0fc0] v1.0.0
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data
[libaom-av1 @ 0x562251cf0fc0] obu_reserved_1bit out of range: 1, but must be in [0,0].
[libaom-av1 @ 0x562251cf0fc0] Failed to parse temporal unit.
[libaom-av1 @ 0x562251cf0fc0] Failed to decode frame: Corrupt frame detected
[libaom-av1 @ 0x562251cf0fc0]   Additional information: Failed to decode tile data

Similarly, this ffprobe command hangs:

ffprobe  -print_format json -show_format -show_streams b1xxx.mp4

Hanging like this causes jobs to get backed up in the media inspection queues.

Figure out if there is a way to prevent hanging, or run the ffmpeg / ffprobe system process with a timeout. Note that the next release (7.2) will use ffprobe without Tika, which makes using a timeout easier.