VAAPI: Extend and refactor VAAPI/OpenGL interop

- there are now 3 different methods for displaying VAAPI frames
-- the old but improved GLX copy which works with GLX but not with
OpenGLES
-- the new GLX Pixmap support which works with GLX and OpenGLES
-- the new and performant DRM/DMA interop - which maps VAAPI surfaces
directly to textures and hence is zero copy but is only available when
using EGL.

- both GLX routes use VAAPI internals for colourspace conversion and
deinterlacing (still only basic at the moment)
- EGL/DRM returns raw I420 or NV12 frames that can be passed into the
OpenGL shaders - or when implemented, through the VAAPI post processing
functionality first.

- MythOpenGLInterop and its new VAAPI subclasses have finally found their
home embedded in the FFmpeg hardware frames context
- the OpenGL video output classes are now entirely codec agnostic and we
avoid any references to the hardware context by having GLX interops use
QObject signalling to pick up picture attribute changes.

- workaround MPEG2 seek errors by increasing the packet error count in
AvFormatDecoder - that code probably needs to go.

Issues:
- pause frame for interop is broken - we retain no reference to old
frames and the frames are cleared (for reference counting purposes) when
we have finished displaying them. Just need to retain old frames in the
same way that VDPAU does.
- full deinterlacing support for EGL/DRM needs more work, as does the
OpenGLVideo integration generally.
- there are new driver/hardware specific workarounds in VAAPIContext.
- they will probably need extending for other intel chipsets
- there is an intermittent error with certain H.264 EGL/DRM streams.
Something in OpenGLVideo state management I think.
- the buildbots will probably fall over in 5 minutes...

mythtv/libs/libmythtv/avformatdecoder.cpp

@@ -104,9 +104,14 @@ __inline AVRational GetAVTimeBaseQ()
 
 #define LOC QString("AFD: ")
 
-// Maximum number of sequential invalid data packet errors
-// before we try switching to software decoder
-#define SEQ_PKT_ERR_MAX 10
+// Maximum number of sequential invalid data packet errors before we try
+// switching to software decoder. Packet errors are often seen when using
+// hardware contexts and, for example, seeking. Hence this needs to be high and
+// is probably best removed as it is treating the symptoms and not the cause
+// (e.g. VAAPI2 does not currently do any hardware support checks).
+// See also comment in MythCodecMap::freeCodecContext re trying to free an
+// active hardware context when it is errored.
+#define SEQ_PKT_ERR_MAX 50
 
 static const int max_video_queue_size = 220;
 
@@ -1557,7 +1562,7 @@ enum AVPixelFormat get_format_vaapi(struct AVCodecContext* ctx,
     while (*valid_fmts != AV_PIX_FMT_NONE)
     {
         if (*valid_fmts == AV_PIX_FMT_VAAPI)
-            if (VAAPIContext::HwDecoderInit(ctx) >= 0)
+            if (VAAPIContext::InitialiseDecoder(ctx) >= 0)
                 return *valid_fmts;
         valid_fmts++;
     }
@@ -2590,7 +2595,7 @@ int AvFormatDecoder::ScanStreams(bool novideo)
                 {
                     MythCodecID vaapi_mcid;
                     AVPixelFormat pix_fmt = AV_PIX_FMT_YUV420P;
-                    vaapi_mcid = VAAPIContext::GetBestSupportedCodec(enc, &codec, dec, mpeg_version(enc->codec_id), pix_fmt);
+                    vaapi_mcid = VAAPIContext::GetSupportedCodec(enc, &codec, dec, mpeg_version(enc->codec_id), pix_fmt);
 
                     if (codec_is_vaapi(vaapi_mcid))
                     {
@@ -3200,8 +3205,7 @@ int get_avf_buffer_vaapi(struct AVCodecContext *c, AVFrame *pic, int flags)
     frame->priv[0] = reinterpret_cast<unsigned char*>(av_buffer_ref(pic->buf[0]));
     // frame->hw_frames_ctx contains a reference to the AVHWFramesContext. Take an additional
     // reference to ensure AVHWFramesContext is not released until we are finished with it.
-    // This also gives the video classes access to the
-    // underlying VAAPI context (and VADisplay) that FFmpeg is using.
+    // This also gives the video classes access to the underlying VAAPI interop.
     frame->priv[1] = reinterpret_cast<unsigned char*>(av_buffer_ref(pic->hw_frames_ctx));
 
     // Set release method

mythtv/libs/libmythtv/libmythtv.pro

@@ -483,9 +483,9 @@ using_frontend {
 
     using_vaapi {
         DEFINES += USING_VAAPI
-        HEADERS += vaapicontext.h
-        SOURCES += vaapicontext.cpp
-        LIBS    += -lva -lva-x11 -lva-glx
+        HEADERS += vaapicontext.h mythvaapiinterop.h
+        SOURCES += vaapicontext.cpp mythvaapiinterop.cpp
+        LIBS    += -lva -lva-x11 -lva-glx -lEGL
         using_opengl_video:DEFINES += USING_GLVAAPI
     }
 

mythtv/libs/libmythtv/mythavutil.cpp

@@ -436,6 +436,9 @@ AVCodecContext *MythCodecMap::hasCodecContext(const AVStream *stream)
     return streamMap.value(stream, nullptr);
 }
 
+/// \note This will not free a hardware or frames context that is in anyway referenced outside
+/// of the decoder. Probably need to force the VideoOutput class to discard buffers
+/// as well. Leaking hardware contexts is a very bad idea.
 void MythCodecMap::freeCodecContext(const AVStream *stream)
 {
     QMutexLocker lock(&mapLock);