Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

videoio: HW decode/encode in FFMPEG backend; new properties with support in FFMPEG/GST/MSMF #19460

Merged
merged 60 commits into from
Mar 1, 2021

Conversation

mikhail-nikolskiy
Copy link
Contributor

@mikhail-nikolskiy mikhail-nikolskiy commented Feb 4, 2021

Merge with extra: opencv/opencv_extra#855

Overview

  • add support for HW accelerated decode/encode in FFMPEG backend of VideoCapture and VideoWriter APIs under #ifdef check for FFMPEG version >= 4.0
  • introduce new properties CAP_PROP_HW_ACCELERATION, CAP_PROP_HW_DEVICE, VIDEOWRITER_PROP_HW_ACCELERATION, VIDEOWRITER_PROP_HW_DEVICE and enum VideoAccelerationType.
  • new properties supported in three HW-capable backends: ffmpeg, gstreamer, msmf
  • setting new properties supported only via params parameter in VideoCapture/VideoWriter constructor or open() function, not supported in setProperty() call after open() call
  • by default (if property not set) HW acceleration enabled in HW-capable backends for both VideoCapture and VideoWriter
  • if HW accelerated decoder/encoder not found or failed to initilize, VideoCapture/VideoWriter gracefully falls back to SW decoder/encoder. Video acceleration status (HW or SW codec) could be queried via getProperty(CAP_PROP_HW_ACCELERATION) / getProperty(VIDEOWRITER_PROP_HW_ACCELERATION)
  • new sample video_acceleration
  • test iterating three backends and acceleration types

HW acceleration types support matrix, in priority order:

OS Backend VideoCapture VideoWriter
Linux FFMPEG VAAPI, MFX MFX, VAAPI
  GStreamer VAAPI (and others HW plugins) VAAPI (and others HW plugins)
Windows FFMPEG D3D11, MFX MFX
  MSMF D3D11 -

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
  • The PR is proposed to proper branch
  • There is reference to original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake
build_contrib:Custom=OFF
build_contrib:Custom Win=OFF

Xbuild_image:Win64=msvs2019

force_builders=Custom,Custom Win,Linux AVX2
Xbuild_image:Custom=centos:7
Xbuild_image:Custom=ubuntu:20.04
build_image:Custom=gstreamer:16.04
buildworker:Custom=linux-1,linux-4,linux-6
test_opencl:Custom=ON

Xbuild_image:Custom Win=ffmpeg
build_image:Custom Win=ffmpeg-plugin
buildworker:Custom Win=windows-2
test_modules:Custom Win=videoio
test_opencl:Custom Win=ON

@asmorkalov asmorkalov mentioned this pull request Feb 5, 2021
6 tasks
Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for sharing Linux testing configurations. This is really useful!

modules/videoio/src/cap_ffmpeg_hw.hpp Show resolved Hide resolved
modules/videoio/src/cap_ffmpeg_hw.hpp Outdated Show resolved Hide resolved
modules/videoio/src/cap_ffmpeg_hw.hpp Outdated Show resolved Hide resolved
modules/videoio/src/cap_ffmpeg_hw.hpp Outdated Show resolved Hide resolved
modules/videoio/src/cap_ffmpeg_hw.hpp Show resolved Hide resolved
modules/videoio/src/cap_ffmpeg_hw.hpp Show resolved Hide resolved
modules/videoio/src/cap_ffmpeg_hw.hpp Outdated Show resolved Hide resolved
Comment on lines 212 to 222
static bool hw_check_codec(AVCodec* codec, AVHWDeviceType hw_type) {
// disable MJPG HW encoder due to low encoding quality
if (AV_CODEC_ID_MJPEG == codec->id && av_codec_is_encoder(codec) && AV_HWDEVICE_TYPE_NONE != hw_type)
return false;
// disable AV1 HW decoder due to SW fallback issue
if (AV_CODEC_ID_AV1 == codec->id && av_codec_is_decoder(codec) && AV_HWDEVICE_TYPE_NONE != hw_type)
return false;
// disable VP8 VAAPI decoder/encoder
if (AV_CODEC_ID_VP8 == codec->id && AV_HWDEVICE_TYPE_VAAPI == hw_type)
return false;
return true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to think how to deal with these rules in a bit robust way.
May be we need some runtime configuration here (to bypass / add checks).

Copy link
Contributor Author

@mikhail-nikolskiy mikhail-nikolskiy Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I extended usage of environment variable OPENCV_FFMPEG_CAPTURE_OPTIONS and introduced OPENCV_FFMPEG_WRITER_OPTIONS.
Besides disabling specific codecs, it could be used for disabling specific acceleration type

export OPENCV_FFMPEG_CAPTURE_OPTIONS="disable_codecs;.vaapi"
export OPENCV_FFMPEG_WRITER_OPTIONS="disable_codecs;.vaapi"

or all HW codecs

export OPENCV_FFMPEG_CAPTURE_OPTIONS="disable_codecs;hw"
export OPENCV_FFMPEG_WRITER_OPTIONS="disable_codecs;hw"

or not disable at all (none as value not matching any codec)

export OPENCV_FFMPEG_CAPTURE_OPTIONS="disable_codecs;none"
export OPENCV_FFMPEG_WRITER_OPTIONS="disable_codecs;none"

Syntax schema inherited from OPENCV_FFMPEG_CAPTURE_OPTIONS with separation symbols ; and |, looks little weird:)

export OPENCV_FFMPEG_CAPTURE_OPTIONS="rtsp_transport;udp|disable_codecs;av1.vaapi,av1_qsv"

ffmpeg commands to query names of all decoders and encoders.

ffmpeg -decoders
ffmpeg -encoders

Additional suffix after codec name means acceleration. av1.vaapi is av1 codec working in vaapi mode. av1_qsv (or av1_qsv.qsv) is codec specifically for qsv/mfx.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thank you

or not disable at all (none as value not matching any codec)

Awesome :)
This is extremely useful for testing.

with separation symbols ; and |, looks little weird:)

Agreed... but it was done 3 years ago (#9292), so we need to keep compatibility here.
Hopefully we will add "string" properties/parameters soon through normal API with improved separation symbols.

FFmpeg internally uses different sets internally, so there is no gold approach:

  • "=", ":" (8 times)
  • "=", "," (2 times)
  • some special like "@", "#", AV_DICT_MULTIKEY

modules/videoio/src/cap_ffmpeg_impl.hpp Outdated Show resolved Hide resolved
Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated your scripts and run them with kernel 5.10.17-200.fc33.x86_64 on i5-6600 (Skylake Gen9):
https://github.com/alalek/opencv-test-videoio-hw-accel

To successfully run HW acceleration some extra efforts are required to properly install dependencies. Perhaps we should enable H/W acceleration explicitly through API (open parameters) or by using some runtime flag. Or use whitelist approach. Need to think how to avoid regressions on the Users side.

device_name = va_ctx->str_vendor;
if (hw_type == AV_HWDEVICE_TYPE_QSV) {
// Workaround for issue fixed in MediaSDK 21.x https://github.com/Intel-Media-SDK/MediaSDK/issues/2595
// Checks VAAPI driver for support of VideoProc operation required by MediaSDK
Copy link
Member

@alalek alalek Feb 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this check is not enough with i965 libva driver.

  • do we need to support this driver or we can disable it and keep iHD only? (something like GST_VAAPI_ALL_DRIVERS option)

You can compare fedora-i965-have_va.Dockerfile vs fedora-i965.Dockerfile results here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We better not use i965 as it's deprecated and not really maintained. Though ffmpeg "knows" this driver.

Starting some Core generation (ex, on Core i7-10xxx) the i965 driver doesn't init even if installed.

@mikhail-nikolskiy
Copy link
Contributor Author

mikhail-nikolskiy commented Feb 26, 2021

I added parameter hw_acceleration to environment variables OPENCV_FFMPEG_CAPTURE_OPTIONS and OPENCV_FFMPEG_WRITER_OPTIONS with the following default values

#ifdef _WIN32
#define FFMPEG_DECODE_ACCELERATION_TYPES    "d3d11va"
#define FFMPEG_ENCODE_ACCELERATION_TYPES    "qsv"
#else
#define FFMPEG_DECODE_ACCELERATION_TYPES    "vaapi.iHD"
#define FFMPEG_ENCODE_ACCELERATION_TYPES    "qsv.iHD,vaapi.iHD"
#endif

For example, "qsv" allows mfx acceleration on any driver, "qsv.iHD" allows mfx acceleration on driver with name containing case-sensitive substring 'iHD' (similar how ffmpeg detects driver), "vaapi.VDPAU" on VAAPI VDPAU driver if such installed.
Acceleration types named according to ffmpeg.

Probably makes sense to change value of CAP_PROP_HW_ACCELERATION parameter from enum VideoAccelerationType to string, so that user can specify/override list of acceleration types? This would require support for string type as parameter value.

Totally agree that opening on default parameters must not introduce any degradation across all OS and HW setups, should either use fully functional HW codec or fallback to SW codec.
Is the following current state in master,
MSMF: HW decode by default (exposes custom param to disable), no HW encode.
GStreamer: HW decode by default via decodebin, HW encode by default (?) via encodebin.

On Windows, "d3d11va" without dot (like "d3d11va.Intel" to check for driver/vendor name) probably safer than Linux because DirectX drivers supposed to pass Microsoft certification?

Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mikhail-nikolskiy Great job! Thank you for contribution 👍

@alalek alalek merged commit 7bcb51e into opencv:master Mar 1, 2021
@@ -1321,6 +1469,9 @@ bool CvCapture_FFMPEG::retrieveFrame(int, unsigned char** data, int* step, int*
*height = frame.height;
*cn = frame.cn;

if (sw_picture != picture) {
av_frame_unref(sw_picture);
Copy link
Contributor

@mshabunin mshabunin Mar 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a build problem with older FFmpeg:

 In file included from ../opencv/modules/videoio/src/cap_ffmpeg.cpp:50:0:
 ../opencv/modules/videoio/src/cap_ffmpeg_impl.hpp: In member function 'bool CvCapture_FFMPEG::retrieveFrame(int, unsigned char**, int*, int*, int*, int*)':
 ../opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:1473:34: error: 'av_frame_unref' was not declared in this scope
          av_frame_unref(sw_picture);
                                   ^
      avcodec:                   YES (54.35.0)
      avformat:                  YES (54.20.4)
      avutil:                    YES (52.3.0)
      swscale:                   YES (2.1.1)
      avresample:                YES (1.0.1)

Perhaps libavutil/frame.h should be included directly in this file.
Looks like it should be guarded by USE_AV_FRAME_GET_BUFFER macro.

cc @alalek

@alalek alalek mentioned this pull request Apr 9, 2021
a-sajjad72 pushed a commit to a-sajjad72/opencv that referenced this pull request Mar 30, 2023
videoio: HW decode/encode in FFMPEG backend; new properties with support in FFMPEG/GST/MSMF

* HW acceleration in FFMPEG backend

* fixes on Windows, remove D3D9

* HW acceleration in FFMPEG backend

* fixes on Windows, remove D3D9

* improve va test

* Copyright

* check LIBAVUTIL_BUILD >= AV_VERSION_INT(55, 78, 100) // FFMPEG 3.4+

* CAP_MSMF test on .mp4

* .mp4 in test

* improve va test

* Copyright

* check LIBAVUTIL_BUILD >= AV_VERSION_INT(55, 78, 100) // FFMPEG 3.4+

* CAP_MSMF test on .mp4

* .mp4 in test

* .avi for GStreamer test

* revert changes around seek()

* cv_writer_open_with_params

* params.warnUnusedParameters

* VideoCaptureParameters in GStreamer

* open_with_params

* params->getUnused

* Reduce PSNR threshold 33->32 (other tests use 30)

* require FFMPEG 4.0+; PSNR 30 as in other tests

* GStreamer AVI-demux plugin not installed in Ubuntu test environment?

* fix build on very old ffmpeg

* fix build on very old ffmpeg

* fix build issues

* fix build issues (static_cast)

* FFMPEG built on Windows without H264 encoder?

* fix for write_nothing test on VAAPI

* fix warnings

* fix cv_writer_get_prop in plugins

* use avcodec_get_hw_frames_parameters; more robust fallback to SW codecs

* internal function hw_check_device() for device check/logging

* two separate tests for HW read and write

* image size 640x480 in encode test

* WITH_VA=ON (only .h headers used in OpenCV, no linkage dependency)

* exception on VP9 SW encoder?

* rebase master; refine info message

* videoio: fix FFmpeg standalone plugin build

* videoio(ffmpeg): eliminate MSVC build warnings

* address review comments

* videoio(hw): update videocapture_acceleration.read test

- remove parallel decoding by SW code path
- check PSNR against the original generated image

* videoio: minor fixes

* videoio(test): disable unsupported MSMF cases (SW and HW)

* videoio(test): update PSNR thresholds for HW acceleration read

* videoio(test): update debug messages

* "hw_acceleration" whitelisting parameter

* little optimization in test

* D3D11VA supports decoders, doesn't support encoders

* videoio(test): adjust PSNR threshold in write_read_position tests

* videoio(ffmpeg): fix rejecting on acceleration device name mismatch

* videoio(ffmpeg): fix compilation USE_AV_HW_CODECS=0, add more debug logging

* videoio: rework VideoAccelerationType behavior

- enum is not a bitset
- default value is backend specific
- only '_NONE' and '_ANY' may fallback on software processing
- specific H/W acceleration doesn't fallback on software processing. It fails if there is no support for specified H/W acceleration.

* videoio(test): fix for current FFmpeg wrapper

Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants