-
-
Notifications
You must be signed in to change notification settings - Fork 56k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
videoio: HW decode/encode in FFMPEG backend; new properties with support in FFMPEG/GST/MSMF #19460
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for sharing Linux testing configurations. This is really useful!
static bool hw_check_codec(AVCodec* codec, AVHWDeviceType hw_type) { | ||
// disable MJPG HW encoder due to low encoding quality | ||
if (AV_CODEC_ID_MJPEG == codec->id && av_codec_is_encoder(codec) && AV_HWDEVICE_TYPE_NONE != hw_type) | ||
return false; | ||
// disable AV1 HW decoder due to SW fallback issue | ||
if (AV_CODEC_ID_AV1 == codec->id && av_codec_is_decoder(codec) && AV_HWDEVICE_TYPE_NONE != hw_type) | ||
return false; | ||
// disable VP8 VAAPI decoder/encoder | ||
if (AV_CODEC_ID_VP8 == codec->id && AV_HWDEVICE_TYPE_VAAPI == hw_type) | ||
return false; | ||
return true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to think how to deal with these rules in a bit robust way.
May be we need some runtime configuration here (to bypass / add checks).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I extended usage of environment variable OPENCV_FFMPEG_CAPTURE_OPTIONS and introduced OPENCV_FFMPEG_WRITER_OPTIONS.
Besides disabling specific codecs, it could be used for disabling specific acceleration type
export OPENCV_FFMPEG_CAPTURE_OPTIONS="disable_codecs;.vaapi"
export OPENCV_FFMPEG_WRITER_OPTIONS="disable_codecs;.vaapi"
or all HW codecs
export OPENCV_FFMPEG_CAPTURE_OPTIONS="disable_codecs;hw"
export OPENCV_FFMPEG_WRITER_OPTIONS="disable_codecs;hw"
or not disable at all (none as value not matching any codec)
export OPENCV_FFMPEG_CAPTURE_OPTIONS="disable_codecs;none"
export OPENCV_FFMPEG_WRITER_OPTIONS="disable_codecs;none"
Syntax schema inherited from OPENCV_FFMPEG_CAPTURE_OPTIONS with separation symbols ; and |, looks little weird:)
export OPENCV_FFMPEG_CAPTURE_OPTIONS="rtsp_transport;udp|disable_codecs;av1.vaapi,av1_qsv"
ffmpeg commands to query names of all decoders and encoders.
ffmpeg -decoders
ffmpeg -encoders
Additional suffix after codec name means acceleration. av1.vaapi is av1 codec working in vaapi mode. av1_qsv (or av1_qsv.qsv) is codec specifically for qsv/mfx.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! Thank you
or not disable at all (none as value not matching any codec)
Awesome :)
This is extremely useful for testing.
with separation symbols ; and |, looks little weird:)
Agreed... but it was done 3 years ago (#9292), so we need to keep compatibility here.
Hopefully we will add "string" properties/parameters soon through normal API with improved separation symbols.
FFmpeg internally uses different sets internally, so there is no gold approach:
- "=", ":" (8 times)
- "=", "," (2 times)
- some special like "@", "#", AV_DICT_MULTIKEY
- remove parallel decoding by SW code path - check PSNR against the original generated image
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have updated your scripts and run them with kernel 5.10.17-200.fc33.x86_64 on i5-6600 (Skylake Gen9):
https://github.com/alalek/opencv-test-videoio-hw-accel
To successfully run HW acceleration some extra efforts are required to properly install dependencies. Perhaps we should enable H/W acceleration explicitly through API (open parameters) or by using some runtime flag. Or use whitelist approach. Need to think how to avoid regressions on the Users side.
device_name = va_ctx->str_vendor; | ||
if (hw_type == AV_HWDEVICE_TYPE_QSV) { | ||
// Workaround for issue fixed in MediaSDK 21.x https://github.com/Intel-Media-SDK/MediaSDK/issues/2595 | ||
// Checks VAAPI driver for support of VideoProc operation required by MediaSDK |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this check is not enough with i965 libva driver.
- do we need to support this driver or we can disable it and keep
iHD
only? (something likeGST_VAAPI_ALL_DRIVERS
option)
You can compare fedora-i965-have_va.Dockerfile
vs fedora-i965.Dockerfile
results here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We better not use i965 as it's deprecated and not really maintained. Though ffmpeg "knows" this driver.
Starting some Core generation (ex, on Core i7-10xxx) the i965 driver doesn't init even if installed.
I added parameter
For example, "qsv" allows mfx acceleration on any driver, "qsv.iHD" allows mfx acceleration on driver with name containing case-sensitive substring 'iHD' (similar how ffmpeg detects driver), "vaapi.VDPAU" on VAAPI VDPAU driver if such installed. Probably makes sense to change value of CAP_PROP_HW_ACCELERATION parameter from enum VideoAccelerationType to string, so that user can specify/override list of acceleration types? This would require support for string type as parameter value. Totally agree that opening on default parameters must not introduce any degradation across all OS and HW setups, should either use fully functional HW codec or fallback to SW codec. On Windows, "d3d11va" without dot (like "d3d11va.Intel" to check for driver/vendor name) probably safer than Linux because DirectX drivers supposed to pass Microsoft certification? |
- enum is not a bitset - default value is backend specific - only '_NONE' and '_ANY' may fallback on software processing - specific H/W acceleration doesn't fallback on software processing. It fails if there is no support for specified H/W acceleration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mikhail-nikolskiy Great job! Thank you for contribution 👍
@@ -1321,6 +1469,9 @@ bool CvCapture_FFMPEG::retrieveFrame(int, unsigned char** data, int* step, int* | |||
*height = frame.height; | |||
*cn = frame.cn; | |||
|
|||
if (sw_picture != picture) { | |||
av_frame_unref(sw_picture); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have a build problem with older FFmpeg:
In file included from ../opencv/modules/videoio/src/cap_ffmpeg.cpp:50:0:
../opencv/modules/videoio/src/cap_ffmpeg_impl.hpp: In member function 'bool CvCapture_FFMPEG::retrieveFrame(int, unsigned char**, int*, int*, int*, int*)':
../opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:1473:34: error: 'av_frame_unref' was not declared in this scope
av_frame_unref(sw_picture);
^
avcodec: YES (54.35.0)
avformat: YES (54.20.4)
avutil: YES (52.3.0)
swscale: YES (2.1.1)
avresample: YES (1.0.1)
Perhaps libavutil/frame.h
should be included directly in this file.
Looks like it should be guarded by USE_AV_FRAME_GET_BUFFER
macro.
cc @alalek
videoio: HW decode/encode in FFMPEG backend; new properties with support in FFMPEG/GST/MSMF * HW acceleration in FFMPEG backend * fixes on Windows, remove D3D9 * HW acceleration in FFMPEG backend * fixes on Windows, remove D3D9 * improve va test * Copyright * check LIBAVUTIL_BUILD >= AV_VERSION_INT(55, 78, 100) // FFMPEG 3.4+ * CAP_MSMF test on .mp4 * .mp4 in test * improve va test * Copyright * check LIBAVUTIL_BUILD >= AV_VERSION_INT(55, 78, 100) // FFMPEG 3.4+ * CAP_MSMF test on .mp4 * .mp4 in test * .avi for GStreamer test * revert changes around seek() * cv_writer_open_with_params * params.warnUnusedParameters * VideoCaptureParameters in GStreamer * open_with_params * params->getUnused * Reduce PSNR threshold 33->32 (other tests use 30) * require FFMPEG 4.0+; PSNR 30 as in other tests * GStreamer AVI-demux plugin not installed in Ubuntu test environment? * fix build on very old ffmpeg * fix build on very old ffmpeg * fix build issues * fix build issues (static_cast) * FFMPEG built on Windows without H264 encoder? * fix for write_nothing test on VAAPI * fix warnings * fix cv_writer_get_prop in plugins * use avcodec_get_hw_frames_parameters; more robust fallback to SW codecs * internal function hw_check_device() for device check/logging * two separate tests for HW read and write * image size 640x480 in encode test * WITH_VA=ON (only .h headers used in OpenCV, no linkage dependency) * exception on VP9 SW encoder? * rebase master; refine info message * videoio: fix FFmpeg standalone plugin build * videoio(ffmpeg): eliminate MSVC build warnings * address review comments * videoio(hw): update videocapture_acceleration.read test - remove parallel decoding by SW code path - check PSNR against the original generated image * videoio: minor fixes * videoio(test): disable unsupported MSMF cases (SW and HW) * videoio(test): update PSNR thresholds for HW acceleration read * videoio(test): update debug messages * "hw_acceleration" whitelisting parameter * little optimization in test * D3D11VA supports decoders, doesn't support encoders * videoio(test): adjust PSNR threshold in write_read_position tests * videoio(ffmpeg): fix rejecting on acceleration device name mismatch * videoio(ffmpeg): fix compilation USE_AV_HW_CODECS=0, add more debug logging * videoio: rework VideoAccelerationType behavior - enum is not a bitset - default value is backend specific - only '_NONE' and '_ANY' may fallback on software processing - specific H/W acceleration doesn't fallback on software processing. It fails if there is no support for specified H/W acceleration. * videoio(test): fix for current FFmpeg wrapper Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
Merge with extra: opencv/opencv_extra#855
Overview
params
parameter in VideoCapture/VideoWriter constructor or open() function, not supported in setProperty() call after open() callvideo_acceleration
HW acceleration types support matrix, in priority order:
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.