ffmpeg video decode is not hardware accelerated on Windows

[ywlke]

Expected behaviour

ffmpeg/libavcodec video decoding should be hardware accelerated via D3D11.

Actual behaviour

ffmpeg/libavcodec video decoding is not hardware accelerated.

Steps to reproduce

cap = cv2.VideoCapture('file.mp4')
frame = True
while frame:
frame = cap.read()

Running on Windows 10 x64, Intel Core i7-3520M, Python 3.9, opencv-python 4.5.3.56.
The same test H264 file opened in VLC with D3D11 hardware acceleration via libavcodec at a similar decoding rate uses much less CPU.

Although hwaccels is no longer disabled in the ffmpeg build script (opencv/opencv_3rdparty@7342d31#diff-6862fa01d6f0bc4c9601c1a0a9d170cb49cf255b25bfc66a02f958fa47be43a2), D3D11 support is not getting built in the Windows ffmpeg DLLs used with opencv-python.

I suspect the D3D11 headers and libraries need to be added for support to be enabled at the time of ffmpeg compile, and either need to be borrowed from VLC or compiled using MSVC which includes the headers in the Windows SDK, instead of MinGW or gcc cross-compile.

Issue submission checklist

• This is not a generic OpenCV usage question (looking for help for coding, other usage questions, homework etc.)
• I have read the README of this repository and understand that this repository provides only an automated build toolchain for OpenCV Python packages (there is no actual OpenCV code here)
• The issue is related to the build scripts in this repository, to the pre-built binaries or is a feature request (such as "please enable this additional dependency")
• I'm using the latest version of opencv-python

[asmorkalov]

@ywlke Thanks for the issue report. OpenCV-Python is wrapper for OpenCV library and the projects uses pre-built FFmpeg provided by core project. I'll talk to Alalek if we can enable hardware acceleration in core OpenCV on Windows. As alternative solution you can switch from FFmpeg to Media Foundation backend for VideoCapture/VideoWriter. Build logs indicate that it'd enabled:

  --   Video I/O:
  --     DC1394:                      NO
  --     FFMPEG:                      YES (prebuilt binaries)
  --       avcodec:                   YES (58.134.100)
  --       avformat:                  YES (58.76.100)
  --       avutil:                    YES (56.70.100)
  --       swscale:                   YES (5.9.100)
  --       avresample:                YES (4.0.0)
  --     GStreamer:                   NO
  --     DirectShow:                  YES
  --     Media Foundation:            YES
  --       DXVA:                      YES

To switch media backend you need to use constructor overload with back-end option each time you work with media, e.g.:

cap = cv.VideoCapture("path/to/video.mp4", cv.cv.CAP_MSMF)

Documentation link: https://docs.opencv.org/master/d8/dfe/classcv_1_1VideoCapture.html#ac4107fb146a762454a8a87715d9b7c96

[ywlke]

Thanks for your help. Unfortunately the MFX backend doesn’t support video containers (https://github.com/opencv/opencv/wiki/MediaSDK-encode-decode-backend) but otherwise I agree that would be a good option for hardware acceleration. I guess adding support for containers would be an upstream OpenCV project. From: Alexander Smorkalov ***@***.***> Sent: Tuesday, August 3, 2021 2:29 AM To: opencv/opencv-python ***@***.***> Cc: ywlke ***@***.***>; Mention ***@***.***> Subject: Re: [opencv/opencv-python] ffmpeg video decode is not hardware accelerated on Windows (#520) @ywlke <https://github.com/ywlke> Thanks for the issue report. OpenCV-Python is wrapper for OpenCV library and the projects uses pre-built FFmpeg provided by core project. I'll talk to Alalek if we can enable hardware acceleration in core OpenCV on Windows. As alternative solution you can switch from FFmpeg to Media Foundation backend for VideoCapture/VideoWriter. Build logs indicate that it'd enabled:

…

-- Video I/O: -- DC1394: NO -- FFMPEG: YES (prebuilt binaries) -- avcodec: YES (58.134.100) -- avformat: YES (58.76.100) -- avutil: YES (56.70.100) -- swscale: YES (5.9.100) -- avresample: YES (4.0.0) -- GStreamer: NO -- DirectShow: YES -- Media Foundation: YES -- DXVA: YES To switch media backend you need to use constructor overload with back-end option each time you work with media, e.g.: cap = cv.VideoCapture("path/to/video.mp4", cv.cv.CAP_MSMF) Documentation link: https://docs.opencv.org/master/d8/dfe/classcv_1_1VideoCapture.html#ac4107fb146a762454a8a87715d9b7c96 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#520 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFY3KTNEXRABPKR55GNWPCLT26EDPANCNFSM5BNTSK7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> . <https://github.com/notifications/beacon/AFY3KTICEHGNX3DBQ5JPJETT26EDPA5CNFSM5BNTSK72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGUSE43Y.gif>

[asmorkalov]

OE in core OpenCV repo: opencv/opencv#19676

[ywlke]

Confirmed no difference in decode performance when manually setting the CAP_PROP_HW_ACCELERATION parameter to VIDEO_ACCELERATION_D3D11 vs setting it to VIDEO_ACCELERATION_NONE.