ProRes Hardware Acceleration #3874
Comments
|
Similar to many open source projects IINA is layered on top of code from many other projects. For video/audio playback IINA uses mpv and FFmpeg. This means that IINA developers categorize feature requests and bug requests based on whether they are "pure IINA", as in something that can be fully completed by changing IINA code, or something that requires changes to other projects. When changes are required to projects IINA is dependent upon it is considered an "upstream" issue. Decoding using hardware acceleration lands in FFmpeg. From FFmpeg HWAccelIntro:
Seems like it should already be supported. Checking the FFmpeg Bug Tracker I found ticket 9546, "Videotoolbox Prores and VP9 does not work". That was fixed 2021-12-22. Not clear when the problem was introduced. This brings up the question of the version of FFmpeg IINA is using. The latest FFmpeg release is version 5.0.1 which was released 2022-04-04 and should have the fix. IINA is using FFmpeg 4.4.2 which was released 2022-04-14, but was built from the 4.4 release branch, which may or may not have the fix. IINA is not using FFmpeg 5 due to reports of regressions under Intel Macs related to hardware acceleration. Currently we are watching these FFmpeg tickets for fixes:
Upgrading to FFmpeg 5.0.1 may provide support for ProRes decode using hardware acceleration, but may introduce some regressions. Would be good to have a small ProRes video sample that reproduces the failure to use hardware acceleration. |
|
Try the demo in the Apple Store, where they use Apple ProRes 4444, I have a five second copy on my laptop, and the hardware acceleration is not working |
|
Thank you for the detailed reply! I can confirm that an out-of-the-box install of IINA does NOT support ProRes hardware acceleration. I have tested this with all types of ProRes files (422LT, 422, 4444 etc, many different resolutions), and all of them use significant CPU resources to play back. If I play any of these files in Quicktime Player, CPU usage is near-zero.
How can I upgrade the FFmpeg version that IINA uses? I only use Apple Silicon Macs these days, so any Intel-related regressions do not apply. Thanks again. |
|
Sorry, I'm currently using a MBP with M1 Pro, and I'm experiencing the exact same thing; however, I have no idea how to switch the FFmpeg version of IINA, guess we'll have to wait for a Pro to reply. |
|
I always dislike posting a half-done rushed analysis, but I ran out of time looking into this today. So this reply has some useful information, but more investigation is needed. Executive SummaryThe quick answer...
More details than you probably care to know follows... IINA & mpv & FFmpegThe IINA application embeds code from many open source projects. You can view the code that is embedded in the application, but you must not alter these files in any way whatsoever. Any changes will invalidate Code Signing causing macOS to terminate IINA at startup with a Listing the 3rd party code embedded in IINA using Terminal: Embedded:low-batt@gag ~$ ls /Applications/IINA.app/Contents/Frameworks/
Sparkle.framework libfontconfig.1.dylib liblept.5.dylib libsodium.23.dylib libswiftFoundation.dylib libuchardet.0.dylib
libXau.6.dylib libfreetype.6.dylib libluajit-5.1.2.dylib libsoxr.0.dylib libswiftIOKit.dylib libunistring.2.dylib
libXdmcp.6.dylib libfribidi.0.dylib liblz4.1.dylib libspeex.1.dylib libswiftMetal.dylib libvidstab.1.1.dylib
libarchive.13.dylib libgif.dylib liblzma.5.dylib libswiftAVFoundation.dylib libswiftObjectiveC.dylib libwebp.7.dylib
libass.9.dylib libglib-2.0.0.dylib libmpv.1.dylib libswiftAppKit.dylib libswiftQuartzCore.dylib libwebpmux.3.dylib
libavcodec.58.dylib libgmp.10.dylib libmujs.dylib libswiftCore.dylib libswiftSafariServices.dylib libxcb-shape.0.dylib
libavdevice.58.dylib libgnutls.30.dylib libnettle.8.dylib libswiftCoreAudio.dylib libswiftXPC.dylib libxcb-shm.0.dylib
libavfilter.7.dylib libgraphite2.3.dylib libopenjp2.7.dylib libswiftCoreData.dylib libswiftos.dylib libxcb-xfixes.0.dylib
libavformat.58.dylib libharfbuzz.0.dylib libp11-kit.0.dylib libswiftCoreFoundation.dylib libswiftsimd.dylib libxcb.1.dylib
libavutil.56.dylib libhogweed.6.dylib libpcre.1.dylib libswiftCoreGraphics.dylib libswresample.3.dylib libzimg.2.dylib
libb2.1.dylib libidn2.0.dylib libpng16.16.dylib libswiftCoreImage.dylib libswscale.5.dylib libzmq.5.dylib
libbluray.2.dylib libintl.8.dylib libpostproc.55.dylib libswiftCoreMedia.dylib libtasn1.6.dylib libzstd.1.dylib
libdav1d.5.dylib libjpeg.9.dylib librubberband.2.dylib libswiftDarwin.dylib libtesseract.5.dylib
libffi.8.dylib liblcms2.2.dylib libsnappy.1.dylib libswiftDispatch.dylib libtiff.5.dylib
low-batt@gag ~$ Many open source project use Autoconf configure scripts that provide many options as to how the source is built. In the case FFmpeg the configure script determines what features the resulting binary supports. An example from the FFmpeg configure script: The Most of the embedded code is built from sources by the IINA project specifically for embedding into IINA. Since IINA does not need FFmpeg encoding support the IINA build does not enable this feature. And of course for IINA these must be universal binaries: low-batt@gag Frameworks$ otool -L libavcodec.58.dylib | grep architecture
libavcodec.58.dylib (architecture x86_64):
libavcodec.58.dylib (architecture arm64):
low-batt@gag Frameworks$And support running under macOS 10.11+. So upgrading/building dependencies is "interesting". Testing FFmpeg & mpvI am using this ProRes video file to test with: mediainfo:low-batt@gag issue-3874$ mediainfo DJI_0023.MOV
General
Complete name : DJI_0023.MOV
Format : MPEG-4
Format profile : QuickTime
Codec ID : qt 0000.02 (qt )
File size : 8.34 GiB
Duration : 50 s 752 ms
Overall bit rate mode : Variable
Overall bit rate : 1 411 Mb/s
Encoded date : UTC 2021-11-03 14:15:51
Tagged date : UTC 2021-11-03 14:15:51
Writing application : DJIMavic3Cine
Cover : Yes
snal : (Binary)
Video
ID : 1
Format : ProRes
Format version : Version 0
Format profile : 422 HQ
Codec ID : apch
Duration : 50 s 751 ms
Bit rate mode : Variable
Bit rate : 1 409 Mb/s
Width : 5 120 pixels
Height : 2 700 pixels
Display aspect ratio : 1.896
Frame rate mode : Constant
Frame rate : 29.970 (30000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:2
Scan type : Progressive
Bits/(Pixel*Frame) : 3.401
Stream size : 8.33 GiB (100%)
Writing library : DJI0
Language : English
Encoded date : UTC 2021-11-03 14:15:51
Tagged date : UTC 2021-11-03 14:15:51
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Other #1
Type : meta
Duration : 50 s 752 ms
Bit rate mode : Variable
Default : No
Other #2
Type : dbgi
Duration : 50 s 751 ms
Bit rate mode : Variable
Default : No
low-batt@gag issue-3874$ The tests are being run on a MacBookPro18,2 with the M1 Max chip under macOS 12.4. The FFmpeg binary is 5.0.1 installed using Homebrew, so native code. Hardware decoding is disabled by default, so I'm passing low-batt@gag issue-3874$ ffmpeg -hwaccel videotoolbox -i DJI_0023.MOV -map 0:a:0? -f null -Montoring CPU usage showed hardware decoding is being used. The mpv binary I'm using for testing is the 0.34.1 build from stolendata, running under Rosetta 2 using FFmpeg 4.4.1. Hardware decoding is disabled by default, so I'm passing low-batt@gag issue-3874$ /Applications/mpv.app/Contents/MacOS/mpv --hwdec=auto --log-file=mpv.log DJI_0023.MOV And the I'm confused. From section options-hwdec of the mpv manual:
Why is mpv enforcing using of the whitelist when From section options-hwdec-codecs of the mpv manual:
Looks like this PR mpv-player/mpv#9537 that adds Trying again with low-batt@gag issue-3874$ /Applications/mpv.app/Contents/MacOS/mpv --hwdec=auto --hwdec-codecs=all --log-file=mpv.log DJI_0023.MOV And the Adding ConclusionsThe main conclusion I have is some more investigation and testing is needed. It does seem like upgrading FFmpeg would provide hardware support for decoding ProRes. Apparently IINA will also need to be changed to add |
|
Took a while, but I was able to build mpv from master with FFmpeg built from master. Playing the ProRes video low-batt@gag mpv-build (master %=)$ mpv/build/mpv --fullscreen --hwdec=auto --log-file=mpv.log ~/Documents/builds/iina/issue-3874/DJI_0023.MOV
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: stream 0, timescale not set
(+) Video --vid=1 (*) (prores 5120x2700 29.970fps)
Video --vid=2 [P] (mjpeg 1.000fps)
Using hardware decoding (videotoolbox).
VO: [libmpv] 5120x2700 videotoolbox[p210]
V: 00:00:50 / 00:00:50 (100%)
Exiting... (End of file)
low-batt@gag mpv-build (master %=)$ From This confirms that IINA should be able to get ProRes hardware decoding to work after upgrading FFmpeg. If at that point IINA is still using mpv 0.34.1 then IINA will need to set |
|
Ran a test of IINA modified to use mpv v0.35.1 and FFmpeg 5.1.2 and confirmed ProRes hardware decoding was used playing the |
|
So all IINA needs to do is to update the dependencies and hw for ProRes will be enabled? |
|
That was a note to other developers that no changes to IINA will be needed. I thought that important to note as In a post above I mentioned IINA needing to change the The above test was very rough. I will be testing again once IINA has officially upgraded dependencies. |
|
Thank you :) |
|
Just ran a test with the upgraded dependencies and: ProRes Hardware Acceleration will work in the next release of IINA. |
|
This is fixed in the released version of IINA. |
Is there a plan to enable IINA to support ProRes hardware acceleration on M1 Pro/Max and M2 devices which have hardware ProRes video encode/decode engines?
VideoToolbox framework has supported this natively for a while now, via Quicktime Player.
The text was updated successfully, but these errors were encountered: