No Hardware Decoding on MacOS

[dnywlsh]

I'm using an Intel Mac (Ice Lake chipset), and according to Intel, my GPU supports hardware decoding of MPEG-2, VC-1, and VP9 using Quick Sync (VideoToolbox on MacOS).

Playing back H.264 and HEVC shows that VideoToolbox (hardware decoding) is being used, but playing back MPEG-2, VC-1, and VP9 says it's not using hardware decoding, and CPU usage is significantly higher, and video playback drops frames.

Any idea why hardware decoding is only working for H.264 and HEVC, but nothing else?

[low-batt]

Like a lot of open source projects IINA is layered on top of other projects. For audio/video playback IINA uses a library from the mpv project. The Hardware decoder setting (preference) in IINA controls how IINA sets the mpv option hwdec. One of the significant differences between IINA and mpv is that IINA defaults to enabling hardware decoding whereas mpv defaults to software decoding. The mpv manual explains:

Hardware decoding is not enabled by default, to keep the out-of-the-box configuration as reliable as possible. However, when using modern hardware, hardware video decoding should work correctly, offering reduced CPU usage, and possibly lower power consumption.

The manual used to say:

Hardware decoding is not enabled by default, because it is typically an additional source of errors. It is worth using only if your CPU is too slow to decode a specific video.

Things are getting better, but there is a history of problems with hardware decoding.

The layers involved are:

• IINA
• mpv
• FFmpeg
• Video Toolbox (macOS version dependent)
• Hardware media engine (Mac model dependent)

It is not just the codec that matters. Details about exactly how the video is encoded can trip up hardware decoding. When investigating hardware decoding issues lots of details matter.

For this post I'm focusing just on VP9. I downloaded the YouTube video VP9 Codec Test. Playing that video with IINA 1.3.1 (which includes mpv 0.34.1 and FFmpeg 4.4.2 libraries) on my MacBookPro18,2 with the M1 chip under macOS Ventura 13.2. The mpv log file shows IINA was unable to use hardware decoding:

[   0.208][v][vd] Codec list:
[   0.208][v][vd]     vp9 - Google VP9
[   0.208][v][vd] Opening decoder vp9
[   0.208][v][vd] No hardware decoding available for this codec.
[   0.208][v][vd] Using software decoding.
[   0.208][v][vd] Detected 10 logical cores.
[   0.208][v][vd] Requesting 11 threads for decoding.
[   0.208][v][vd] Selected codec: vp9 (Google VP9)

For this next test I modified IINA 1.3.1 to use Homebrew installed mpv 0.35.0 and FFmpeg 5.1.2. The mpv log shows hardware decoding worked:

[   2.456][v][vd] Codec list:
[   2.457][v][vd]     vp9 - Google VP9
[   2.457][v][vd]     libvpx-vp9 (vp9) - libvpx VP9
[   2.457][v][vd] Opening decoder vp9
[   2.457][v][vd] Looking at hwdec vp9-videotoolbox...
[   2.457][v][vd] Trying hardware decoding via vp9-videotoolbox.
[   2.457][v][vd] Selected codec: vp9 (Google VP9)

At the time IINA 1.3.1 was built FFmpeg 5 had just been released. IINA stayed with version 4 due to concerns over regressions in the area of hardware decoding. Of note is FFmpeg ticket 9599 "VideoToolbox VP9 hwaccel freezes ffmpeg". A bit worrisome that ticket is still open. Hopefully the regressions have been fixed in in FFmpeg 5.1.2. My test above was on Apple silicon. Need to also test some Intel processors.

I do see a IINA problem that needs to be fixed. FFmpeg Compilation Guide under "Additional libraries" says:

libvpx is a VP8 and VP9 encoder. Use --enable-libvpx.

I do not see the --enable-libvpx build flag being passed to configure in ffmpeg-iina.rb, the script used to build FFmpeg libraries for inclusion in IINA. That script needs to be updated to build FFmpeg with support for VP9 hardware decoding.

I'll try and find some time to look into the situation with the other codecs.

[low-batt]

Just ran a test with the upgraded dependencies and:

[  13.667][v][vd] Looking at hwdec vp9-videotoolbox...
[  13.667][v][vd] Trying hardware decoding via vp9-videotoolbox.
[  13.667][v][vd] Selected codec: vp9 (Google VP9)

VP9 hardware decoding will work in the next release of IINA.

I will comment on the other formats when I get a chance.

[low-batt]

I'm leaving this one open for now as there is a known problem with VP9 and hardware decoding on Intel Macs and some of the other encodings are not working as well. The VP9 problem has been reported to FFmpeg.