"RealTime" VT property + "limit bitrate" setting cause heavy framedrops on ARM macs with HW VideoToolbox encoder

[AuroraWright]

Operating System Info

macOS 12.0

Other OS

No response

OBS Studio Version

Git

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/mCaYXIhd7YJsEBtj

OBS Studio Crash Log URL

No response

Expected Behavior

Frames should not be dropped when using the hardware VideoToolbox encoder and "limit bitrate" turned on in preferences.

Current Behavior

Heavy frame drops occur no matter the target bitrate limit when the "limit bitrate" checkbox is toggled. After some tests I found out this is caused by the kVTCompressionPropertyKey_RealTime property, which is currently always set by OBS here:

obs-studio/plugins/mac-vth264/encoder.c

Line 325 in 2c5b1a8

code = session_set_prop(s, kVTCompressionPropertyKey_RealTime,

. No drops at all occur (even at really high bitrates) if that property is not set. A possible solution could be to make that property user-selectable (if you like the idea I might make a PR as soon as I have some time to check out how Qt UI stuff works).

Steps to Reproduce

1. Select the VideoToolbox hardware encoder on macOS
2. Check "limit bitrate" and enter any bitrate you like
3. Check output
   ...

Anything else we should know?

No response

[RytoEX]

A review of Apple's documentation for kVTCompressionPropertyKey_RealTime shows:

A Boolean value indicating whether it's recommended that the video encoder perform compression in real time.

For offline compression, clients may set this property to kCFBooleanFalse, which indicates that it is OK for the video encoder to work slower than real time in order to produce a better result.

For real-time compression, clients may set this property to kCFBooleanTrue to recommend that encoding stay timely.

OBS is not doing "offline compression", so frames must be encoded within the frame rate requirements. I wonder if disabling this just starts filling a buffer of backlogged frames or something and then writes them eventually. That, or Apple's documentation is incomplete or incorrect.

Tagging in @gxalpha and @PatTheMav as they are far more qualified to answer this and ask exploratory questions than I am.

[HealsCodes]

Since @AuroraWright tagged me regarding this in the linked issue - I made the realtime flag configurable in my local sources and tested the exact same stream with each setting. Subjectively disabling realtime compression made things feel "smoother" to a minimal degree. I did not see a difference in frame drops (both none in my case, native build on M1 8 core).

For a 6000kbit (bitrate and limit) stream 1080p@60 of 2min length the only noticeable difference is in a minimal increase in max and min bitrate:

kVTCompressionPropertyKey_RealTime = kCFBooleanTrue

Bitrate Average: 5,157 kbps Bitrate Max: 5,655 kbps Bitrate Min: 3,613 kbps

kVTCompressionPropertyKey_RealTime = kCFBooleanFalse

Bitrate Average: 5,150 kbps Bitrate Max: 5,699 kbps Bitrate Min: 4,027 kbps

[rcombs]

This was my recommendation; my understanding is that the apple docs are either incorrect or incomplete. We had to explicitly set realtime=0 by default in ffmpeg; we speculate this to be a power-saving measure, but it's not really clear. As far as we're aware, this doesn't result in any perf issues on Intel machines either. See FFmpeg/FFmpeg@4778ab2

[RytoEX]

We've reached out to our Apple contacts for clarification.