Add initial support for the OpenH264 H.264 software codec

[Conan-Kudo]

Description

This allows users to leverage the OpenH264 codec from Cisco to encode H.264 video content. It is significantly reduced in capability from alternatives, but it does the job.

This also provides a framework for adding support for other H.264 software codecs provided through FFmpeg.

Motivation and Context

A number of distributions use OpenH264 distributed by Cisco to provide a fully licensed H.264 implementation (in particular, both Fedora Linux and openSUSE Linux do). However, OBS Studio cannot currently use it.

This change allows OBS Studio to use OpenH264 through FFmpeg if it's available.

How Has This Been Tested?

• Tested recording a screencast using OpenH264 + AAC with both Simple and Advanced settings modes
• Tested streaming a screencast using OpenH264 + AAC with both Simple and Advanced settings modes

This was tested on Fedora Linux 38 for x86_64 with ffmpeg 6.0.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[RytoEX]

Does this require #8015?

[Conan-Kudo]

Does this require #8015?

Not yet. This is only the wiring for the ffmpeg plugin, which I think means it only shows up in the "advanced view" right now? I literally just banged this out and am taking a bit of a break before continuing to iterate on it. I've got local builds going so I can test the functionality...

[Conan-Kudo]

@RytoEX That said, having #8015 land first would make it tremendously easier for the UI wiring I need to do...

[Conan-Kudo]

I've pulled in #8015 into this pull request to simplify things for me and to show the full context of this change.

[Conan-Kudo]

I've now tested that this works, thanks to @Gawdl3y's help! 🎉

[Lain-B]

Hey @Conan-Kudo, remind me to come back to this after we make our v30 release branch. Sorry about the delay

[PatTheMav]

I agree with @tytan652 that the PR has its selection logic backwards: It installs OpenH264 as a fallback for x264, but x264 is a core module and the former is a very distro-specific option.

As such the PR needs to add OpenH264 as an optional codec and check for its availability and still fall back to x264 as the default.

[Conan-Kudo]

As such the PR needs to add OpenH264 as an optional codec and check for its availability and still fall back to x264 as the default.

I added it this way because OBS crashes the other way because there will not be x264 in the first place in Fedora's build unless you install the plugin from RPM Fusion.

[Conan-Kudo]

And ffmpeg is a "core module" too. Both are always present in upstream OBS builds, it depends on detection logic and whether ffmpeg is compiled with OpenH264 support.

[PatTheMav]

x264 is the built-in fallback and the app is structured around that.

The better way to solve this would be for Fedora to patch obs-x264 to use OpenH264 so that the program is no wiser.

(Also I’m unsure as to whether that should be implemented in obs-x264 itself with a specific check for Fedora, because I’m not happy about distro-specific workarounds while we have to maintain support for 3 operating systems already.)

Afaik we officially support Ubuntu and Flatpak and any distro-specific changes need to be made by distributors while packaging the app.

[Conan-Kudo]

obs-x264 is libx264 specific, an equivalent for OpenH264 would be to write a whole new plugin for libopenh264. I used ffmpeg because it already has an interface for this, we prefer OpenH264 to be accessed through ffmpeg anyway, and it would relatively straightforward to wire in, based on how AV1 was done.

[PatTheMav]

Again, that will not work because you are creating a fallback for a fallback, you are not even checking if ffmpeg supports the encoder at runtime before making that decision and with that you also make macOS and Windows fall back to OpenH264 if x264 is not found.

It creates additional code complexity and maintenance burden for an edge case on a single distribution and IMO that’s not a good deal.

[Conan-Kudo]

It creates additional code complexity and maintenance burden for an edge case on a single distribution and IMO that’s not a good deal.

It's not just a single distribution. At this point in time, it benefits four distributions:

• CentOS/RHEL
• Fedora Linux
• openSUSE Tumbleweed
• openSUSE Leap/SLE

This is also supported by the FFmpeg library included in the Freedesktop runtime for Flatpak too.

[PatTheMav]

We can add generic support for OpenH264 if it is available (as in "checked if available via FFmpeg" because we do not ship FFmpeg with OpenH264 support on Windows and macOS as we do not support that codec at all), and that availability will probably be limited to Linux only and depends on the distribution-specific variant of FFmpeg available. If that check is positive, it can be added to the list of available encoders.

But distribution-specific fixes need to be implemented/patched by distributors. Adding a global (as in: affecting all platforms) fallback on OpenH264 if the actual fallback x264 (which is commonly shipped by OBS Studio itself or expected to be available on the platform) is not a solution IMO.

Upstream distributors can then handle the added complexity of adding an OpenH264 fallback if it is available at all and abort the OBS Studio launch if it cannot be found.

Also AV1 is not a "baseline/fallback" encoder, so the comparison with its implementation is not applicable.

[derrod]

It would probably be sensible to split the encoder addition and the UI changes into separate PRs so we can move things along. The former is probably fine (with some minor code review nits).

[Conan-Kudo]

I've split out the first half of this pull request as #10135, per @derrod's request.