Enable proprietary codecs with OS decoding support only

[magreenblatt]

Is your feature request related to a problem? Please describe.
By default, CEF/Chromium builds have proprietary codecs (H.264/AAC) disabled [1]. This is due to patent portfolio licensing requirements when using the FFmpeg library for software decoding of proprietary codecs (via the ffmpeg_branding=Chrome part). If you enable and distribute (via FFmpeg configuration) a software implementation of the patent protected technology as part of your application then you will likely be subject to said licensing requirements and associated costs.

[1] Proprietary codecs are marked as "Google Chrome only" at https://www.chromium.org/audio-video/ and can generally be enabled by building CEF/Chromium with the proprietary_codecs=true ffmpeg_branding=Chrome GN arguments.

Describe the solution you'd like
On Windows and MacOS, a (preferred) hardware video decoding path using OS platform APIs already exists in Chromium, and this implementation does not depend on configuring or distributing FFmpeg with software decoding of proprietary codecs enabled (leave out the ffmpeg_branding=Chrome part [2]). The OS manufacturers (Microsoft and Apple respectively) allow applications to use these OS platform APIs with no added licensing requirements or costs. We can consequently enable hardware-only support for proprietary video codecs (via OS platform APIs) with only minor changes to the existing CEF/Chromium build configuration.

[2] A small patch to disable the assert in media/BUILD.gn is also required.

Additional context
The problem with this approach is that devices lacking hardware decoding support will not be able to play proprietary codecs (due to the lack of software decoding support via FFmpeg). This is most likely to impact Windows users with old or blocklisted GPU hardware. This hardware-only configuration may still make sense for software developers/distributors who are minimally impacted by this population of older Windows hardware or who are otherwise willing to accept this licensing vs functionality trade-off.

8/28/23 SUMMARY OF TEST RESULTS (M117)

• Chromium supports hardware h264 video decoding via OS APIs with minimal changes on Win/Mac.
• Chromium does not (directly) support AAC audio decoding.
  • FFmpeg on Mac should theoretically support AAC decoding via AudioToolbox, but we were unable to get it working.
  • FFmpeg on Windows does not support decoding via Media Foundation (only encoding).
• Chromium with proprietary_codecs=true includes additional (non-FFmpeg) proprietary code in //media that may potentially need to be replaced by OS APIs.

See related discussion below.

[magreenblatt]

Sample MP4 files are available in many places (for example, here). With proprietary codecs support disabled these files will not play.

Hardware decoding support for proprietary video codecs can be disabled by adding the --disable-accelerated-video-decode command-line flag or (when using the Chrome Runtime) via chrome://flags/#disable-accelerated-video-decode. This can be used to verify that resulting builds do not play proprietary codecs when hardware decoding is disabled.

[magreenblatt]

Patch for media/BUILD.gn to allow builds with just GN arg proprietary_codecs=true (tested at current master/M117):

diff --git media/BUILD.gn media/BUILD.gn
index 8ebd7c57579ab..ced67056b1a22 100644
--- media/BUILD.gn
+++ media/BUILD.gn
@@ -9,6 +9,7 @@ import("//build/config/chromecast_build.gni")
 import("//build/config/features.gni")
 import("//build/config/linux/pkg_config.gni")
 import("//build/config/ui.gni")
+import("//cef/libcef/features/features.gni")
 import("//media/media_options.gni")
 import("//testing/libfuzzer/fuzzer_test.gni")
 import("//testing/test.gni")
@@ -74,7 +75,7 @@ buildflag_header("media_buildflags") {
   }
 }
 
-if (proprietary_codecs && media_use_ffmpeg) {
+if (proprietary_codecs && media_use_ffmpeg && !enable_cef) {
   assert(
       ffmpeg_branding != "Chromium",
       "proprietary codecs and ffmpeg_branding set to Chromium are incompatible")

[magreenblatt]

Testing this configuration on MacOS 13.5.1 and Windows 10 (video link), h264 video works but it appears that aac audio is unsupported.

[0825/101954.039553:INFO:ffmpeg_demuxer.cc(238)] FFmpegDemuxer: created video stream, config codec: h264, profile: h264 high, level: not available, alpha_mode: is_opaque, coded size: [1920,1080], visible rect: [0,0,1920,1080], natural size: [1920,1080], has extra data: true, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}
[0825/101954.039958:INFO:ffmpeg_demuxer.cc(214)] Warning, FFmpegDemuxer failed to create a valid/supported audio decoder configuration from muxed stream, config:codec: aac, profile: unknown, bytes_per_channel: 0, channel_layout: STEREO, channels: 2, samples_per_second: 44100, sample_format: Unknown sample format, bytes_per_frame: 0, seek_preroll: 0us, codec_delay: 0, has extra data: true, encryption scheme: Unencrypted, discard decoder delay: true, target_output_channel_layout: NONE, target_output_sample_format: Unknown sample format, has aac extra data: true
[0825/101954.040059:INFO:ffmpeg_demuxer.cc(1383)] FFmpegDemuxer: skipping invalid or unsupported audio track
[0825/101954.041200:INFO:decoder_selector.cc(315)] Cannot select DecryptingVideoDecoder for video decoding
[0825/101954.090807:INFO:web_media_player_impl.cc(1077)] Effective playback rate changed from 0 to 1
[0825/101954.107040:INFO:decoder_stream.cc(443)] Selected VDAVideoDecoder for video decoding, config: codec: h264, profile: h264 high, level: not available, alpha_mode: is_opaque, coded size: [1920,1080], visible rect: [0,0,1920,1080], natural size: [1920,1080], has extra data: true, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}

MP4 + Widevine (example) also fails due to lack of audio support.

Shaka Error MEDIA.VIDEO_ERROR (4,,DECODER_ERROR_NOT_SUPPORTED: audio decoder initialization failed)

[magreenblatt]

It should be possible to support AAC using OS decoding APIs (AudioToolbox on MacOS and Media Foundation on Windows). From the related conversation here:

FFmpeg on Mac includes own implementation of AAC decoder using AudioToolbox and that can be made to work with Chromium.

That might be the FFmpeg --enable-audiotoolbox configure option. Looks like current FFmpeg flag is --disable-audiotoolbox (related conversation here), however this feature is auto-detected and appears to be disabled in the Chromium config.

FFmpeg support for AAC audio (and other format) decoding via Media Foundation is also available (add --enable-mf to configure, related conversation here). Looks like the current FFmpeg flag is --enable-mediafoundation, and this only supports AAC/h264 encoding (not decoding).

Enabling this functionality via the FFmpeg configuration should therefore be possible as follows:

1. Add the additional configure command-line flags in third_party/ffmpeg/chromium/scripts/build_ffmpeg.py
2. Follow the instructions here for configuring FFmpeg in Chromium (step 1b: Generate new FFmpeg build files). Note that all platforms must be built/configured on a Linux host (Win/Mac host is not supported).

On MacOS, we can attempt to manually enable AudioToolbox support.

Note that additional Chromium changes may still be required.

[magreenblatt]

FFmpeg support for AAC audio (and other format) decoding via Media Foundation is also available (add --enable-mf to configure, related conversation here).

This option also provides an h264 decoder using Media Foundation. It would be worth testing if this enables (or could enable) support for software decoding on Windows. Looks like the current FFmpeg flag is --enable-mediafoundation, and this only supports AAC/h264 encoding (not decoding).

[magreenblatt]

I believe the following patch to third_party/ffmpeg should enable AudioToolbox decoding via FFmpeg on Mac (x64). However, it appears that additional changes are required since decoding still fails with the following warning:

[0828/131743.997089:INFO:ffmpeg_demuxer.cc(214)] Warning, FFmpegDemuxer failed to create a valid/supported audio decoder configuration from muxed stream, config:codec: aac, profile: unknown, bytes_per_channel: 0, channel_layout: STEREO, channels: 2, samples_per_second: 44100, sample_format: Unknown sample format, bytes_per_frame: 0, seek_preroll: 0us, codec_delay: 0, has extra data: true, encryption scheme: Unencrypted, discard decoder delay: true, target_output_channel_layout: NONE, target_output_sample_format: Unknown sample format, has aac extra data: true
[0828/131743.997176:INFO:ffmpeg_demuxer.cc(1383)] FFmpegDemuxer: skipping invalid or unsupported audio track

diff --git BUILD.gn BUILD.gn
index 91e2f508c3..c0ef47b261 100644
--- BUILD.gn
+++ BUILD.gn
@@ -370,6 +370,12 @@ target(link_target_type, "ffmpeg_internal") {
       libs -= [
         "rt",
       ]
+    } else {
+      frameworks = [
+        "AudioToolbox.framework",
+        "CoreFoundation.framework",
+        "CoreMedia.framework",
+      ]
     }
   } else if (is_win) {
     defines += [
diff --git chromium/config/Chromium/mac/x64/config.asm chromium/config/Chromium/mac/x64/config.asm
index a853fb1031..ccb46d7237 100644
--- chromium/config/Chromium/mac/x64/config.asm
+++ chromium/config/Chromium/mac/x64/config.asm
@@ -571,7 +571,7 @@
 %define CONFIG_OMX 0
 %define CONFIG_OPENCL 0
 %define CONFIG_AMF 0
-%define CONFIG_AUDIOTOOLBOX 0
+%define CONFIG_AUDIOTOOLBOX 1
 %define CONFIG_CRYSTALHD 0
 %define CONFIG_CUDA 0
 %define CONFIG_CUDA_LLVM 0
diff --git chromium/config/Chromium/mac/x64/config.h chromium/config/Chromium/mac/x64/config.h
index c727314c52..e2b26f7f1e 100644
--- chromium/config/Chromium/mac/x64/config.h
+++ chromium/config/Chromium/mac/x64/config.h
@@ -587,7 +587,7 @@
 #define CONFIG_OMX 0
 #define CONFIG_OPENCL 0
 #define CONFIG_AMF 0
-#define CONFIG_AUDIOTOOLBOX 0
+#define CONFIG_AUDIOTOOLBOX 1
 #define CONFIG_CRYSTALHD 0
 #define CONFIG_CUDA 0
 #define CONFIG_CUDA_LLVM 0
diff --git chromium/config/Chromium/mac/x64/libavcodec/codec_list.c chromium/config/Chromium/mac/x64/libavcodec/codec_list.c
index 653357d362..5445085782 100644
--- chromium/config/Chromium/mac/x64/libavcodec/codec_list.c
+++ chromium/config/Chromium/mac/x64/libavcodec/codec_list.c
@@ -1,4 +1,5 @@
 static const FFCodec * const codec_list[] = {
+    &ff_aac_at_decoder,
     &ff_theora_decoder,
     &ff_vp3_decoder,
     &ff_vp8_decoder,
diff --git ffmpeg_generated.gni ffmpeg_generated.gni
index 375489c10e..3cc44be44e 100644
--- ffmpeg_generated.gni
+++ ffmpeg_generated.gni
@@ -14,6 +14,10 @@ ffmpeg_asm_sources = []
 
 use_linux_config = is_linux || is_chromeos || is_fuchsia
 
+if (is_apple && !is_ios) {
+  ffmpeg_c_sources += [ "libavcodec/audiotoolboxdec.c" ]
+}
+
 if ((is_android && current_cpu == "arm" && arm_use_neon) || (is_android && current_cpu == "arm64") || (is_android && current_cpu == "x64") || (is_android && current_cpu == "x86") || (is_apple) || (is_win) || (use_linux_config)) {
   ffmpeg_c_sources += [
     "libavcodec/ac3_channel_layout_tab.c",

[magreenblatt]

Looks like the AAC configuration is failing in AVSampleFormatToSampleFormat called via FFmpegDemuxerStream:Create, AVStreamToAudioDecoderConfig and AVCodecContextToAudioDecoderConfig.

[0828/135601.391068:VERBOSE1:ffmpeg_common.cc(301)] Unknown AVSampleFormat: -1

[magreenblatt]

It appears that Chromium (and not just FFmpeg) includes potentially problematic code (here) when setting proprietary_codecs=true. For example, this code is used for parsing the AAC stream data to detect the profile type. That code would likely also need to be replaced by calling OS API functions.

[TechInterMezzo]

@magreenblatt
Did you have a look at the Vivaldi source code?
https://vivaldi.com/de/source/

It contains a folder named platform_media that seems to be solely directed at solving the problem of using H.264 and AAC under Windows and Mac by using the OS API.

Hi,
i'm also interested on this topic, can this be a solution to use the built in codecs from lets say Windows?

--disable-accelerated-video-decode

i tried this from commandline with:
cefclient.exe --disable-accelerated-video-decode
cef is build with:
proprietary_codecs=true ffmpeg_branding=Chrome
but the videos are still playing, did i miss something or do wrong?
thanks