build: fix --lto=on build on GCC Linux

[lokok8]

fixes #4699 ffmpeg build failure with --lto=on on GCC Linux, by using ffmpeg's --enable-pic configure flag.

snippet of build failure logs without this PR

: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h: Assembler messages:
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:202: Error: junk `(%rip)+512' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:212: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:214: Error: junk `(%rip)+1024+128' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:230: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:202: Error: junk `(%rip)+512' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:212: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:214: Error: junk `(%rip)+1024+128' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:230: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:202: Error: junk `(%rip)+512' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:212: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:214: Error: junk `(%rip)+1024+128' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:230: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:202: Error: junk `(%rip)+512' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:212: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:214: Error: junk `(%rip)+1024+128' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:230: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:202: Error: junk `(%rip)+512' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:212: Error: junk `(%rip)+0' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:214: Error: junk `(%rip)+1024+128' after expression
: contrib/ffmpeg/ffmpeg-5.1.2/libavcodec/x86/cabac.h:230: Error: junk `(%rip)+0' after expression
: lto-wrapper: fatal error: /usr/local/bin/g++ returned 1 exit status
: compilation terminated.
: /usr/bin/ld: error: lto-wrapper failed
: collect2: error: ld returned 1 exit status
: gmake: *** [../test/module.rules:49: HandBrakeCLI] Error 1

Discovered when I tried to build HandbrakeCLI with Docker in debian:11.5-slim using the packaged GCC 10.2.1-6. I also confirmed the same error on GCC 12.2.0 and GCC 11.3.0

---

Test on:

• Windows 10+ (via MinGW)
• macOS 10.13+
• Ubuntu Linux

[lokok8]

Note that I do not know why --enable-pic fixes the build failure for me and that my knowledge in C/C++ builds is very limited.

This fix was found mostly by trial and error of seemingly relevant google findings, as seen in the linked issue. Listing some possibly related ones:

• 2015: ffmpeg mailing list entry. Similar error message with ffmpeg LTO build. --enable-shared found as possible solution. Also mentions -fPIC -fno-PIE which I dont understand.
• 2017: ffmpeg mailing list entry. Discussing -ffat-lto-objects and --enable-pic for some other LTO related build failure in static ffmpeg(?). Mentions concern of influence on performance.
• 2019: Clear Linux Forum Post. "FFMPEG does build nicely with the link-time optimizer, but putting -flto in the flags or configuring with --enable-lto tends to cause the build to fail with lots of undefined symbols. Instead, put -ffat-lto-objects in the flags"
• 2019: InBetweenNames/gentooLTO GH-issue from 2019. Discussing PIC and -ffat-lto-objects in context of LTO ffmpeg builds.

With my testing I can confirm that either --enable-shared or --enable-pic fixes the build for me.

[lokok8]

Until now I mainly tested building HandbrakeCLI, but not actually using it. I still need to see if everything actually works and whether I can notice any negative performance impact. If you have any recommendations on what exactly I should test, please let me know.

FYI: For now I plan to focus on testing with my setup, where I build and execute HandbrakeCLI inside a debian:11.5-slim docker image. You can find the Dockerfile here, if its relevant or interesting.

[Nomis101]

Your title does say, you want to fix this for Linux, but your patch does apply it for all architectures. At least it does build also on macOS with pic enabled, don't know for windows. But if you want to fix this for Linux only, you would need to ifdef the flag.

[lokok8]

Thanks @Nomis101,
I force pushed a change. Hopefully thats how the ifdef is supposed to be done.

---

Today I wanted to thoroughly test the changes, but I discovered something else instead.

• https://hub.docker.com/_/gcc (debian based) images have this issue with latest GCC 10, 11, 12
• https://hub.docker.com/_/debian images do not have this issue
• a clean debian VM does not have this issue

This PR does fix the issue in the GCC images, but it is not necessary in the other two scenarios. Currently I am not too motivated to find out what exactly is going wrong there, so I am considering to just drop this PR.

Whats your opinions?

[Nomis101]

I don't have Linux, so I can't really test. I can say, that it builds on Mac with this patch. If it builds on Linux with this patch, I think this is fine then. Its similar to this one

HandBrake/contrib/libvpx/module.defs

Line 37 in 3d99f18

ifeq (linux,$(HOST.system))

So, the logic should also be fine.

[bradleysepos]

I've been a bit concerned that enabling PIC might be a band-aid that just happens to work around a bigger issue. Thankfully, I was able to get some input from an FFmpeg developer. I'll attempt to sum up the findings here.

It seems there's an issue with inline assembly in libavcodec/x86/cabac.h expanding to invalid instructions when GCC's LTO codegen is in play. Basically, the code generation defies certain assumptions made by the inline assembly, which is guarded by compiler version checks that aren't helpful in this case. Setting the BROKEN_RELOCATIONS flag solves the problem, resulting in proper instructions, but without a compiler define to know we're in LTO mode, the decision to set BROKEN_RELOCATIONS cannot be made when and where it needs to be made.

For whatever reason, enabling PIC causes the LTO codegen to do things differently enough that the conflict with the assumptions made by the inline assembly is removed. So, it's a semi-practical workaround, but not one that really gets at the root issue, and perhaps even a bit like driving a small nail with a sledgehammer.

It's a tough call, but I think this is too broad. Given the issue seems to only affect certain builds/configurations of GCC and not others, I'm afraid I'm not too comfortable enabling PIC across the board. We should instead treat this as a known issue and try to assist getting a proper fix into FFmpeg, which we can incorporate in the future. PIC can be enabled by CFLAGS if someone experimenting with LTO is so inclined.

[Nomis101]

Thanks for this insight. Based on your explanation I also now think this might be a semi-practical workaround. Maybe adding a line to the documentation "Building HandBrake for Linux" about enabling PIC by CFLAGS would then be a good idea?