No idea what caused your error, but I found this, which looks similar:
http://ffmpeg.org/pipermail/ffmpeg-user/2015-September/028415.html
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91298

Can you build FFmpeg on its own, with --enable-lto or will this fail with the same error?

I tried building ffmpeg by replicating this guide, and it seems that I have the exact same error there.

I copied --enable-lto --enable-shared --ar=gcc-ar --ranlib=true from the ffmpeg mailing list and it fixes the build.
Without investigating, I don't know how to apply these flags to the handbrake build.

Below some more information on my build setup. Note that I am no expert in building C/C++ projects.

Handbrake --lto=on build

• gcc:12.2.0-bullseye Dockerfile
• build-handbrake.logs.zip

ffmpeg --enable-lto build

• gcc:12.2.0-bullseye Dockerfile
• ffmpeg-build.logs.zip

ffmpeg --enable-lto --enable-shared --ar=gcc-ar --ranlib=true build

• gcc:12.2.0-bullseye Dockerfile
• ffmpeg-build-2.logs.zip

I confirmed locally that only --enable-shared fixes my standalone ffmpeg build.

ffmpeg --enable-lto --enable-shared build

• gcc:12.2.0-bullseye Dockerfile
• ffmpeg-build-5.logs.zip

This seems more like a ffmpeg bug to me then. It would be good to report this to ffmpeg, so they can fix this.

As a workaround you can try to edit the file HandBrake/contrib/ffmpeg/module.defs
and replace

ifeq (on,$(GCC.lto))
    FFMPEG.CONFIGURE.extra += --enable-lto
endif

with

ifeq (on,$(GCC.lto))
    FFMPEG.CONFIGURE.extra += --enable-lto --enable-shared
endif

This should fix your issue, until there is a proper fix for this.

Thank you for your help!

• I will try to create a working Handbrake build with your suggestions and actually test it in the coming days
• This topic is completely over my head, so if anyone else can actually replicate my issue and is interested in bringing it up with upstream ffmpeg, please do! Maybe I'll get into it at some point, but no promises.

Meanwhile I also got a successful build of ffmpeg with --enable-lto --enable-pic

I have absolutely no clue what I am doing or what all this means. I just tried it out, because something like that was mentioned in the same ffmpeg mail and I also found mentions of it from skimming this Clear Linux forum post and this gentooLTO GitHub issue from 2019.

Thanks for your test. I also found the --enable-pic information here.
http://ffmpeg.org/pipermail/ffmpeg-user/2017-August/036977.html

Could you then try, if

ifeq (on,$(GCC.lto))
    FFMPEG.CONFIGURE.extra += --enable-lto --enable-pic
endif

in the file HandBrake/contrib/ffmpeg/module.defs works for you? If yes, I could make a patch for this to at least make it possible to build HandBrake. And, if it builds with the above, please also test, that the resulting HandBrake build works as expected. This would be very helpful.