LLD link command taking longer than usual unless you disable CFI

[Sharjeel-Khan]

In Android, we have been noticing a few builds taking longer than usual (18-20 mins) at linking steps. When we disabled CFI, those same builds would take few seconds. I am no sure why CFI is causing the big difference.

I have attached reproducer for a specific test in AOSP to showcase the difference for both the CFI (repro-cfi-*.tar) and non-CFI (repro-no-cfi-*.tar) versions. You can just run ld.lld @response.txt after running tar -xf <tar-file>. There is both a 64-bit version and 32-bit version of the test. In addition, I attached the tests' bitcodes when we build with CFI because it might be due to other passes since Android has LTO on by default. Here is the link: https://drive.google.com/drive/folders/1i2tLBUJAnoGJkqtuJpGA9PUwmv1856DO?usp=sharing

[Sharjeel-Khan]

Hey @pcc, do you have any idea why this could be happening?

[pcc]

It looks like the bitcode files in at least repro-cfi-64.tar are full LTO bitcode files, so the linker uses full LTO instead of ThinLTO, which will lead to a much slower link time. Are you building them with -flto=thin?

[Sharjeel-Khan]

Yeah, these were built using Soong and Soong is supposed to use -flto=thin as seen here: https://android.googlesource.com/platform/build/soong/+/refs/heads/main/cc/lto.go#109.

[pcc]

Could you extract the compile command for one of the files (e.g.

ninja -f "out/${TARGET_PRODUCT}.ninja" -t commands -s out/soong/.intermediates/packages/modules/Bluetooth/system/test/suite/net_test_bluetooth/android_x86_64_cfi_cov/obj/packages/modules/Bluetooth/system/test/mock/mock_udrv_ulinux_uipc.o

) and paste it here?

[pirama-arumuga-nainar]

Note that CFI still uses full LTO: https://android.googlesource.com/platform/build/soong/+/refs/heads/main/cc/sanitize.go#67. b/243476017 is the google-internal bug tracking the use of thin LTO for CFI.

[pcc]

Looking at the case in more detail, I discovered that this was a regression from #137081 that only affects certain builds such as coverage collecting builds (which explains why I didn't catch it in my testing). Let me send a fix.