ld.lld: error: Entry trampoline text too big (arm64)

[arndb]

I see this error in one randconfig with the latest lld:

$ ld.lld --reproduce 0x93895626.tar -EL -maarch64elf -z norelro --no-undefined -X --fix-cortex-a53-843419 --gc-sections --build-id=sha1 --orphan-handling=warn --strip-debug -o .tmp_vmlinux.kallsyms1 -T ./arch/arm64/kernel/vmlinux.lds --whole-archive arch/arm64/kernel/head.o init/built-in.a usr/built-in.a arch/arm64/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a arch/arm64/lib/built-in.a lib/built-in.a arch/arm64/lib/lib.a lib/lib.a drivers/built-in.a sound/built-in.a samples/built-in.a virt/built-in.a --no-whole-archive --start-group ./drivers/firmware/efi/libstub/lib.a --end-group
ld.lld: error: Entry trampoline text too big
ld.lld: error: Entry trampoline text too big
ld.lld: error: Entry trampoline text too big

I have not tried to figure out what is going on, but this is a tarball for reproduction

0x93895626.tar.xz.gz

[arndb]

I just noticed that this uses --gc-sections, which I only allowed being used on arm64 yesterday. Also checked this happens with all versions I have installed (10/11/12/13).

[MaskRay]

Saw the patch [PATCH] arm64: vmlinux.lds.S: keep .entry.tramp.text section. It is the appropriate solution, not a workaround.

As I said

However, I worry that many other KEEP keywords in vmlinux.lds are unnecessary:
https://lore.kernel.org/linux-arm-kernel/20210226211323.arkvjnr4hifxapqu@google.com/

git log -S KEEP -- include/asm-generic/vmlinux.lds.h => there is quite a
bit unjustified usage. Sure, adding KEEP (GC root) is easy and
works around problems, but it not good for CONFIG_LD_DEAD_CODE_DATA_ELIMINATION.

It'd be nice if sections (which are required to be GC roots) are annotated at .section directive time, not in a linker script at KEEP time.

SHF_GNU_RETAIN is designed for this use case. It requires new assembler (GNU as>=2.36 or LLVM integrated assembler>=13.0 https://reviews.llvm.org/D95730). It can be used with very old GNU ld because GNU ld ignores unrecognized section flags (ELF spirit).

[arndb]

Thanks for taking a closer look. I still don't see why we need to treat these as a GC root when there are no references to the section from other code, but I trust that you found out how this is meant to work.

I'm sure you are right about the other KEEP() flags being overly conservative, as CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is only used on two minor architectures at the moment, and probably not a lot of time went into fine-tuning these.

We definitely cannot require binutils-2.36, I would say the minimum version would have to be at least as old as the oldest gcc we support (currently 4.9.x) , and the current minimum of 2.23 isn't that much older. I suppose we could use both the SHF_GNU_RETAIN flag and the KEEP() logic for now, but make the use dependent on the binutils version so it can get dropped in a few years when binutils-2.36 becomes the minimum

[nathanchance]

Arnd's patch upstream: https://lore.kernel.org/r/20210226140352.3477860-1-arnd@kernel.org/

[nickdesaulniers]

so @kees can you pick this up then? It sounds like Catalin wasn't planning to?

[kees]

My understanding is that it is currently an unreachable situation, since CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is unsupported on arm64.

[nickdesaulniers]

Right, but don't we want to be able to support CONFIG_LD_DEAD_CODE_DATA_ELIMINATION on arm64 one day? So wouldn't this be a blocker to that?

[nickdesaulniers]

Maybe we can isolate configs down further that reproduce this?

Here's what I have applied locally. allnoconfig and defconfig can't reproduce this, and I didn't have time to test allyesconfig. Requires setting EXPERT=y LD_DEAD_CODE_DATA_ELIMINATION=y.

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9fb9fff08c94..3c64958e31e5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -38,6 +38,7 @@ config ARM64
        select ARCH_HAS_SET_DIRECT_MAP
        select ARCH_HAS_SET_MEMORY
        select ARCH_STACKWALK
+       select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
        select ARCH_HAS_STRICT_KERNEL_RWX
        select ARCH_HAS_STRICT_MODULE_RWX
        select ARCH_HAS_SYNC_DMA_FOR_DEVICE

I do see new warnings from defconfig though:

ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub.stub.o):(.init.bss.efi_system_table) is being placed in '.init.bss.efi_system_table'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub.stub.o):(.init.bss.flat_va_mapping) is being placed in '.init.bss.flat_va_mapping'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_nokaslr) is being placed in '.init.bss.efi_nokaslr'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.data.efi_loglevel) is being placed in '.init.data.efi_loglevel'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_noinitrd) is being placed in '.init.bss.efi_noinitrd'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_nochunk) is being placed in '.init.bss.efi_nochunk'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_novamap) is being placed in '.init.bss.efi_novamap'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_disable_pci_dma) is being placed in '.init.bss.efi_disable_pci_dma'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.0) is being placed in '.init.bss.cmdline.0'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.1) is being placed in '.init.bss.cmdline.1'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.2) is being placed in '.init.bss.cmdline.2'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.3) is being placed in '.init.bss.cmdline.3'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.4) is being placed in '.init.bss.cmdline.4'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub.stub.o):(.init.bss.efi_system_table) is being placed in '.init.bss.efi_system_table'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub.stub.o):(.init.bss.flat_va_mapping) is being placed in '.init.bss.flat_va_mapping'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_nokaslr) is being placed in '.init.bss.efi_nokaslr'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.data.efi_loglevel) is being placed in '.init.data.efi_loglevel'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_noinitrd) is being placed in '.init.bss.efi_noinitrd'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_nochunk) is being placed in '.init.bss.efi_nochunk'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_novamap) is being placed in '.init.bss.efi_novamap'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_disable_pci_dma) is being placed in '.init.bss.efi_disable_pci_dma'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.0) is being placed in '.init.bss.cmdline.0'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.1) is being placed in '.init.bss.cmdline.1'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.2) is being placed in '.init.bss.cmdline.2'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.3) is being placed in '.init.bss.cmdline.3'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.4) is being placed in '.init.bss.cmdline.4'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub.stub.o):(.init.bss.efi_system_table) is being placed in '.init.bss.efi_system_table'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub.stub.o):(.init.bss.flat_va_mapping) is being placed in '.init.bss.flat_va_mapping'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_nokaslr) is being placed in '.init.bss.efi_nokaslr'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.data.efi_loglevel) is being placed in '.init.data.efi_loglevel'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_noinitrd) is being placed in '.init.bss.efi_noinitrd'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_nochunk) is being placed in '.init.bss.efi_nochunk'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_novamap) is being placed in '.init.bss.efi_novamap'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(efi-stub-helper.stub.o):(.init.bss.efi_disable_pci_dma) is being placed in '.init.bss.efi_disable_pci_dma'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.0) is being placed in '.init.bss.cmdline.0'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.1) is being placed in '.init.bss.cmdline.1'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.2) is being placed in '.init.bss.cmdline.2'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.3) is being placed in '.init.bss.cmdline.3'
ld.lld: warning: ./drivers/firmware/efi/libstub/lib.a(gop.stub.o):(.init.bss.cmdline.4) is being placed in '.init.bss.cmdline.4'

Those are likely from --orphan-handling=warn; @ardbiesheuvel suggests that the linker scripts might be missing the .init.bss.* (wildcard suffix).