lld: arm64 big-endian: ld.lld: error: unknown emulation: aarch64linuxb

[arndb]

This is what I get when building an arm64 big-endian kernel:

ld.lld: error: unknown emulation: aarch64linuxb
make[4]: *** [/git/arm-soc/arch/arm64/kernel/vdso/Makefile:56: arch/arm64/kernel/vdso/vdso.so.dbg] Error 1

I suspect this trivial to workaround, as lld seems to support big-endian in principle, but I could not figure out what the actual problem is.

[nathanchance]

We probably need https://git.kernel.org/linus/28187dc8ebd938d574edfc6d9e0f9c51c21ff3f4 for arm64 as well. LLD does not support big-endian ARM: #380

[arndb]

I don't think that's the case actually: I can link a big-endian arm64 user space executable with ld.lld, it sounds more like something in the way we call ld on arm64 is slightly different between ld.bfd and ld.lld.

[MaskRay]

Both big-endian arm and aarch64 are unsupported.

See https://reviews.llvm.org/D58655#1410282 for aarch64be (@smithp35)

The unclear things are what relocation types need write32/write64 instead of write32le/write64le.

[arndb]

Should lld refuse to link big-endian binaries then, instead of producing invalid output?

FWIW, I found that I only needed a small patch to actually get the big-endian kernel to build with both linkers:

diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6a87d592bd00..18a3011fbaed 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -110,7 +110,7 @@ KBUILD_CPPFLAGS     += -mbig-endian
 CHECKFLAGS     += -D__AARCH64EB__
 # Prefer the baremetal ELF build target, but not all toolchains include
 # it so fall back to the standard linux version if needed.
-KBUILD_LDFLAGS += -EB $(call ld-option, -maarch64elfb, -maarch64linuxb)
+KBUILD_LDFLAGS += -EB
 UTS_MACHINE    := aarch64_be
 else
 KBUILD_CPPFLAGS        += -mlittle-endian
diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
index d808ad31e01f..9aa0f4704cfe 100644
--- a/arch/arm64/kernel/vdso/vdso.lds.S
+++ b/arch/arm64/kernel/vdso/vdso.lds.S
@@ -12,7 +12,6 @@
 #include <asm/page.h>
 #include <asm/vdso.h>
 
-OUTPUT_FORMAT("elf64-littleaarch64", "elf64-bigaarch64", "elf64-littleaarch64")
 OUTPUT_ARCH(aarch64)
 
 SECTIONS

This avoids the two build failures i see with lld, and the resulting kernel still boots when built with ld.bfd, but is silently broken with lld, and that seems worse than an error message rejecting the '-EB' command line flag to the linker.

[smithp35]

I agree that an explicit error message is preferable to silently not working.

It may be worth approaching TCWG in Linaro if BE support is important enough to work on.

This is my memory of what we'd need to do to get BE support working.

AArch64 support in LLD shouldn't be too far away. Instructions in AArch64 are always LE so we'd have to look through the code for any if BE do this based on the template parameters and force LE instead. I expect that it is mostly a matter of going through the source and making changes then do a lot of testing. I'd expect that getting applications to work on top of a bfd linked kernel first will be easier than getting the kernel to boot though.

Arm support in LLD needs more work.

• Pre v6 (arm9 etc.) work similarly to other architectures, everything from ELF metadata, instructions and data is LE or BE.
• v6 and above (arm11 and Cortex) relocatable objects are like Pre v6, all LE or BE. Executables and shared objects have LE instructions but BE data and BE ELF metadata. This means that for BE inputs are BE ELF files and LLD has to endian reverse instructions, leaving data including literal data in executable sections alone.

The endian reversal of instructions isn't a huge amount of work but it is messy. The mapping symbols $a $t and $d give the range of Arm, Thumb and Data. The linker (after relocation) needs to endian reverse a word at a time for $a (4-byte instructions), a halfword at a time for $t (2-byte instructions) and leave $d alone.

[arndb]

I tried various combinations of clang/gcc, ld.bfd/ld.lld and big-endian/little-endian to understand what the behavior is today. This is what I found:

• ld.bfd rejects mismatched endianness between input files and command line arguments with an error message, while ld.lld ignores the '-EB' and '-EL' arguments and instead just use output of the same format as input. This seems to work correctly, but it would be better to have consistent behavior in lld and also error out. lld does produce an error when passed incompatible input files, i.e. both big- and little-endian objects.

• gcc and clang produce the same big-endian relocatable object types with byteswapped instructions for 32-bit and non-swapped instructions for 64-bit. Both produce working kernels when linking with ld.bfd.

• ld.lld does not understand the '--be8' argument that is required for big-endian ARMv6/v7 Linux binaries (the kernel always uses be8 even when the CPU supports both be32 and be8) but reject that with an error message. The behavior seems to always be be32 mode on 32-bit arm, where nothing is swapped by the linker, but data and code just get taken as-is from the compiler. For 64-bit, it looks like it does the right thing, but the resulting kernel does not boot.

I don't have a big-endian user space, so I was unable to try running user space executables. It should be possible to test them using qemu-user though, with libc as the only required dependency.

[nathanchance]

Patch sent to disable the config for ld.lld like we did for 32-bit ARM: https://lore.kernel.org/r/20210202022441.1451389-1-nathan@kernel.org/

[MaskRay]

Thalheim kindly shared with me an aarch64_be rootfs. With aarch64_be-linux-musl-native.tgz on https://musl.cc , I verified that
https://reviews.llvm.org/D96188 linked aarch64_be hello world works.

You may change the LD_IS_LLD patch to check <13.0.0 instead.

[nathanchance]

@MaskRay thanks a lot! I can confirm that D96188 works with https://github.com/nathanchance/boot-utils/commits/b8883fd4d8cc28f45cd43d9905084b2f0c8c6de3.

There is still one other issue that prevents ARCH=arm64 defconfig + CONFIG_CPU_BIG_ENDIAN=y from working, which is #1025:

$ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LLVM=1 distclean defconfig

$ scripts/config -d CPU_LITTLE_ENDIAN -e CPU_BIG_ENDIAN

$ make -skj"$(nproc)" ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LLVM=1 olddefconfig Image.gz
ld.lld: error: arch/arm64/kernel/vdso/vgettimeofday.o is incompatible with elf64-littleaarch64
ld.lld: error: arch/arm64/kernel/vdso/note.o is incompatible with elf64-littleaarch64
ld.lld: error: arch/arm64/kernel/vdso/sigreturn.o is incompatible with elf64-littleaarch64
...

For the sake of testing this, I did:

diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
index 61dbb4c838ef..12cdb72a37c6 100644
--- a/arch/arm64/kernel/vdso/vdso.lds.S
+++ b/arch/arm64/kernel/vdso/vdso.lds.S
@@ -12,7 +12,7 @@
 #include <asm/page.h>
 #include <asm/vdso.h>

-OUTPUT_FORMAT("elf64-littleaarch64", "elf64-bigaarch64", "elf64-littleaarch64")
+OUTPUT_FORMAT("elf64-bigaarch64")
 OUTPUT_ARCH(aarch64)

 SECTIONS

[nickdesaulniers]

Thanks @MaskRay !

Do we want to cherry pick the LLVM fix to 12.x branch, or let this ride the 13.x train? We probably need to amend the kernel patch to check the specific LLD version.

[nathanchance]

I think that it would be nice to have this and the fix for #1025 in LLVM 12.x, that is up to @MaskRay and @tstellar though.

Regardless, I will resend the patch to be either

depends on !LD_IS_LLD || LLD_VERSION >= 120000

or

depends on !LD_IS_LLD || LLD_VERSION >= 130000

depending on the outcome.

[MaskRay]

Supporting aarch64_be is a new feature, and we lack testing on goodness, so I think it is not suitable for 12.x.
Since the features are early in 13.x, people with aarch64_be usage have sufficient time to test it.

depends on !LD_IS_LLD || LLD_VERSION >= 130000 looks good to me.

[nathanchance]

That is fine with me, I will send that patch along this afternoon.

[nathanchance]

v2: https://lore.kernel.org/r/20210209005719.803608-1-nathan@kernel.org/

[nickdesaulniers]

Should mark #380 a dup of this.

[nathanchance]

Kernel side patch accepted: https://git.kernel.org/arm64/c/e9c6deee00e9

[nathanchance]

Merged into mainline: https://git.kernel.org/torvalds/c/e9c6deee00e9