ppc64 pseries_defconfig ld.lld errors

[nathanchance]

ld.lld: error: undefined symbol: .early_setup
>>> referenced by arch/powerpc/kernel/head_64.o:(.ref.text+0x72)

ld.lld: error: undefined symbol: .sys_restart_syscall
>>> referenced by kernel/systbl.o:(.rodata+0x0) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xD98) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_exit
>>> referenced by kernel/systbl.o:(.rodata+0x8) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDA0) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_read
>>> referenced by kernel/systbl.o:(.rodata+0x18) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDB0) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_write
>>> referenced by kernel/systbl.o:(.rodata+0x20) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDB8) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_open
>>> referenced by kernel/systbl.o:(.rodata+0x28) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_close
>>> referenced by kernel/systbl.o:(.rodata+0x30) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDC8) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_waitpid
>>> referenced by kernel/systbl.o:(.rodata+0x38) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDD0) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_creat
>>> referenced by kernel/systbl.o:(.rodata+0x40) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDD8) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_link
>>> referenced by kernel/systbl.o:(.rodata+0x48) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDE0) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_unlink
>>> referenced by kernel/systbl.o:(.rodata+0x50) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDE8) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_execve
>>> referenced by kernel/systbl.o:(.rodata+0x58) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_chdir
>>> referenced by kernel/systbl.o:(.rodata+0x60) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xDF8) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_time
>>> referenced by kernel/systbl.o:(.rodata+0x68) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_mknod
>>> referenced by kernel/systbl.o:(.rodata+0x70) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xE08) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_chmod
>>> referenced by kernel/systbl.o:(.rodata+0x78) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xE10) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_lchown
>>> referenced by kernel/systbl.o:(.rodata+0x80) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xE18) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_ni_syscall
>>> referenced by kernel/systbl.o:(.rodata+0x88) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0x90) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xB0) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xE0) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xF8) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0x100) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0x118) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0x160) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0x1A8) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0x1C0) in archive arch/powerpc/built-in.a
>>> referenced 123 more times

ld.lld: error: undefined symbol: .sys_lseek
>>> referenced by kernel/systbl.o:(.rodata+0x98) in archive arch/powerpc/built-in.a

ld.lld: error: undefined symbol: .sys_getpid
>>> referenced by kernel/systbl.o:(.rodata+0xA0) in archive arch/powerpc/built-in.a
>>> referenced by kernel/systbl.o:(.rodata+0xE38) in archive arch/powerpc/built-in.a

To reproduce (on current mainline or -next):

curl -LSs https://raw.githubusercontent.com/nathanchance/patches/ce2a9dd5db891946ecf94906e451554103d40b9e/linux/error-fixes/0003-powerpc-vdso32-Drop-mabi-elfv1-for-32-bit-objects.patch | git am

curl -LSs https://raw.githubusercontent.com/nathanchance/patches/ce2a9dd5db891946ecf94906e451554103d40b9e/linux/error-fixes/0004-powerpc-Kconfig-Don-t-enable-HAVE_FUNCTION_TRACER-on.patch | git am

make -j$(nproc) ARCH=powerpc CC=clang CROSS_COMPILE=powerpc-linux-gnu- LD=ld.lld O=out distclean pseries_defconfig vmlinux

[nickdesaulniers]

What's special about these symbols that leads to LLD discarding them? Is there a linker script involved that LLD is not handling the same way as BFD?

[MaskRay]

This big-endian configuration doesn't work, even if ld.bfd is used:

# HEAD commit 4010b622f1d2a6112244101f38225eaee20c07f2 Tue Jul 30 17:32:46 2019 -0700
% curl -LSs https://raw.githubusercontent.com/nathanchance/patches/ce2a9dd5db891946ecf94906e451554103d40b9e/linux/error-fixes/0003-powerpc-vdso32-Drop-mabi-elfv1-for-32-bit-objects.patch | git am
% curl -LSs https://raw.githubusercontent.com/nathanchance/patches/ce2a9dd5db891946ecf94906e451554103d40b9e/linux/error-fixes/0004-powerpc-Kconfig-Don-t-enable-HAVE_FUNCTION_TRACER-on.patch | git am

# Install powerpc64-linux-gnu-gcc (big-endian)
% make -j"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu- HOSTCC=clang HOSTLD=ld.lld CC=clang LD=ld.lld O=out.ppc64 distclean pseries_defconfig vmlinux
... ld.lld: error: undefined symbol as noted above

% cd out.ppc64; /tmp/powerpc64/ld/ld-new -EB -m elf64ppc -Bstatic --orphan-handling=warn --build-id -o .tmp_vmlinux1 -T ./arch/powerpc/kernel/vmlinux.lds --whole-archive arch/powerpc/kernel/head_64.o arch/powerpc/kernel/entry_64.o arch/powerpc/kernel/fpu.o arch/powerpc/kernel/vector.o arch/powerpc/kernel/prom_init.o init/built-in.a usr/built-in.a arch/powerpc/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a --no-whole-archive --start-group lib/lib.a --end-group
/tmp/powerpc64/ld/ld-new: arch/powerpc/kernel/head_64.o .opd not allowed in ABI version 2
/tmp/powerpc64/ld/ld-new: arch/powerpc/kernel/head_64.o: error adding symbols: bad value

/tmp/powerpc64/ld/ld-new: arch/powerpc/kernel/head_64.o .opd not allowed in ABI version 2

.opd is some ELFv1 stuff.

[nathanchance]

This is interesting... Building without LD=ld.lld works perfectly fine.

$ make -j"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc-linux-gnu- HOSTCC=clang HOSTLD=ld.lld CC=clang O=out.ppc64 V=1 distclean pseries_defconfig vmlinux

However, trying to link all of those files with ld.lld results in a similar set of errors:

$ ( cd out.ppc64; "${CBL_BIN}"/ld.lld -EB -m elf64ppc -Bstatic --orphan-handling=warn --build-id -o .tmp_vmlinux1 -T ./arch/powerpc/kernel/vmlinux.lds --whole-archive arch/powerpc/kernel/head_64.o arch/powerpc/kernel/entry_64.o arch/powerpc/kernel/fpu.o arch/powerpc/kernel/vector.o arch/powerpc/kernel/prom_init.o init/built-in.a usr/built-in.a arch/powerpc/built-in.a
 kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a --no-whole-archive --start-group lib/lib.a --end-group )
ld.lld: error: arch/powerpc/kernel/head_64.o: ABI version 1 is not supported
ld.lld: error: arch/powerpc/kernel/fpu.o: ABI version 1 is not supported
ld.lld: error: arch/powerpc/kernel/vector.o: ABI version 1 is not supported
ld.lld: error: init/built-in.a(main.o): ABI version 1 is not supported
ld.lld: error: init/built-in.a(do_mounts.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/cputable.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/irq.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/pmc.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/process.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/idle.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/sysfs.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/time.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/prom.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/traps.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/setup-common.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/misc.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/io.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/misc_64.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/setup_64.o): ABI version 1 is not supported
ld.lld: error: arch/powerpc/built-in.a(kernel/paca.o): ABI version 1 is not supported
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)

Wonder what changes the ABI version.

[ihalip]

Wonder what changes the ABI version.

Took me a while to figure this out. TLDR: arch/powerpc/Makefile uses ELFv1 ABI for PPC64 & big endian. Since this is not supported by lld, would it be ok to just exit early in the build process with a message clarifying that?

[nathanchance]

I don't know that we need a build error for this right out the gate since as far as I am aware, I am the only one who has tried this :)

If there isn't support in ld.lld for elfv1, I think we can just close this as it is a moot point.

[nathanchance]

I sent CONFIG_LD_IS_LLD to deal with #785.

With that patch, we could tackle this issue using something like this:

diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index f310c32e88a4..97af461e8b8e 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile
@@ -143,7 +143,13 @@ endif
 endif

 CFLAGS-$(CONFIG_PPC64) := $(call cc-option,-mtraceback=no)
-ifndef CONFIG_CC_IS_CLANG
+ifdef CONFIG_CC_IS_CLANG
+ifdef CONFIG_LD_IS_LLD
+ifdef CONFIG_CPU_BIG_ENDIAN
+$(error ld.lld does not support PowerPC big endian due to lack of elfv1 support)
+endif
+endif
+else
 ifdef CONFIG_CPU_LITTLE_ENDIAN
 CFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mabi=elfv2,$(call cc-option,-mcall-aixdesc))
 AFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mabi=elfv2)

[MaskRay]

+ifdef CONFIG_CC_IS_CLANG

I guess CONFIG_CC_IS_CLANG is not needed.

[nathanchance]

Yes, fair point :)

If I end up sending that along, I'll add it.

[MaskRay]

@q66 told me that @leahneukirchen has a patch to drop ELFv1 legacy from arch/powerpc/Makefile. It is simplification on its own (likely upstreamable) and can potentially help ld.lld https://github.com/void-linux/void-packages/blob/master/srcpkgs/linux5.10/patches/ppc64-be-elfv2.patch

[q66]

that's my patch, @leahneukirchen just pulled it in from the previous patches for a previous kernel

[MaskRay]

Thanks for clarification!

https://www.kernel.org/doc/html/latest/process/changes.html requires GCC 4.9. GCC gained support for -mabi=elfv2 in 2013. Perhaps we can make -mabi=elfv2 unconditional and entirely drop -mcall-aixdesc (not supported by Clang as of 12.0.0).

[q66]

the patch should still fall back to ELFv1 and -mcall-aixdesc as it is when the support is not available

there were efforts upstream to add the ELFv2 option previously, first one some years back (which got abandoned later) and IIRC there's a second one that was under review just recently (not sure what became of it)

[nickdesaulniers]

@q66 was there a thread for the upstream discussions that you can link to? This is still reproducible with ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- make CC=clang LD=ld.lld -j71 pseries_defconfig all; we may want to send a revised version of @nathanchance 's patch for now.

[nathanchance]

Big endian ELFv2 support was sent upstream but the commit message of the second patch in the series mentions that LLVM did not receive testing so it was restricted to GCC and binutils in Kconfig: https://lore.kernel.org/r/20210611093959.821525-1-npiggin@gmail.com/

It does not look like it was ever picked up.

[nickdesaulniers]

@q66 told me that @leahneukirchen has a patch to drop ELFv1 legacy from arch/powerpc/Makefile. It is simplification on its own (likely upstreamable) and can potentially help ld.lld https://github.com/void-linux/void-packages/blob/master/srcpkgs/linux5.10/patches/ppc64-be-elfv2.patch

We should ask @mpe on the list whether big endian could be moved to ELFv2 (via that patch).

@nathanchance found https://lore.kernel.org/all/20210611093959.821525-1-npiggin@gmail.com/, but mentioned that there's some Kconfig changes we need to make+test. @nathanchance can you provide more info?

@mandlebug mentioned that FreeBSD switched their BE ppc64 build to ELFv2.

[nickdesaulniers]

https://lore.kernel.org/all/20210611093959.821525-3-npiggin@gmail.com/ mentions:

although it is rumored to work with clang
that's not been tested yet.

Let's hop on that thread and help test.

[nathanchance]

This should not be visible in mainline after https://git.kernel.org/linus/9d90161ca5c7234e80e14e563d198f322ca0c1d0. Closing this up for now.