[META] Using LLD as FreeBSD's system linker

[emaste]

Bugzilla Link	23214
Resolution	FIXED
Resolved on	Nov 04, 2020 12:16
Version	unspecified
OS	FreeBSD
Depends On	#21350 #23280 #23398 #23409 #23605 #23606 #23607 #25562 #25902 #25961 #26164 #26172 #27050 #27067 #27073 #27079 #27086 #27103 #27104 #27105 #27106 #27111 #27187 #27188 #27192 #27252 #27342 #27390 #27602 #27780 #27781 #28021 #28267 #28723 #28731 #28733 #28788 #29062 #29063 #29064 #29092 #29346 #29369 #29463 #29485 #29566 #29569 #29570 #29585 #29616 #29617 #29630 #29659 #29660 #29754 #29763 #30239 #30405 #30496 #30497 #30507 #30544 #30561 #30569 #30625 #30643 #30843 #30930 #30967 #31064 #31110 #31851 #32769 #32770 #32829 #33113 #33167 #33168 #33512 #33659 #34918 #35068 #35099 #35136 #35357 #35365
CC	@adalava,@asl,@glaubitz,@grimreaper,@markmi,@rengolin,@rui314,@lattera

Extended Description

This is a meta-bug to track dependencies needed for LLD to be functional as FreeBSD's system linker.

[rui314]

Do you have a list of missing features for this bug?

[emaste]

There are the depends-on PRs:

• We need _start
  and _stop

  as we use linker sets in a few programs
• -r relocatable link

I just closed #23409 as the change was reapplied with a fixed test case, but have found a new issue relating to dependency handling (no PR yet):

link main.o -l -l, where libstatic.a depends on libshared.a. LLD reports an undefined symbol from libstatic.a. GNU ld accepts this. Arguably the link order ought to be changed around in our build, but other software will likely encounter this too.

For the actual case in FreeBSD the static library was NTP - it builds a libntp.a that depends on libedit, and a binary linked with -ledit -lntp.

I also observed one "error: unsupported emulation 'elf_i386_fbsd'."

PRs for these two and other issues we find will be added as dependencies of this one.

[emaste]

We also need linker script support, although I don't have a list of the specific linker script features we rely on just yet.

[emaste]

Unknown arguments during world (userland) build:

warning: ignoring unknown argument: --version-script=Version.map
warning: ignoring unknown argument: -x
warning: ignoring unknown argument: --fatal-warnings
warning: ignoring unknown argument: --warn-shared-textrel

From FreeBSD arm64 kernel link:

--- kernel.debug ---
linking kernel.debug
warning: ignoring unknown argument: -Bdynamic
warning: ignoring unknown argument: --no-warn-mismatch
warning: ignoring unknown argument: --warn-common
warning: ignoring unknown argument: --dynamic-linker
warning: ignoring unknown argument: -X
SymbolTable: error while merging __bss_start
LLVM ERROR: duplicate symbol error

[emaste]

Note that this meta PR was created based on the first version of LLD ELF support, and needs to be revisited for elf2.

[emaste]

Trying with ELF2 turned up the issues below; we're not yet at the point in ELF2's development where it makes sense to track individual PRs for them.

• PT_INTERP and ABI notes must be in the first page

ELF2 currently rounds up to a page and stores the data for PT_INTERP
and the NT_FREEBSD_ABI_TAG and NT_FREEBSD_NOINIT_TAG ELF notes there.
These have to be in the first page on FreeBSD.

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flg Align
INTERP 0x00000000000010cc 0x00000000000110cc 0x00000000000110cc
0x0000000000000015 0x0000000000000015 R 0x1
[Requesting program interpreter: /libexec/ld-elf.so.1]

• Undefined sym in shared library provided by regular object

This is the FreeBSD __progname issue fixed in r218259 for the original
ELF linker.

• Options used in the FreeBSD base system

These are options that need to be ignored or implemented (excluding
those already ignored by ELF2)

--enable-new-dtags
--fatal-warnings and --no-fatal-warnings
--warn-shared-textrel
--output-filetype
--version-script
-Bsymbolic
-r
-znodelete
--no-warn-mismatch
--warn-common
-T

• Linker script support

The FreeBSD base system linker scripts use (at least) the following set:
ALIGN ENTRY GROUP KEEP OUTPUT_ARCH OUTPUT_FORMAT SEARCH_DIR SECTIONS PROVIDE

(http://lists.llvm.org/pipermail/llvm-dev/2015-October/091072.html)

[llvmbot]

--enable/disable-new-dtags is r249428

[rui314]

What kind of expressions are used in SECTIONS linker script directive?

[emaste]

What kind of expressions are used in SECTIONS linker script directive?

Examples can be found here: https://svnweb.freebsd.org/base/head/sys/conf/
For amd64: https://svnweb.freebsd.org/base/head/sys/conf/ldscript.amd64?view=log

Some examples:

kernphys = CONSTANT (MAXPAGESIZE);
. = kernbase + kernphys + SIZEOF_HEADERS;
.interp : AT (kernphys + SIZEOF_HEADERS) { *(.interp) }
.hash : { (.hash) }
.rel.text : { (.rel.text .rel.text. .rel.gnu.linkonce.t.) }

.init :
{
KEEP (*(.init))
} =0x90909090

PROVIDE (__etext = .);

.eh_frame : ONLY_IF_RO { KEEP (*(.eh_frame)) }

. = ALIGN (CONSTANT (MAXPAGESIZE)) - ((CONSTANT (MAXPAGESIZE) - .) & (CONSTANT (MAXPAGESIZE) - 1)); . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE));

.preinit_array :
{
PROVIDE_HIDDEN (__preinit_array_start = .);
KEEP (*(.preinit_array))
PROVIDE_HIDDEN (__preinit_array_end = .);
}

. = ALIGN(. != 0 ? 64 / 8 : 1);

.debug_pubtypes 0 : { *(.debug_pubtypes) }

[emaste]

With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD "hello world."

The current userland world build fails due to unhandled R_X86_64_TLSGD:

--- libc.so.7 ---
unrecognized reloc 19
cc: error: linker command failed with exit code 1 (use -v to see invocation)

The kernel build fails in the linker script parser:

--- kernel.full ---
linking kernel.full
) expected, but got ,

[llvmbot]

With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD
"hello world."

The current userland world build fails due to unhandled R_X86_64_TLSGD:

--- libc.so.7 ---

unrecognized reloc 19

cc: error: linker command failed with exit code 1 (use -v to see invocation)

Michael Spencer is working on it -- there's an initial patch under review so that should be unlocked soon.

The kernel build fails in the linker script parser:

--- kernel.full ---

linking kernel.full

) expected, but got ,

http://reviews.llvm.org/D13668

[emaste]

Update,

With the WIP -r patch (http://reviews.llvm.org/D14382) linking static libpam produces a new error:

$ ld -o openpam_static_modules.o -r --whole-archive openpam_static.o ../modules/pam_chroot/libpam_chroot.a ../modules/pam_deny/libpam_deny.a [...]
FDE doesn't reference another section

If I'm able to investigate and produce a small reproduction case I will submit a new PR.

Also, the lack of GNU symbol version support is now becoming a blocking issue for making further progress on FreeBSD - our libc has some cases where it is not usable without a symver-supporting linker.

[rui314]

Do you keep track of the progress? I'm interested in the numbers of packages we can and can't link with lld (with test success).

[emaste]

Do you keep track of the progress? I'm interested in the numbers of packages we
can and can't link with lld (with test success).

We're not yet at the point of trying to link the ports tree with lld -- I'm working on the base system now.

After recent commits and the WIP -r patch I've been able to reduce the set of hacks in my build tree to a small handful:

• default library search paths - ld.bfd includes /lib as a built-in search path, for lld I need to specify it explicitly (in the libc.so DSO linker script)
• skip libpam/static_modules due to "FDE doesn't reference another section" as mentioned in an earlier comment
• skip the i386 boot loaders due to lack of -N support in lld (proposed implementation for the old elf linker in http://reviews.llvm.org/D968)
• skip the gdb build - it failed because lld does not provide an _etext symbol
• change -Tscript to -T script
• remove the EFI boot loaders because they require linker script functionality not yet implemented
• skip uathload as it relies on an unusual use of -r (converting a binary blob into an ELF object)

For now I'm not trying to build the kernel with lld.

[llvmbot]

Do you keep track of the progress? I'm interested in the numbers of packages we
can and can't link with lld (with test success).

We're not yet at the point of trying to link the ports tree with lld -- I'm
working on the base system now.

After recent commits and the WIP -r patch I've been able to reduce the set
of hacks in my build tree to a small handful:

• default library search paths - ld.bfd includes /lib as a built-in search
  path, for lld I need to specify it explicitly (in the libc.so DSO linker
  script)
• skip libpam/static_modules due to "FDE doesn't reference another section"
  as mentioned in an earlier comment
• skip the i386 boot loaders due to lack of -N support in lld (proposed
  implementation for the old elf linker in http://reviews.llvm.org/D968)
• skip the gdb build - it failed because lld does not provide an _etext
  symbol
• change -Tscript to -T script
• remove the EFI boot loaders because they require linker script
  functionality not yet implemented
• skip uathload as it relies on an unusual use of -r (converting a binary
  blob into an ELF object)

If you can open individual bugs for each one that would be awesome :-)