[META] Using LLD as FreeBSD's system linker

[emaste]

Bugzilla Link	23214
Resolution	FIXED
Resolved on	Nov 04, 2020 12:16
Version	unspecified
OS	FreeBSD
Depends On	#21350 #23280 #23398 #23409 #23605 #23606 #23607 #25562 #25902 #25961 #26164 #26172 #27050 #27067 #27073 #27079 #27086 #27103 #27104 #27105 #27106 #27111 #27187 #27188 #27192 #27252 #27342 #27390 #27602 #27780 #27781 #28021 #28267 #28723 #28731 #28733 #28788 #29062 #29063 #29064 #29092 #29346 #29369 #29463 #29485 #29566 #29569 #29570 #29585 #29616 #29617 #29630 #29659 #29660 #29754 #29763 #30239 #30405 #30496 #30497 #30507 #30544 #30561 #30569 #30625 #30643 #30843 #30930 #30967 #31064 #31110 #31851 #32769 #32770 #32829 #33113 #33167 #33168 #33512 #33659 #34918 #35068 #35099 #35136 #35357 #35365
CC	@adalava,@asl,@glaubitz,@grimreaper,@markmi,@rengolin,@rui314,@lattera

Extended Description

This is a meta-bug to track dependencies needed for LLD to be functional as FreeBSD's system linker.

[rui314]

Do you have a list of missing features for this bug?

[emaste]

There are the depends-on PRs:

• We need _start
  and _stop

  as we use linker sets in a few programs
• -r relocatable link

I just closed #23409 as the change was reapplied with a fixed test case, but have found a new issue relating to dependency handling (no PR yet):

link main.o -l -l, where libstatic.a depends on libshared.a. LLD reports an undefined symbol from libstatic.a. GNU ld accepts this. Arguably the link order ought to be changed around in our build, but other software will likely encounter this too.

For the actual case in FreeBSD the static library was NTP - it builds a libntp.a that depends on libedit, and a binary linked with -ledit -lntp.

I also observed one "error: unsupported emulation 'elf_i386_fbsd'."

PRs for these two and other issues we find will be added as dependencies of this one.

[emaste]

We also need linker script support, although I don't have a list of the specific linker script features we rely on just yet.

[emaste]

Unknown arguments during world (userland) build:

warning: ignoring unknown argument: --version-script=Version.map
warning: ignoring unknown argument: -x
warning: ignoring unknown argument: --fatal-warnings
warning: ignoring unknown argument: --warn-shared-textrel

From FreeBSD arm64 kernel link:

--- kernel.debug ---
linking kernel.debug
warning: ignoring unknown argument: -Bdynamic
warning: ignoring unknown argument: --no-warn-mismatch
warning: ignoring unknown argument: --warn-common
warning: ignoring unknown argument: --dynamic-linker
warning: ignoring unknown argument: -X
SymbolTable: error while merging __bss_start
LLVM ERROR: duplicate symbol error

[emaste]

Note that this meta PR was created based on the first version of LLD ELF support, and needs to be revisited for elf2.

[emaste]

Trying with ELF2 turned up the issues below; we're not yet at the point in ELF2's development where it makes sense to track individual PRs for them.

• PT_INTERP and ABI notes must be in the first page

ELF2 currently rounds up to a page and stores the data for PT_INTERP
and the NT_FREEBSD_ABI_TAG and NT_FREEBSD_NOINIT_TAG ELF notes there.
These have to be in the first page on FreeBSD.

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flg Align
INTERP 0x00000000000010cc 0x00000000000110cc 0x00000000000110cc
0x0000000000000015 0x0000000000000015 R 0x1
[Requesting program interpreter: /libexec/ld-elf.so.1]

• Undefined sym in shared library provided by regular object

This is the FreeBSD __progname issue fixed in r218259 for the original
ELF linker.

• Options used in the FreeBSD base system

These are options that need to be ignored or implemented (excluding
those already ignored by ELF2)

--enable-new-dtags
--fatal-warnings and --no-fatal-warnings
--warn-shared-textrel
--output-filetype
--version-script
-Bsymbolic
-r
-znodelete
--no-warn-mismatch
--warn-common
-T

• Linker script support

The FreeBSD base system linker scripts use (at least) the following set:
ALIGN ENTRY GROUP KEEP OUTPUT_ARCH OUTPUT_FORMAT SEARCH_DIR SECTIONS PROVIDE

(http://lists.llvm.org/pipermail/llvm-dev/2015-October/091072.html)

[llvmbot]

--enable/disable-new-dtags is r249428

[rui314]

What kind of expressions are used in SECTIONS linker script directive?

[emaste]

What kind of expressions are used in SECTIONS linker script directive?

Examples can be found here: https://svnweb.freebsd.org/base/head/sys/conf/
For amd64: https://svnweb.freebsd.org/base/head/sys/conf/ldscript.amd64?view=log

Some examples:

kernphys = CONSTANT (MAXPAGESIZE);
. = kernbase + kernphys + SIZEOF_HEADERS;
.interp : AT (kernphys + SIZEOF_HEADERS) { *(.interp) }
.hash : { (.hash) }
.rel.text : { (.rel.text .rel.text. .rel.gnu.linkonce.t.) }

.init :
{
KEEP (*(.init))
} =0x90909090

PROVIDE (__etext = .);

.eh_frame : ONLY_IF_RO { KEEP (*(.eh_frame)) }

. = ALIGN (CONSTANT (MAXPAGESIZE)) - ((CONSTANT (MAXPAGESIZE) - .) & (CONSTANT (MAXPAGESIZE) - 1)); . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE));

.preinit_array :
{
PROVIDE_HIDDEN (__preinit_array_start = .);
KEEP (*(.preinit_array))
PROVIDE_HIDDEN (__preinit_array_end = .);
}

. = ALIGN(. != 0 ? 64 / 8 : 1);

.debug_pubtypes 0 : { *(.debug_pubtypes) }

[emaste]

With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD "hello world."

The current userland world build fails due to unhandled R_X86_64_TLSGD:

--- libc.so.7 ---
unrecognized reloc 19
cc: error: linker command failed with exit code 1 (use -v to see invocation)

The kernel build fails in the linker script parser:

--- kernel.full ---
linking kernel.full
) expected, but got ,

[llvmbot]

With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD
"hello world."

The current userland world build fails due to unhandled R_X86_64_TLSGD:

--- libc.so.7 ---

unrecognized reloc 19

cc: error: linker command failed with exit code 1 (use -v to see invocation)

Michael Spencer is working on it -- there's an initial patch under review so that should be unlocked soon.

The kernel build fails in the linker script parser:

--- kernel.full ---

linking kernel.full

) expected, but got ,

http://reviews.llvm.org/D13668

[emaste]

Update,

With the WIP -r patch (http://reviews.llvm.org/D14382) linking static libpam produces a new error:

$ ld -o openpam_static_modules.o -r --whole-archive openpam_static.o ../modules/pam_chroot/libpam_chroot.a ../modules/pam_deny/libpam_deny.a [...]
FDE doesn't reference another section

If I'm able to investigate and produce a small reproduction case I will submit a new PR.

Also, the lack of GNU symbol version support is now becoming a blocking issue for making further progress on FreeBSD - our libc has some cases where it is not usable without a symver-supporting linker.

[rui314]

Do you keep track of the progress? I'm interested in the numbers of packages we can and can't link with lld (with test success).

[emaste]

Do you keep track of the progress? I'm interested in the numbers of packages we
can and can't link with lld (with test success).

We're not yet at the point of trying to link the ports tree with lld -- I'm working on the base system now.

After recent commits and the WIP -r patch I've been able to reduce the set of hacks in my build tree to a small handful:

• default library search paths - ld.bfd includes /lib as a built-in search path, for lld I need to specify it explicitly (in the libc.so DSO linker script)
• skip libpam/static_modules due to "FDE doesn't reference another section" as mentioned in an earlier comment
• skip the i386 boot loaders due to lack of -N support in lld (proposed implementation for the old elf linker in http://reviews.llvm.org/D968)
• skip the gdb build - it failed because lld does not provide an _etext symbol
• change -Tscript to -T script
• remove the EFI boot loaders because they require linker script functionality not yet implemented
• skip uathload as it relies on an unusual use of -r (converting a binary blob into an ELF object)

For now I'm not trying to build the kernel with lld.

[llvmbot]

Do you keep track of the progress? I'm interested in the numbers of packages we
can and can't link with lld (with test success).

We're not yet at the point of trying to link the ports tree with lld -- I'm
working on the base system now.

After recent commits and the WIP -r patch I've been able to reduce the set
of hacks in my build tree to a small handful:

• default library search paths - ld.bfd includes /lib as a built-in search
  path, for lld I need to specify it explicitly (in the libc.so DSO linker
  script)
• skip libpam/static_modules due to "FDE doesn't reference another section"
  as mentioned in an earlier comment
• skip the i386 boot loaders due to lack of -N support in lld (proposed
  implementation for the old elf linker in http://reviews.llvm.org/D968)
• skip the gdb build - it failed because lld does not provide an _etext
  symbol
• change -Tscript to -T script
• remove the EFI boot loaders because they require linker script
  functionality not yet implemented
• skip uathload as it relies on an unusual use of -r (converting a binary
  blob into an ELF object)

If you can open individual bugs for each one that would be awesome :-)

[emaste]

If you can open individual bugs for each one that would be awesome :-)

Unlike the recent PRs I've opened up I'm not convinced all of these are bugs (in lld):

• default library search paths - ld.bfd includes /lib as a built-in search
  path, for lld I need to specify it explicitly (in the libc.so DSO linker
  script)

The clang driver passes -L/usr/lib to ld and we expect to resolve -llib args from there; this issue only happens because /usr/lib/libc.so is a linker script that contains:

GROUP ( libc.so.7 libc_nonshared.a libssp_nonshared.a )

I think it should probably be:

GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a /usr/lib/libssp_nonshared.a )

• skip libpam/static_modules due to "FDE doesn't reference another section"
  as mentioned in an earlier comment

This one indeed looks like an lld issue and I'll submit a PR once I have at least some detail.

• skip the i386 boot loaders due to lack of -N support in lld (proposed
  implementation for the old elf linker in http://reviews.llvm.org/D968)

-N should be easy to implement, but I'm not sure how useful or widely used it is, and it seems reasonable for us to rework our boot components to avoid it.

• skip the gdb build - it failed because lld does not provide an _etext
  symbol

PR 26729

• change -Tscript to -T script

PR 26730
We can easily make this change in FreeBSD, but as we start trying to compile large sets of packages with lld I'm sure this will come up again.

• remove the EFI boot loaders because they require linker script
  functionality not yet implemented

Will submit a PR referencing specific missing functionality

• skip uathload as it relies on an unusual use of -r (converting a binary
  blob into an ELF object)

While I believe -r support is important in lld I'm not sure I can argue for this use, and will probably change the FreeBSD build to use objcopy instead.

[emaste]

The set of workarounds I'm using to try building FreeBSD world with lld is in my github FreeBSD tree: https://github.com/emaste/freebsd/tree/lld-buildworld
Other than those changes I'm testing with upstream FreeBSD and upstream lld.

The major outstanding issue is lack of expression support in linker scripts - in my tree the kernel is being linked with ld.bfd for this reason. There are also a few other components that are disabled or linked with ld.bfd.

I've built a VM image from this tree and large portions work; the system boots to a login prompt, shells work, clang works, etc. Anything linked against libxo (which includes ls and ps) segfaults - PR 26818. ps is used in many places in the startup scripts and thus results in a lot of console spam.

Once PR 26818 is fixed I believe enough will be working that we can test lld in a FreeBSD ports experimental run, trying to build the ~25K third party software packages in the ports tree with lld.

[rui314]

Thank you very much for the updates. Linker script is indeed the next big one (and hopefully the last one), but before start working on it, I'm trying to support versioned symbols.

[emaste]

I'm trying to support versioned symbols.

Great.

For my immediate testing I'm building a full system w/o symver and the lack of support isn't holding me up -- I might have a couple of workarounds that I'll need to apply, but it's not a significant problem for buildworld (userland). Using a ld.bfd-linked kernel fine for testing.

Indeed, versioned symbol support will be very important for us if we're to provide a /usr/bin/ld.lld, in order to interoperate with system components -- particularly libc.so -- linked by GNU ld.

[llvmbot]

little note about:

• skip libpam/static_modules due to "FDE doesn't reference another section" >as mentioned in an earlier comment

That should be fixed in r262590. That message was shown becauses eh_frame sections were proccessed during generating relocatable output and from that revision it is not happens anymore as well as this adds support for generating relocations against local symbols which were sections in that case.

[rui314]

In r262910, I add /usr and /usr/lib as default search paths. I think that should resolve one issue for FreeBSD.

[emaste]

Changes have been committed to FreeBSD to address some of these issues:

• Library search paths, added to lld and then reverted. Now solved by leaving the full paths in the /usr/lib/libc.so linker script: https://svnweb.freebsd.org/changeset/base/296921

• ld -r to convert a binary object to ELF. Solved by rewriting the utility to avoid having a compiled-in binary object: https://svnweb.freebsd.org/changeset/base/296889

I'm now starting to test FreeBSD/arm64 (AArch64) buildworld with lld. It currently fails when building libc:

/usr/local/aarch64-freebsd/bin/ld: getutxent.So(.debug_info+0x3c): R_AARCH64_ABS64 used with TLS symbol udb
/usr/local/aarch64-freebsd/bin/ld: getutxent.So(.debug_info+0x59): R_AARCH64_ABS64 used with TLS symbol uf
/usr/local/aarch64-freebsd/bin/ld: utxdb.So(.debug_info+0x5c): R_AARCH64_ABS64 used with TLS symbol futx_to_utx.ut
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.So(.debug_info+0x3d): R_AARCH64_ABS64 used with TLS symbol __je_tsd_tls
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.So(.debug_info+0x12dc): R_AARCH64_ABS64 used with TLS symbol __je_tsd_initialized
/usr/local/aarch64-freebsd/bin/ld: xlocale.So(.debug_info+0x404): R_AARCH64_ABS64 used with TLS symbol __thread_locale
/usr/local/aarch64-freebsd/bin/ld: setrunelocale.So(.debug_info+0x3d): R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale

Note that patches are open for arm64 TLS relocation work:

http://reviews.llvm.org/D16201
http://reviews.llvm.org/D17980
http://reviews.llvm.org/D18026
http://reviews.llvm.org/D18031

[llvmbot]

• skip the i386 boot loaders due to lack of -N support in lld (proposed
  implementation for the old elf linker in http://reviews.llvm.org/D968)

-N should be easy to implement, but I'm not sure how useful or widely used
it is, and it seems reasonable for us to rework our boot components to avoid
it.

So, just in case. We are not going to support -N, right ? (otherwise I would take that).

[rui314]

Because -N option is for compatibility with old Unix systems that have died out in 80s or 90s. If you still want that behavior for some special case (such as bootloader or something), you should use a linker script instead.

[emaste]

Because -N option is for compatibility with old Unix systems that have died
out in 80s or 90s. If you still want that behavior for some special case
(such as bootloader or something), you should use a linker script instead.

Yes, I assumed that I will be changing the FreeBSD build to implement what's needed via a linker script.

I believe lld's linker script support is not yet fully able to implement -N's functionality and I will continue to link these components with GNU ld until it can.

[llvmbot]

Because -N option is for compatibility with old Unix systems that have died
out in 80s or 90s. If you still want that behavior for some special case
(such as bootloader or something), you should use a linker script instead.

That's why I asked :)

[emaste]

Attempting to link FreeBSD/mips64 with lld reports:

warning: unknown argument: -EB

I do not yet know why there's an explicit big/little endian argument passed in by Clang.

[rui314]

It looks like lib/Driver/Tools.cpp always produces -m elf{32,64}{bt,lt}smip flags, so maybe -EB/EL are just redundant?

[emaste]

It looks like lib/Driver/Tools.cpp always produces -m elf{32,64}{bt,lt}smip flags, so maybe -EB/EL are just redundant?

Yes, I'm having trouble imagining how -EB/-EL could be used, where endianness is not already determined by either -m or the object file(s). For testing I've just added -EB and -EL as ignored options.

[silvasean]

Moving to "ELF" bugzilla component (instead of generic "All Bugs" for LLD).