[lld] Thread-related non-reproducibility when linking FreeBSD EFI loader

[DimitryAndric]

As part of investigations for https://bugs.freebsd.org/276961 ("buildworld artifacts not reproducible between native and cross build") @emaste and I also noticed that parts of the FreeBSD EFI boot loaders would be different between each build.

This turned out to be an lld issue, where linking exactly same binary twice results in a different output file. It is related to threading, because adding -Wl,--threads=1 makes the output reproducible. (And this is the workaround we have committed in freebsd/freebsd-src@f5ce3f4.)

Since this does not happen to any other ELF binaries in our tree, the custom linker script is probably the cause, but we would still expect the output of lld to be deterministic.

I have attached a tarball with a full reproduction test case:
loader_simp.tar.gz

[llvmbot]

@llvm/issue-subscribers-lld-elf

Author: Dimitry Andric (DimitryAndric)

As part of investigations for https://bugs.freebsd.org/276961 ("buildworld artifacts not reproducible between native and cross build") @emaste and I also noticed that parts of the FreeBSD EFI boot loaders would be different between each build.

This turned out to be an lld issue, where linking exactly same binary twice results in a different output file. It is related to threading, because adding -Wl,--threads=1 makes the output reproducible. (And this is the workaround we have committed in freebsd/freebsd-src@f5ce3f4.)

Since this does not happen to any other ELF binaries in our tree, the custom linker script is probably the cause, but we would still expect the output of lld to be deterministic.

I have attached a tarball with a full reproduction test case:
loader_simp.tar.gz

[DimitryAndric]

Bisection shows that this regressed with llvmorg-17-init-09355-gfea8c07 ("[Support][Parallel] Add sequential mode to TaskGroup::spawn()") by @avl-llvm (cc @MaskRay @nga888).

[avl-llvm]

Will take a look. Thank you for reporting the issue.