Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linker error for atomic functions on aarch64-linux #201254

Closed
avdv opened this issue Nov 14, 2022 · 5 comments · Fixed by #209870
Closed

Linker error for atomic functions on aarch64-linux #201254

avdv opened this issue Nov 14, 2022 · 5 comments · Fixed by #209870
Labels
0.kind: bug 2.status: fix-in-staging This issue has been fixed in staging but is not yet in the channels or master

Comments

@avdv
Copy link
Member

avdv commented Nov 14, 2022

Describe the bug

When using some atomic functions on aarch64 linux fails during the linking phase when using the -z defs linker option (as observed while trying to compile bobcat on aarch64-linux):

/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: /build/test.o: in function `foo(int)':
zzfr0vqndwngbh5b2skgdgl90aqqrpwl-test.cpp:(.text+0x2c): undefined reference to `__aarch64_ldadd4_acq_rel'
collect2: error: ld returned 1 exit status

Steps To Reproduce

Steps to reproduce the behavior:

  1. save the following nix code into fail.nix
  2. on an aarch64 Linux system, run nix-build -I nixpkgs=PATH_TO_NIXPKGS fail.nix
{ pkgs ? import <nixpkgs> {} }:

pkgs.gcc10Stdenv.mkDerivation {
  name = "link failure on aarch64";
  version = "0.1";

  dontUnpack = true;

  src = pkgs.writeText "test.cpp" ''
      #include <atomic>

      int foo(int i) {
         std::atomic<int> f(i);

         return f.fetch_add(3);
      }
  '';

  buildPhase = ''
    g++ --std=c++20 -c $src -o $TMP/test.o
    g++ --std=c++20 -shared -Wl,-z,defs $TMP/test.o -o libtest.so 
  '';
}

Expected behavior

Linking should succeed.

Additional context

According to https://bugzilla.redhat.com/show_bug.cgi?id=1830472

On aarch64 -moutline-atomics has been turned on by default, and those symbols are solely in libgcc.a, not in libgcc_s.so.*.

Indeed libgcc_s.so from gcc-lib is a linker script:

$ cat /nix/store/q2nal9l25ff4zvd99l2ifjnpaw1ij21n-gcc-10.4.0-lib/lib/libgcc_s.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library.  */
GROUP ( libgcc_s.so.1 -lgcc )

Adding the -v flag to the invocation of g++ one can observe this call to collect2:

/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/libexec/gcc/aarch64-unknown-linux-gnu/10.4.0/collect2 \
-plugin \ 
/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/libexec/gcc/aarch64-unknown-linux-gnu/10.4.0/liblto_plugin.so \
-plugin-opt=/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/libexec/gcc/aarch64-unknown-linux-gnu/10.4.0/lto-wrapper \
-plugin-opt=-fresolution=/run/user/1000/ccYTLoAU.res \
-plugin-opt=-pass-through=-lgcc_s \
-plugin-opt=-pass-through=-lc \                                                                                                                                                  
-plugin-opt=-pass-through=-lgcc_s \                                                     
--eh-frame-hdr \
-shared \
-X \
-EL \
-maarch64linux \
-o \
libtest.so \
/nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/crti.o \
/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/lib/gcc/aarch64-unknown-linux-gnu/10.4.0/crtbeginS.o \
-L/nix/store/dwk83gbvz3dfzm7kk18lrphsbjn8qxq0-libmilter-8.15.2/lib \
-L/nix/store/bl0wv9qaj5ws3d1j9hl983qlpghx334f-libxcb-1.14/lib \
-L/nix/store/0gx2fkm4ndif74l9vaw2gi8ffm8vqp84-libX11-1.8.1/lib \
-L/nix/store/l95ixfjbvi6wj1il5a062b8sprcc1m3a-openssl-3.0.7/lib \
-L/nix/store/c51zf8aa3zna0qzjbyl2ac3j2q9nns27-ncurses-6.3-p20220507/lib \
-L/nix/store/cbdlrdy51iiszglwyq96q8dk3yg725hs-readline-6.3p08/lib \
-L/nix/store/94qlip4nc7c8kwvrrwav3sc008maxwhk-util-linux-2.38.1-lib/lib \
-L/nix/store/dwk83gbvz3dfzm7kk18lrphsbjn8qxq0-libmilter-8.15.2/lib \
-L/nix/store/bl0wv9qaj5ws3d1j9hl983qlpghx334f-libxcb-1.14/lib \
-L/nix/store/0gx2fkm4ndif74l9vaw2gi8ffm8vqp84-libX11-1.8.1/lib \
-L/nix/store/l95ixfjbvi6wj1il5a062b8sprcc1m3a-openssl-3.0.7/lib \
-L/nix/store/c51zf8aa3zna0qzjbyl2ac3j2q9nns27-ncurses-6.3-p20220507/lib \
-L/nix/store/cbdlrdy51iiszglwyq96q8dk3yg725hs-readline-6.3p08/lib \
-L/nix/store/94qlip4nc7c8kwvrrwav3sc008maxwhk-util-linux-2.38.1-lib/lib \
-L/nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib \
-L/nix/store/vqlcm9k500r8wsnam86rlan95yxnalxa-gcc-10.4.0-lib/lib \
-L/nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib \
-L/nix/store/vqlcm9k500r8wsnam86rlan95yxnalxa-gcc-10.4.0-lib/lib \
-L/nix/store/zk5rgbk1nrcy6r6fafamfqnnsc59z9n8-gcc-wrapper-10.4.0/bin \
-L/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/lib/gcc/aarch64-unknown-linux-gnu/10.4.0 \
-L/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/lib/gcc/aarch64-unknown-linux-gnu/10.4.0/../../../../lib64 \
-L/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/lib/gcc/aarch64-unknown-linux-gnu/10.4.0/../../.. \
-dynamic-linker=/nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/ld-linux-aarch64.so.1 \
-z \
defs \
test.o \
-rpath \
/nix/store/bspxp2j6ggc7v5an4s3qky6k6h6fw5gj-bobcat-5.10.01/lib \
-lstdc++ \
-lm \
-lgcc_s \
-lc \
-lgcc_s \
/nix/store/rvjj7j3hl1zm3xgy9vh8wck5r7cqp0py-gcc-10.4.0/lib/gcc/aarch64-unknown-linux-gnu/10.4.0/crtendS.o \
/nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/crtn.o \

Removing the glibc lib directory from the library path (ie. removing -L/nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib), makes this call successful.

The root cause is that glibc contains a libgcc_so.1 library in its lib folder:

$ file /nix/store/xl52bf8jch51zzh84rszxn0hpbxadll4-glibc-2.35-163/lib/libgcc_s.so
/nix/store/xl52bf8jch51zzh84rszxn0hpbxadll4-glibc-2.35-163/lib/libgcc_s.so: symbolic link to libgcc_s.so.1

This is found before the right one from gcc, and this breaks linking when it would just work if libgcc_so.1 would be the proper linker script.

Why does glibc contain the libgcc_s.so library?

Notify maintainers

@edolstra @Ma27

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 5.15.74-v8+, Debian GNU/Linux, 11 (bullseye), nobuild`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.1`
 - channels(claudio): `"nixpkgs"`
 - nixpkgs: `/home/claudio/.nix-defexpr/channels/nixpkgs`
@avdv
Copy link
Member Author

avdv commented Nov 15, 2022

BTW, this also makes building llvm fail on aarch64-linux:

...
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt10unique_ptrINS0_29MaterializationResponsibil
ityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x2e4): undefined reference to `__aarch64_ldadd8_acq_rel'                                                                 
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt10unique_ptrINS0_29MaterializationResponsibil
ityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x390): undefined reference to `__aarch64_ldadd4_acq_rel'                                                                 
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt10unique_ptrINS0_29MaterializationResponsibil
ityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x3a0): undefined reference to `__aarch64_ldadd4_acq_rel'                                                                 
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt10unique_ptrINS0_29MaterializationResponsibil
ityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x3b0): undefined reference to `__aarch64_ldadd4_acq_rel'                                                                 
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt10unique_ptrINS0_29MaterializationResponsibil
ityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x3d8): undefined reference to `__aarch64_ldadd4_acq_rel'                                                                 
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt10unique_ptrINS0_29MaterializationResponsibil
ityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x3e8): undefined reference to `__aarch64_ldadd4_acq_rel'                                                                 
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: ../../lib/libLLVMOrcJIT.a(Speculation.cpp.o):Speculation.cpp:(.text._ZN4llvm3orc18IRSpeculationLayer4emitESt
10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x3f8): more undefined references to `__aarch64_ldadd4_acq_rel' follow       
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: ../../lib/libLLVMOrcJIT.a(Speculation.cpp.o): in function `llvm::orc::Speculator::launchCompile(unsigned lon
g)':                                                                                                                                                                          
Speculation.cpp:(.text._ZN4llvm3orc10Speculator13launchCompileEm[_ZN4llvm3orc10Speculator13launchCompileEm]+0x164): undefined reference to `__aarch64_ldadd8_acq_rel'         
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc10Speculator13launchCompileEm[_ZN4llvm3orc10Speculator13launchCompileEm]+
0x200): undefined reference to `__aarch64_ldadd8_acq_rel'                                                                                                                     
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc10Speculator13launchCompileEm[_ZN4llvm3orc10Speculator13launchCompileEm]+
0x2d8): undefined reference to `__aarch64_ldadd8_acq_rel'                                                                                                                     
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc10Speculator13launchCompileEm[_ZN4llvm3orc10Speculator13launchCompileEm]+
0x37c): undefined reference to `__aarch64_ldadd8_acq_rel'                                                                                                                     
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: Speculation.cpp:(.text._ZN4llvm3orc10Speculator13launchCompileEm[_ZN4llvm3orc10Speculator13launchCompileEm]+
0x5b8): undefined reference to `__aarch64_ldadd8_acq_rel'                                                                                                                     
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: ../../lib/libLLVMOrcJIT.a(Speculation.cpp.o):Speculation.cpp:(.text._ZN4llvm3orc10Speculator13launchCompileE
m[_ZN4llvm3orc10Speculator13launchCompileEm]+0x680): more undefined references to `__aarch64_ldadd8_acq_rel' follow                                                           
/nix/store/1g5g7vzlcprmg1d6864fs5r4nf1h5b5j-binutils-2.39/bin/ld: ../../lib/libLLVMOrcJIT.a(ExecutorProcessControl.cpp.o): in function `void llvm::detail::UniqueFunctionBase<
void, llvm::orc::shared::WrapperFunctionResult>::CallImpl<llvm::orc::SelfExecutorProcessControl::jitDispatchViaWrapperFunctionManager(void*, void const*, char const*, unsigne
d long)::{lambda(llvm::orc::shared::WrapperFunctionResult)#1}>(void*, llvm::orc::shared::WrapperFunctionResult&)':                                                            
ExecutorProcessControl.cpp:(.text._ZN4llvm6detail18UniqueFunctionBaseIvJNS_3orc6shared21WrapperFunctionResultEEE8CallImplIZNS2_26SelfExecutorProcessControl36jitDispatchViaWra
pperFunctionManagerEPvPKvPKcmEUlS4_E_EEvS8_RS4_+0x110): undefined reference to `__aarch64_swp4_rel'

ehmry pushed a commit that referenced this issue Nov 30, 2022
@trofi
Copy link
Contributor

trofi commented Jan 3, 2023

It's a long-standing build problem in nixpkgs bootstrap process. We'll try to address it in #208412.

winterqt added a commit to winterqt/nixpkgs that referenced this issue Jan 4, 2023
This change switches to using GCC 11 by default on aarch64-linux, as well as passing `-lgcc` to the linker, per NixOS#201485.

See NixOS#201254 and NixOS#208412 for wider context on the issue.
flokli pushed a commit that referenced this issue Jan 18, 2023
This is required to workaround #201254
wegank added a commit to wegank/nixpkgs that referenced this issue Jan 26, 2023
wegank added a commit to wegank/nixpkgs that referenced this issue Jan 26, 2023
wegank added a commit that referenced this issue Jan 26, 2023
wegank pushed a commit to wegank/nixpkgs that referenced this issue Feb 1, 2023
#### Immediate Benefits

- Allow `gcc11` on `aarch64`
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more [static `lib{mpfr,mpc,gmp,isl}.a`
  hack](https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380)
- *Zero* additional `gcc` builds (stage1+stage2+stageCompare)
  - The `gcc` derivation builds `gcc` once instead of three times.
  - The libraries that are linked into the final `pkgs.gcc` (`mpfr`,
    `mpc`, `gmp`, `isl`, `glibc`) are built by
    `stdenv.__bootPkgs.gcc` rather than by the `bootstrapFiles`.  No
    more Frankenstein compiler!
  - stageCompare runs **concurrently** with (not in series with)
    with `stdenv`'s dependees.
- Many other `stdenv` hacks eliminated.
  - `gcc` and `clang` share the same codepath for more of
    `cc-wrapper`.
  - Makes the cross and native codepaths much more similar --
    another step towards "cross by default".

Note that *all* the changes in this PR are controlled by flags; no
old codepaths need to be removed until/if we're completely certain
that this is the right way to go.

#### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- There will be an "avalanche of simplification" when we set
  `enableGccExternalBootstrap=true` and run dead code elimination.
  It's really quite a huge amount of code that goes away.
  Native-gcc has its own special codepath in so many places, while
  cross-gcc and clang work the same way (and are much simpler).
- This should allow each of the libraries that ship with `gcc`
  (`lib{backtrace,atomic,cc1,decnumber,ffi,gomp,iberty,offloadatomic,quadmath,sanitizer,ssp,stdc++-v3,vtv}`)
  to be built in separate (one-liner) derivations which `inherit
  src;` from `gcc`.
  - Building `libstdc++-v3` in a separate derivation will eliminate
    a lot of accidental-reference-to-the-`bootstrapFiles` landmines.

#### Incorporates

- NixOS#209054
- NixOS#210004
- NixOS#36948 (unreverted)
- NixOS#210325
- NixOS#210118
- NixOS#210132
- NixOS#210109

#### Closes

- Closes NixOS#208412
- Closes NixOS#108111
- Closes NixOS#108305
- Closes NixOS#201254

#### Build history

- First successful builds (stage1/stage2):
  - powerpc64le-linux at 9c7e9ef
  - x86_64-linux at 9c7e9ef
  - aarch64-linux at 4d5bc7d

- First successful comparisons (stageCompare):
  - at 81949cf
  - [aarch64-linux][aarch64-compare-ofborg]
  - [x86\_64-linux][amd64-compare-ofborg]

#### Credits

This project was made possible by three important insights, none of
which were mine:

1. @Ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  External
   bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds.  Now that gcc is written in C++, it depends on
   `libstdc++`, builds a copy of `libstdc++`, and builds auxiliary
   products (like `libplugin`) which depend on `libstdc++`.  @trofi
   developed two important techniques for keeping this straight: the
   use of a [nonexistent sysroot] and moving the `bootstrapFiles`'
   `libstdc++` into a [versioned directory].  Without these two
   discoveries, external bootstrap would be impossible, because the
   final gcc would still have references to the `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: NixOS#210004
[versioned directory]: NixOS#209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253
[mentioned]: NixOS#210112 (comment)
[crisis]: NixOS#108305
[foreign]: NixOS#170857 (comment)
xanderio pushed a commit to xanderio/nixpkgs that referenced this issue Feb 13, 2023
winterqt added a commit to winterqt/nixpkgs that referenced this issue Feb 15, 2023
This change switches to using GCC 11 by default on aarch64-linux, as well as passing `-lgcc` to the linker, per NixOS#201485.

See NixOS#201254 and NixOS#208412 for wider context on the issue.

(cherry picked from commit 8442601)
winterqt pushed a commit to winterqt/nixpkgs that referenced this issue Feb 15, 2023
This is required to workaround NixOS#201254

(cherry picked from commit ba3db3e)
@bew
Copy link
Contributor

bew commented Apr 7, 2023

PR #209870 was marked to close this issue. Since that PR is now merged, this can be closed?

@lovesegfault
Copy link
Member

We should wait until it hits master, it's still in staging.

@bew
Copy link
Contributor

bew commented Apr 7, 2023

Oh you're right
On the PR, all other related issues/PR are already closed, so I thought if was just a mistake that this one isn't.

@Artturin Artturin added the 2.status: fix-in-staging This issue has been fixed in staging but is not yet in the channels or master label Apr 7, 2023
@K900 K900 closed this as completed Apr 11, 2023
@K900 K900 reopened this Apr 11, 2023
@vcunat vcunat closed this as completed in 7553d0f Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug 2.status: fix-in-staging This issue has been fixed in staging but is not yet in the channels or master
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants