Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build with pure LLVM toolchain on Linux #65051

Closed
markand opened this issue Oct 3, 2019 · 16 comments
Closed

Build with pure LLVM toolchain on Linux #65051

markand opened this issue Oct 3, 2019 · 16 comments
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-feature-request Category: A feature request, i.e: not implemented / a PR. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@markand
Copy link

markand commented Oct 3, 2019

Hello,

I'm creating a Linux distribution that does not use GCC and use LLVM/Clang and their libraries by default (libc++, libc++abi, compiler-rt and libunwind). No libstdc++ or gcc installed. The distribution is based on musl as well.

I'd like to build rust with that pure LLVM toolchain and since nobody did that before I'm having few issues and unable to find many resource on this topic. Alpine Linux also build with musl but they still use GCC instead.

What I've tried:

  1. I've downloaded the prebuilt musl image from the official binaries and installed in a temporary location.
wget https://static.rust-lang.org/dist/rust-1.37.0-x86_64-unknown-linux-musl.tar.gz
tar xf rust-1.37.0-x86_64-unknown-linux-musl.tar.gz
cd rust-1.37.0-x86_64-unknown-linux-musl && ./install --prefix= --destdir=/tmp/bootstrap-rust
  1. I've downloaded rust sources and configured the config.toml before build:
wget http://static.rust-lang.org/dist/rustc-1.37.0-src.tar.gz
tar xf rustc-1.37.0-src.tar.gz

The config.toml:

cat > rustc-1.37.0-src/config.toml <<EOF
[build]
rustc = "/tmp/bootstrap-rust/bin/rustc"
cargo = "/tmp/bootstrap-rust/bin/cargo"
python = "/bin/python"
build = 'x86_64-unknown-linux-musl'
host = ['x86_64-unknown-linux-musl']
target = ['x86_64-unknown-linux-musl']
vendor = true

[install]
prefix = ""

[rust]
channel = "stable"
rpath = false
codegen-tests = false
verbose-tests = false
default-linker = "clang"
llvm-libunwind = true

[dist]
src-tarball = false

[llvm]
static-libstdcpp = false
ninja = true
use-libcxx = true
link-shared = true

[target.x86_64-unknown-linux-musl]
cc = clang
cxx = clang++
linker = clang
llvm-config = "/bin/llvm-config"
crt-static = false
EOF
  1. Then trying to build using ./x.py build will abort not finding -lgcc_s (which unsurprisingly, does not exist).
error: linking with `clang` failed: exit code: 1
  |
  = note: "clang" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.0.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.1.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.10.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.11.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.12.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.13.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.14.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.15.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.2.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.3.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.4.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.5.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.6.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.7.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.8.rcgu.o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.build_script_build.qjbobtg7-cgu.9.rcgu.o" "-o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/build/proc-macro2-b821ea8110df742f/build_script_build-b821ea8110df742f.813shqno7rs2e7d.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/bootstrap/debug/deps" "-L" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-9a431e54f29d011f.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-541cee8c5b6f056a.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace-9b9843a21c950d24.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace_sys-f4f15dbb47e9d653.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-53359f68302a289b.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-dfa7f1d62fd88f75.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-c6c7bf94cfe1287c.rlib" "/tmp/rustcihQO53/libunwind-c02db0b699a2e957.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-ffad3faca04b6ecf.rlib" "/tmp/rustcihQO53/liblibc-63baa27af2c30c40.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-c55e6587bce48104.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-e2f6ada331b532fd.rlib" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-06ad1fc9bd858d00.rlib" "-Wl,--end-group" "/src/vanilla/lang/rust/bootstrap/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-2ec319ee584bfdc0.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc"
  = note: ld.lld: error: unable to find library -lgcc
          clang-8: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

error: Could not compile `syn`.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

error: Could not compile `proc-macro2`.

What should I do to force linking against libunwind, libc++abi rather than GCC equivalents?

@MaskRay
Copy link
Contributor

MaskRay commented Oct 3, 2019

Configure llvm with -DCLANG_DEFAULT_RTLIB=compiler-rt so that clang will link against libclang_rt.builtin*.a instead of -lgcc or -lgcc_s.

Alternatively, use link option -rtlib=compiler-rt.

Since clang 9, --unwindlib={none,libunwind} (-DCLANG_DEFAULT_UNWINDLIB) is available. This option can be used to avoid -lgcc_s or -lgcc_eh. I don't remember the status before r356508, but with --rtlib=compiler-rt, by default no unwind lib is linked in. If you use clang 8 and somehow some object files require a unwind lib, you may need explicit -Wl,-lunwind.

-DCLANG_DEFAULT_CXX_STDLIB=libc++ if you want to use libc++ by default. On most Linux distributions, the default is libstdc++. libc++abi.so is linked because libc++.so is a linker script (INPUT(libc++.so.1 -lc++abi)) that adds libc++abi.so. Your compiler driver is clang, not clang++, so C++ stdlib will not be linked.

@markand
Copy link
Author

markand commented Oct 3, 2019

Thanks for the answer,

Obviously, my LLVM and clang compiler are already built with those options and that's why by default clang++ will link to libc++, libunwind and use compiler-rt by default without any switches.

See:

/tmp # echo "int main() {}" > main.cpp

/tmp # clang++ -v main.cpp
Vanilla Linux clang version 8.0.1 (tags/RELEASE_801/final) (based on LLVM 8.0.1)
Target: x86_64-unknown-linux-musl
Thread model: posix
InstalledDir: /bin
 "/bin/clang-8" -cc1 -triple x86_64-unknown-linux-musl -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /lib/clang/8.0.1 -internal-isystem /bin/../include/c++/v1 -internal-isystem /usr/local/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /lib/clang/8.0.1/include -fdeprecated-macro -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 151 -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o /tmp/main-39d04f.o -x c++ main.cpp -faddrsig
clang -cc1 version 8.0.1 based upon LLVM 8.0.1 default target x86_64-linux-musl
ignoring nonexistent directory "/usr/local/include"
ignoring duplicate directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /bin/../include/c++/v1
 /include
 /lib/clang/8.0.1/include
End of search list.
 "/bin/ld.lld" -pie --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib/ld-musl-x86_64.so.1 -o a.out /bin/../lib/Scrt1.o /bin/../lib/crti.o /lib/clang/8.0.1/lib/linux/crtbeginS.o -L/bin/../lib -L/lib -L/usr/lib /tmp/main-39d04f.o -lc++ -lm /lib/clang/8.0.1/lib/linux/libclang_rt.builtins-x86_64.a -lc /lib/clang/8.0.1/lib/linux/libclang_rt.builtins-x86_64.a /lib/clang/8.0.1/lib/linux/crtendS.o /bin/../lib/crtn.o

/tmp # ldd a.out
        /lib/ld-musl-x86_64.so.1 (0x7f5979fe7000)
        libc++.so.1 => /lib/libc++.so.1 (0x7f5979f24000)
        libc++abi.so.1 => /lib/libc++abi.so.1 (0x7f5979ee7000)
        libunwind.so.1 => /lib/libunwind.so.1 (0x7f5979ed7000)
        libc.so => /lib/ld-musl-x86_64.so.1 (0x7f5979fe7000)

What I suspect is that -lgcc_s is probably "hardcoded" in the bootstrap tools and forced while re-building rustc with this bootstrap itself.

@mati865
Copy link
Contributor

mati865 commented Oct 3, 2019

What I suspect is that -lgcc_s is probably "hardcoded" in the bootstrap tools and forced while re-building rustc with this bootstrap itself.

Correct, IIRC gcc libs were hardcoded in multiple places in the past (don't really know about how it currently is).
You can replace gcc_s here with unwind but then your toolchain will require compatible libunwind installed in the system

#[link(name = "gcc_s", cfg(not(target_feature = "crt-static")))]

@markand
Copy link
Author

markand commented Oct 3, 2019

For the moment, I'm fooling the build process by creating a fake libgcc_s.so which simply contains:

INPUT(-lunwind)

It's a bit hackish but I went further into the build process now.

@csmoe csmoe added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Oct 3, 2019
@jonas-schievink jonas-schievink added A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-feature-request Category: A feature request, i.e: not implemented / a PR. labels Oct 3, 2019
@markand
Copy link
Author

markand commented Oct 3, 2019

I got further in the build with the fake libgcc_s.so file and the proposal to change gcc_s in lib.rs file from libunwind. But it looks like something is still not linked to LLVM's libunwind because after stage1 I get:


    Finished release [optimized] target(s) in 8m 14s
Assembling stage1 compiler (x86_64-unknown-linux-musl)
Building stage1 std artifacts (x86_64-unknown-linux-musl -> x86_64-unknown-linux-musl)
warning: rustc-1.37.0-src/src/tools/miri/Cargo.toml: the cargo feature `default-run` is now stable and is no longer necessary to be listed in the manifest
error: process didn't exit successfully: `rustc-1.37.0-src/build/bootstrap/debug/rustc -vV` (exit code: 127)
--- stderr
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_Backtrace: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_DeleteException: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_RaiseException: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetDataRelBase: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetIPInfo: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetLanguageSpecificData: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetRegionStart: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetTextRelBase: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_SetGR: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_SetIP: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_GetIP: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libstd-7ee15cbd53b7b254.so: _Unwind_FindEnclosingFunction: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_interface-f6372f68ca4c37fc.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_borrowck-7373c9341a309f71.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_mir-5c1c1819075ebd26.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_save_analysis-846e375bc77c7d8e.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_utils-65d9103bc435188d.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_metadata-64d587906dd5eba7.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_FindEnclosingFunction: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_GetIP: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Backtrace: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libtest-dc489fd1ade7cf99.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax-29c4688773e43a24.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_target-1fdfcbe13b408e9a.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_errors-bcce3650c698371e.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_pos-91929928d8d8df07.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libarena-40fb647db2c6c16f.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_data_structures-b3d1b53deed1cc21.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libgraphviz-c924c23f873259cc.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_cratesio_shim-451cc189886af978.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libserialize-ded14d9c4374ae5b.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_lint-cb9cab5a661e0fc4.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_traits-768f999b81f7c152.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_resolve-24af22f041ddbc99.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_privacy-d66f878948e250ee.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_typeck-0c55c94c2fe00bc3.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_plugin-aaf21d589d6f9a96.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_passes-230e54df788c1f06.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_ssa-ffbc4d353d0a6eba.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_incremental-93604673bdae5cca.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_allocator-61b49b916bd03a67.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_ext-894cd6a808d68b68.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libfmt_macros-2f4f0c96b22175f2.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_fs_util-eae473008a93e789.so: _Unwind_Resume: symbol not found
Error relocating rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libterm-143f9562b76da2bf.so: _Unwind_Resume: symbol not found

@mati865
Copy link
Contributor

mati865 commented Oct 3, 2019

@markand could you paste here output of ldd rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so?

@markand
Copy link
Author

markand commented Oct 3, 2019

There are lot of missing symbols.

# ldd build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so
        ldd (0x7f56a02c9000)
Error loading shared library librustc_interface-f6372f68ca4c37fc.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_borrowck-7373c9341a309f71.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_mir-5c1c1819075ebd26.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_save_analysis-846e375bc77c7d8e.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_codegen_utils-65d9103bc435188d.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_metadata-64d587906dd5eba7.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc-cb39e474659e15c8.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libtest-dc489fd1ade7cf99.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libsyntax-29c4688773e43a24.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_target-1fdfcbe13b408e9a.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_errors-bcce3650c698371e.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libsyntax_pos-91929928d8d8df07.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libarena-40fb647db2c6c16f.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_data_structures-b3d1b53deed1cc21.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libgraphviz-c924c23f873259cc.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library librustc_cratesio_shim-451cc189886af978.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libserialize-ded14d9c4374ae5b.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
Error loading shared library libstd-7ee15cbd53b7b254.so: No such file or directory (needed by build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so)
        libc.so => ldd (0x7f56a02c9000)
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _Unwind_Resume: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _ZN4core9panicking5panic17h8fe75b76845d641eE: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _ZN5alloc5alloc18handle_alloc_error17heeed78525e569dfeE: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _ZN5alloc7raw_vec17capacity_overflow17h7261c2dd34ad30e0E: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_alloc: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_alloc_zeroed: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_dealloc: symbol not found
Error relocating build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: __rust_realloc: symbol not found

It clearly isn't linked to libunwind as readelf shows.

# readelf -d build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so

Dynamic section at offset 0x12e530 contains 38 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [librustc_interface-f6372f68ca4c37fc.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_borrowck-7373c9341a309f71.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_mir-5c1c1819075ebd26.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_save_analysis-846e375bc77c7d8e.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_codegen_utils-65d9103bc435188d.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_metadata-64d587906dd5eba7.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc-cb39e474659e15c8.so]
 0x0000000000000001 (NEEDED)             Shared library: [libtest-dc489fd1ade7cf99.so]
 0x0000000000000001 (NEEDED)             Shared library: [libsyntax-29c4688773e43a24.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_target-1fdfcbe13b408e9a.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_errors-bcce3650c698371e.so]
 0x0000000000000001 (NEEDED)             Shared library: [libsyntax_pos-91929928d8d8df07.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarena-40fb647db2c6c16f.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_data_structures-b3d1b53deed1cc21.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgraphviz-c924c23f873259cc.so]
 0x0000000000000001 (NEEDED)             Shared library: [librustc_cratesio_shim-451cc189886af978.so]
 0x0000000000000001 (NEEDED)             Shared library: [libserialize-ded14d9c4374ae5b.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstd-7ee15cbd53b7b254.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW
 0x0000000000000007 (RELA)               0x15e68
 0x0000000000000008 (RELASZ)             42552 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          1085
 0x0000000000000017 (JMPREL)             0x204a0
 0x0000000000000002 (PLTRELSZ)           120 (bytes)
 0x0000000000000003 (PLTGOT)             0x12f6f8
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x238
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x6810
 0x000000000000000a (STRSZ)              63061 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x4b58
 0x0000000000000004 (HASH)               0x4fa8
 0x000000000000000c (INIT)               0x127328
 0x000000000000000d (FINI)               0x127330
 0x0000000000000000 (NULL)               0x0

@mati865
Copy link
Contributor

mati865 commented Oct 3, 2019

There are lot of missing symbols.

This is expected, at least when it comes to missing Rust libraries. You'd have to set LD_LIBRARY_PATH to avoid it but it doesn't matter right now.

It clearly isn't linked to libunwind as readelf shows.

I'd expect it to be linked at this point but I can be wrong.


IIRC libunwind doesn't provide _Unwind_Resume on it's own but relies on libgcc or compiler-rt. I assume that's why it's missing.
Can you confirm your libunwind or one of the libs it depends on provides _Unwind_Resume symbol?

I haven't been playing with non-trunk LLVM for a bit but I recall there were issues around libc++abi and libunwind. Hoping that @MaskRay will confirm or deny my guess.

@markand
Copy link
Author

markand commented Oct 3, 2019

Thanks again for the quick answer, libunwind.so has _Unwind_Resume symbol, yes.

$ nm /lib/libunwind.so.1.0 | grep Unwind_Resume
000000000000bca0 T _Unwind_Resume
000000000000c360 T _Unwind_Resume_or_Rethrow

@markand
Copy link
Author

markand commented Oct 16, 2019

Hello,

This is the list of built libraries that have undefined symbols to LLVM's libunwind, I've tried adding #[link(name = "unwind")] in their respective lib.rs but with no luck. Can I edit their Cargo.toml to force linking to it? I'm not yet confident with Rust tooling yet.

Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_target-1fdfcbe13b408e9a.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_metadata-64d587906dd5eba7.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_mir-5c1c1819075ebd26.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_data_structures-b3d1b53deed1cc21.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_interface-f6372f68ca4c37fc.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_typeck-0c55c94c2fe00bc3.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_FindEnclosingFunction: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_GetIP: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc-cb39e474659e15c8.so: _Unwind_Backtrace: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_utils-65d9103bc435188d.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_resolve-24af22f041ddbc99.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_privacy-d66f878948e250ee.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_codegen_ssa-ffbc4d353d0a6eba.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_plugin-aaf21d589d6f9a96.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_cratesio_shim-451cc189886af978.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libtest-dc489fd1ade7cf99.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_pos-91929928d8d8df07.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_passes-230e54df788c1f06.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_lint-cb9cab5a661e0fc4.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libterm-143f9562b76da2bf.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_driver-297236e98e3b11ce.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax_ext-894cd6a808d68b68.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_traits-768f999b81f7c152.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libserialize-ded14d9c4374ae5b.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_allocator-61b49b916bd03a67.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_borrowck-7373c9341a309f71.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_incremental-93604673bdae5cca.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libsyntax-29c4688773e43a24.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libgraphviz-c924c23f873259cc.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_fs_util-eae473008a93e789.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libfmt_macros-2f4f0c96b22175f2.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_save_analysis-846e375bc77c7d8e.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/libarena-40fb647db2c6c16f.so: _Unwind_Resume: symbol not found
Error relocating ./rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage1/lib/librustc_errors-bcce3650c698371e.so: _Unwind_Resume: symbol not found

@mati865
Copy link
Contributor

mati865 commented Oct 16, 2019

I have an theory.
Unless you patched Clang it won't link libunwind by default. In #65051 (comment) it worked because it was pulled by libcxx(abi?).
If that's correct then LDFLAGS="-Wl,-lunwind" mentioned by MaskRay should help.

@markand
Copy link
Author

markand commented Oct 16, 2019

Yes it is pulled by libc++.so. In fact we have the following input file (FreeBSD does something similar with compiler-rt).

# cat /lib/libc++.so
INPUT(libc++.so.1 -lc++abi -lunwind)

I have not seen the LDFLAGS proposal (maybe the message was edited)? Anyway sorry, I'll try immediately :-)

@markand
Copy link
Author

markand commented Oct 17, 2019

So the LDFLAGS didn't work and I've also rebuilt clang with the option to use libunwind by default with no luck. So I've rebuilt musl library to force inclusion of libunwind symbols into it. A bit hackish but I went further... Once I get a native rustc I'll revert my custom libc.so.

Now it fails because of an unmatched function definition.

TARGET = Some("x86_64-unknown-linux-musl")
OPT_LEVEL = Some("2")
HOST = Some("x86_64-unknown-linux-musl")
CXX_x86_64-unknown-linux-musl = Some("clang++")
CXXFLAGS_x86_64-unknown-linux-musl = Some("-ffunction-sections -fdata-sections -fPIC --target=x86_64-unknown-linux-musl")
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("false")
running: "clang++" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-musl" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-unknown-linux-musl" "-I//include" "-std=c++11" "-stdlib=libc++" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_INTERPRETER" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_MCJIT" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-o" "/src/vanilla/lang/rust/rustc-1.37.0-src/build/x86_64-unknown-linux-musl/stage0-codegen/x86_64-unknown-linux-musl/release/build/rustc_llvm-73aad4fac499ea8f/out/../rustllvm/PassWrapper.o" "-c" "../rustllvm/PassWrapper.cpp"
cargo:warning=../rustllvm/PassWrapper.cpp:917:3: error: no matching function for call to 'thinLTOResolvePrevailingInIndex'
cargo:warning=  thinLTOResolvePrevailingInIndex(Ret->Index, isPrevailing, recordNewLinkage);
cargo:warning=  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cargo:warning=//include/llvm/LTO/LTO.h:49:6: note: candidate function not viable: requires 4 arguments, but 3 were provided
cargo:warning=void thinLTOResolvePrevailingInIndex(
cargo:warning=     ^
cargo:warning=1 error generated.
exit code: 1

Is rust 1.37 compatible with llvm 9? I'll try with rust 1.38 instead.

@mati865
Copy link
Contributor

mati865 commented Oct 17, 2019

Is rust 1.37 compatible with llvm 9? I'll try with rust 1.38 instead.

LLVM 9 support was added in version 1.38.

@markand
Copy link
Author

markand commented Oct 28, 2019

I've finally built a version using a patched version of my musl. Now it builds fine itself with my native compiler.

Thanks for the help, though I think something should be changed to bootstrap Rust easier when “non traditional” toolchains are used but I have no knowledge in rust internals to help unfortunately.

@markand markand closed this as completed Oct 28, 2019
ekrich referenced this issue in scala-native/scala-native Aug 17, 2021
paulalesius added a commit to paulalesius/guix-rust that referenced this issue Oct 9, 2022
rust-lang/rust#65051
Signed-off-by: Paul Alesius <paul@unnservice.com>
@theoparis
Copy link

Any updates on this? I still have to follow the fix mentioned above which is obviously not a great solution.

echo "INPUT(-lunwind)" > /usr/lib/libgcc_s.so

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-feature-request Category: A feature request, i.e: not implemented / a PR. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

No branches or pull requests

6 participants