Don't dynamically link LLVM tools unless rustc is too

[Mark-Simulacrum]

This PR initially tried to support link-shared on all of our target platforms (other than Windows), but ran into a number of difficulties:

• LLVM doesn't really support a shared link on macOS (llvm-config runs into problems with the version suffix)
• LLVM doesn't seem to support a shared link when cross-compiling (the libLLVM.so ends up empty and symbols are not found)

So, this PR has now been revised such that we don't attempt to dynamically link LLVM tools (even if that would, otherwise, be supported) on targets where LLVM is statically linked to rustc. Currently that's basically everything except for x86_64-unknown-linux-gnu (where we dynamically link to avoid rerunning ThinLTO in each stage).

Follow-up to #76708.
Fixes #76698.

[rust-highfive]

r? @kennytm

(rust_highfive has picked a reviewer for you, use r? to override)

[Mark-Simulacrum]

@bors rollup=never

I suspect the original PR had essentially no effect as a result of this "bug" so we'll want the no-rolling policy here too.

[jyn514]

@bors r+

[bors]

📌 Commit 7a8fedf313e05e43208144a35bde20861a59f441 has been approved by jyn514

[bors]

⌛ Testing commit 7a8fedf313e05e43208144a35bde20861a59f441 with merge 50a9d19a155b90d56767ece8b126a9477b703091...

[bors]

💔 Test failed - checks-azure

[jyn514]

running: "ar" "crs" "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out/libllvm-wrapper.a" "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out/llvm-wrapper/PassWrapper.o" "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out/llvm-wrapper/RustWrapper.o" "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out/llvm-wrapper/ArchiveWrapper.o" "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out/llvm-wrapper/CoverageMappingWrapper.o" "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out/llvm-wrapper/Linker.o"
  exit code: 0
  cargo:rustc-link-lib=static=llvm-wrapper
  cargo:rustc-link-search=native=/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/build/rustc_llvm-7998fe2b8433c8ac/out
  cargo:rerun-if-env-changed=LLVM_LINK_SHARED

  --- stderr
  llvm-config: error: libLLVM-11-rust-1.48.0-nightly.dylib is missing
  thread 'main' panicked at 'command did not execute successfully: "/Users/runner/work/1/s/build/x86_64-apple-darwin/llvm/build/bin/llvm-config" "--link-shared" "--libs" "--system-libs" "aarch64" "arm" "asmparser" "avr" "bitreader" "bitwriter" "coverage" "hexagon" "instrumentation" "ipo" "linker" "lto" "mips" "msp430" "nvptx" "powerpc" "riscv" "sparc" "systemz" "webassembly" "x86"
  expected success, got: exit code: 1', src/build_helper/lib.rs:139:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
[RUSTC-TIMING] indexmap test:false 0.940
[RUSTC-TIMING] serde test:false 9.905
error: build failed
command did not execute successfully: "/Users/runner/work/1/s/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "--target" "x86_64-apple-darwin" "-Zbinary-dep-depinfo" "-j" "4" "--release" "--locked" "--color" "always" "--features" "jemalloc llvm max_level_info" "--manifest-path" "/Users/runner/work/1/s/compiler/rustc/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101
failed to run: /Users/runner/work/1/s/build/bootstrap/debug/bootstrap dist

[Mark-Simulacrum]

r? @alexcrichton -- for context, this is "just" a continuation of #76708 which essentially is just a re-implementation of that PR. The first commit fixes the enabling of link-shared and the second fixes builds with link-shared on macOS.

My impression is that before we landed #76349 (e.g., on current stable) rustc_driver and rustc in general statically linked to LLVM, but we still shipped a libLLVM.dylib, and tools (like rust-lld) linked to that. After #76349 changed the logic to say "if llvm.link-shared is false, don't put the LLVM dylib in the sysroot," this broke tools like rust-lld on macOS. This PR, then, enables shared linking of LLVM to the rustc binary/rustc_driver, and fixes the LLVM build on macOS to support shared linking.

This provides us with two benefits:

• Likely (though I haven't tested) the "download-ci-llvm" is now possible on macOS since we now support linking to a llvm dylib
• We avoid shipping LLVM in both librustc_driver and libLLVM.dylib

We could explore an alternative which tries to statically link llvm tools, like we do on Windows, but we'd then likely not be able to support the CI-built LLVM locally, which seems unfortunate.

This PR is not something I'm personally a fan of -- I guess the "proper" thing to do is probably to try and talk to LLVM build system maintainers to try and work out why the macOS shared linking story is so oddly constructed with various parts of LLVM creating and doing different things (see commit message and comments for details). I tried to do some git history searching and googling but wasn't able to find anything that hinted at answers.

[bors]

📌 Commit 546fb5189f95393316cb84552cf7d8c522145b28 has been approved by alexcrichton

[bors]

⌛ Testing commit 546fb5189f95393316cb84552cf7d8c522145b28 with merge 384fb80f1c96953f5e787a17d20fe5695416a020...

[bors]

💔 Test failed - checks-actions

[jyn514]

[2312/2616] Linking CXX executable bin/llvm-itanium-demangle-fuzzer
FAILED: : && /x-tools/x86_64-unknown-netbsd/bin/x86_64--netbsd-g++-sysroot   -ffunction-sections -fdata-sections -fPIC -m64 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG  -Wl,-allow-shlib-undefined    -Wl,-rpath-link,/checkout/obj/build/x86_64-unknown-netbsd/llvm/build/./lib  -Wl,-O3 -Wl,--gc-sections tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.o  -o bin/llvm-ar  lib/libLLVM-11-rust-1.48.0-nightly.so -lpthread -Wl,-rpath,"\$ORIGIN/../lib" && :
tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.o: In function `llvm::detail::provider_format_adapter<std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&>::format(llvm::raw_ostream&, llvm::StringRef)':
llvm-ar.cpp:(.text._ZN4llvm6detail23provider_format_adapterIRNSt6chrono10time_pointINS2_3_V212system_clockENS2_8durationIlSt5ratioILl1ELl1000000000EEEEEEE6formatERNS_11raw_ostreamENS_9StringRefE[_ZN4llvm6detail23provider_format_adapterIRNSt6chrono10time_pointINS2_3_V212system_clockENS2_8durationIlSt5ratioILl1ELl1000000000EEEEEEE6formatERNS_11raw_ostreamENS_9StringRefE]+0x5): undefined reference to `llvm::format_provider<std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, void>::format(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&, llvm::raw_ostream&, llvm::StringRef)'
tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.o: In function `llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>::~BumpPtrAllocatorImpl()':
llvm-ar.cpp:(.text._ZN4llvm20BumpPtrAllocatorImplINS_15MallocAllocatorELm4096ELm4096ELm128EED2Ev[_ZN4llvm20BumpPtrAllocatorImplINS_15MallocAllocatorELm4096ELm4096ELm128EED5Ev]+0x2d): undefined reference to `llvm::deallocate_buffer(void*, unsigned long, unsigned long)'
llvm-ar.cpp:(.text._ZN4llvm20BumpPtrAllocatorImplINS_15MallocAllocatorELm4096ELm4096ELm128EED2Ev[_ZN4llvm20BumpPtrAllocatorImplINS_15MallocAllocatorELm4096ELm4096ELm128EED5Ev]+0x7d): undefined reference to `llvm::deallocate_buffer(void*, unsigned long, unsigned long)'
tools/llvm-ar/CMakeFiles/llvm-ar.dir/llvm-ar.cpp.o: In function `getDefaultForHost()':

and a whole slew of other linker errors following.

[Mark-Simulacrum]

r? @alexcrichton since the latest update represents a pretty major change to this PR

I've given up on trying to make link-shared work well across the board -- see PR description and the last few CI failures -- it seems link LLVM is just not ready for a shared link on macOS or when cross-compiling, which excludes almost all of our builders :)

That said, this took the approach of disabling the shared LLVM link if we're not doing so with rustc, but it could also take the approach of editing dist.rs to distribute libLLVM.dylib (or the platform equivalent) whenever we're dynamically linking LLVM tools. I think I personally prefer to statically link -- we're likely to have slightly smaller aritfacts as a result and overall statically linking things is just much simpler. But I'm happy to do whichever, though I've not tested.

I'm currently building this in docker and on a mac, but given that this approach is much simpler, I'm pretty confident in it working out.

[Mark-Simulacrum]

I've confirmed that the latest commit seems to fix things on macOS (link-shared still errors, now explicitly rather than opaquely).

[takkuumi]

@Mark-Simulacrum Is there a temporary solution? I tried to compile with the stable version and the same problem occurred. This problem has been bothering me for a week, our team cannot proceed to the next step.

[Mark-Simulacrum]

Stable and beta branches should not be affected by this bug: if you're seeing a bug there, please file a new issue with a description of how to reproduce what you're seeing and cc me.

[takkuumi]

@Mark-Simulacrum sorry, my bad. i hadn't restart the terminal. now i'm using the stable version. but some api just run on nightly version.

[alexcrichton]

@bors: r+

Longer-term I do think it's best to get back to a point where we're consistently doing the same thing for LLVM across all our platforms. It's been the case for quite some time that LLVM is built with ThinLTO on Linux but nowhere else. Differing in linkage is also unfortunate across the major platforms too. It seems good though to get things working again at least...

[bors]

📌 Commit 389b7ff has been approved by alexcrichton

[bors]

⌛ Testing commit 389b7ff with merge 0da5800...

[bors]

☀️ Test successful - checks-actions, checks-azure
Approved by: alexcrichton
Pushing 0da5800 to master...