LLVM 18.1.8 breaks paths for sanitizers in connection with Rust

[kamulos]

My project is using Rust together with clang and the address sanitizer. Both the build orchestration and the linker invocations are done by cargo which is based on LLVM 18.1.7.

I install llvm via apt.llvm.org and when I do that now I get version 18.1.8, I found no way to pin it to a specific LLVM version. To me it seems that the paths of the static ASAN libraries have changed with the newest patch release, I get the following error:

ld: error: cannot open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan_static-x86_64.a: No such file or directory
ld: error: cannot open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a: No such file or directory

But in reality the libraries are now in /usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu under different names.

Such a breaking change is surprising for a patch release, especially if there is no real possibility to install an older version.

[azat]

Such a breaking change is surprising for a patch release, especially if there is no real possibility to install an older version

Indeed, but the problem here is not the llvm repo itself, but the debian manifests.

LLVM_ENABLE_PER_TARGET_RUNTIME_DIR had been enabled here - https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/9ca35f30383d89e4fdd45d15e0eb82c832df4b8c

And this leads to this issue

[alesapin]

I really recommend to try https://github.com/goreleaser/nfpm for linux distros packages management to avoid such surprises in future.

[azat]

And note that it is broken not only for rust, but for custom --target as well:

broken with --target:

root@2d2f47793e85:/workdir# clang-18 --target=x86_64-linux-gnu -### -fsanitize=address -nodefaultlibs -xc -  |& grep -o /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a
/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a
root@2d2f47793e85:/workdir# ls /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a
ls: cannot access '/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a': No such file or directory

Does works without:

root@2d2f47793e85:/workdir# clang-18 -### -fsanitize=address -nodefaultlibs -xc -  |& grep /usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a -o | head -1
/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a
root@2d2f47793e85:/workdir# ls /usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a
/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a

root@2d2f47793e85:/workdir# clang-18 -print-file-name=libclang_rt.asan-x86_64.a
libclang_rt.asan-x86_64.a
root@2d2f47793e85:/workdir# clang-18 -print-file-name=libclang_rt.asan.a
/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a

[azat]

FWIW I cannot reproduce this issue by building clang-19 from sources:

cmake -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi;libunwind;compiler-rt' -DLLVM_ENABLE_RTTI=ON -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_BUILD_LLVM_DYLIB=ON -DCMAKE_BUILD_TYPE=Release -G Ninja -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON ../llvm

So likely it had been fixed already

[azat]

I've compiled 18.1.8 from sources and it works correctly (using the same command above).

Then I take a look at debian packages one more time, and the problem there I guess, the libraries placed into wrong location, it uses x86_64-pc-linux-gnu instead x86_64-unknown-linux-gnu (see also rust-lang/compiler-team#441)

# clang-18 --target=x86_64-linux-gnu -### -fsanitize=address -nodefaultlibs -xc -
Ubuntu clang version 18.1.8 (++20240615103753+3b5b5c1ec4a3-1~exp1~20240615223858.136)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-18/bin/clang" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "-mrelax-all" "-dumpdir" "a-" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "-" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fdebug-compilation-dir=/workdir" "-fcoverage-compilation-dir=/workdir" "-resource-dir" "/usr/lib/llvm-18/lib/clang/18" "-internal-isystem" "/usr/lib/llvm-18/lib/clang/18/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-ferror-limit" "19" "-fmessage-length=295" "-fsanitize=address" "-fsanitize-system-ignorelist=/usr/lib/llvm-18/lib/clang/18/share/asan_ignorelist.txt" "-fno-sanitize-memory-param-retval" "-fsanitize-address-use-after-scope" "-fsanitize-address-globals-dead-stripping" "-fno-assume-sane-operator-new" "-fgnuc-version=4.2.1" "-fskip-odr-check-in-gmf" "-fcolor-diagnostics" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/tmp/--068ca9.o" "-x" "c" "-"
 "/usr/bin/x86_64-linux-gnu-ld" "-z" "relro" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "/usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtbeginS.o" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/11" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../lib64" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib/../lib64" "-L/lib" "-L/usr/lib" "--whole-archive" "/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan_static-x86_64.a" "--no-whole-archive" "--whole-archive" "/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a" "--no-whole-archive" "--export-dynamic" "/tmp/--068ca9.o" "/usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"

# apply workaround
mv /usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu /usr/lib/llvm-18/lib/clang/18/lib/x86_64-unknown-linux-gnu

# clang-18 --target=x86_64-linux-gnu -### -fsanitize=address -nodefaultlibs -xc -
Ubuntu clang version 18.1.8 (++20240615103753+3b5b5c1ec4a3-1~exp1~20240615223858.136)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
 "/usr/lib/llvm-18/bin/clang" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "-mrelax-all" "-dumpdir" "a-" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "-" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-fdebug-compilation-dir=/workdir" "-fcoverage-compilation-dir=/workdir" "-resource-dir" "/usr/lib/llvm-18/lib/clang/18" "-internal-isystem" "/usr/lib/llvm-18/lib/clang/18/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include" "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-ferror-limit" "19" "-fmessage-length=295" "-fsanitize=address" "-fsanitize-system-ignorelist=/usr/lib/llvm-18/lib/clang/18/share/asan_ignorelist.txt" "-fno-sanitize-memory-param-retval" "-fsanitize-address-use-after-scope" "-fsanitize-address-globals-dead-stripping" "-fno-assume-sane-operator-new" "-fgnuc-version=4.2.1" "-fskip-odr-check-in-gmf" "-fcolor-diagnostics" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/tmp/--bb3a52.o" "-x" "c" "-"
 "/usr/bin/x86_64-linux-gnu-ld" "-z" "relro" "--hash-style=gnu" "--build-id" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "/usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtbeginS.o" "-L/usr/lib/llvm-18/lib/clang/18/lib/x86_64-unknown-linux-gnu" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/11" "-L/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../lib64" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib/../lib64" "-L/lib" "-L/usr/lib" "--whole-archive" "/usr/lib/llvm-18/lib/clang/18/lib/x86_64-unknown-linux-gnu/libclang_rt.asan_static.a" "--no-whole-archive" "--whole-archive" "/usr/lib/llvm-18/lib/clang/18/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a" "--no-whole-archive" "--dynamic-list=/usr/lib/llvm-18/lib/clang/18/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a.syms" "/tmp/--bb3a52.o" "/usr/bin/../lib/gcc/x86_64-linux-gnu/11/crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"

So I've tried to take a look at debian manifests (https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/), 18 branch

And found that there are two possible input for arch:

$ clang -dumpmachine
x86_64-pc-linux-gnu
$ grep LLVM_DEFAULT_TARGET_TRIPLE CMakeCache.txt|cut -d= -f2
x86_64-unknown-linux-gnu

So apparently it uses clang -dumpmachine incorrectly somewhere.

I also found one log for deb packages build of llvm-18 -
https://buildd.debian.org/status/fetch.php?pkg=llvm-toolchain-18&arch=arm64&ver=1%3a18.1.8-1%2bb1&stamp=1719628542&raw=0 (from this page - https://tracker.debian.org/pkg/llvm-toolchain-18)

But it is different, since LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is OFF there

[azat]

It could be fixed by adding a fallback code into ToolChain::getTargetSubDirPath, but I guess the packages should be fixed instead, though it is unclear where to submit issues

[azat]

OK, I've found the logs for this package - https://llvm-jenkins.debian.net/view/Ubuntu%20Jammy/job/llvm-toolchain-jammy-18-binaries/architecture=amd64,distribution=jammy,label=amd64/

And the root cause of this issue is likely some of this options:

-DCMAKE_C_COMPILER_TARGET=x86_64-pc-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-pc-linux-gnu
-DLLVM_HOST_TRIPLE=x86_64-pc-linux-gnu
-DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-pc-linux-gnu

What do you think about actually adding a fallback inside clang?

[azat]

What do you think about actually adding a fallback inside clang?

Something like this - #97802