LLVM: broken sanitizer objects in installed archives

[nazavode]

Trying to use any kind of sanitizer (e.g.: -fsanitize=address) with the llvm@9.0.0 produced by spack@af65146ef69fcde10d53b2ee12f82a9c28361e23, I get the following error messages by the linker complaining about broken ELF sections:

asan_preinit.cc.o: unable to initialize decompress status for section .debug_info
libclang_rt.asan-x86_64.a: member libclang_rt.asan-x86_64.a(asan_preinit.cc.o) in archive is not an object

Steps to reproduce the issue

The issue has been observed on all the systems I'm currently using spack on (several flavours of rhel and ubuntu), but I'm able to consistently reproduce it using this Dockerfile and the following steps:

$ docker run -ti spack bash
spacker@c4670bdb5fee:~$ source /spack/share/spack/setup-env.sh
spacker@c4670bdb5fee:~$ spack load llvm
spacker@c4670bdb5fee:~$ echo "int main(){}" > main.c
spacker@c4670bdb5fee:~$ clang -fsanitize=address main.c 
/usr/bin/ld: /spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.4.0/llvm-9.0.0-qtuly6r6joyoqun54yier5k3sujcqkzb/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a(asan_preinit.cc.o): unable to initialize decompress status for section .debug_info
/usr/bin/ld: /spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.4.0/llvm-9.0.0-qtuly6r6joyoqun54yier5k3sujcqkzb/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a(asan_preinit.cc.o): unable to initialize decompress status for section .debug_info
/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.4.0/llvm-9.0.0-qtuly6r6joyoqun54yier5k3sujcqkzb/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a: member /spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.4.0/llvm-9.0.0-qtuly6r6joyoqun54yier5k3sujcqkzb/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a(asan_preinit.cc.o) in archive is not an object
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

Trying to strip the offending debug info section from objects leads to a similar issue with another ELF section (just a warning this time) but finally the linker fails with a quite suspicious __asan_version_mismatch_check_v8 missing symbol:

spacker@c4670bdb5fee:~$ strip /spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.4.0/llvm-9.0.0-qtuly6r6joyoqun54yier5k3sujcqkzb/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a
spacker@c4670bdb5fee:~$ clang -fsanitize=address main.c 
/usr/bin/ld: error in /spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.4.0/llvm-9.0.0-qtuly6r6joyoqun54yier5k3sujcqkzb/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a(asan_allocator.cc.o)(.eh_frame); no .eh_frame_hdr table will be created.

[omissis, lots of warnings]

/tmp/main-894c8c.o: In function `asan.module_ctor':
main.c:(.text+0x12): undefined reference to `__asan_init'
main.c:(.text+0x17): undefined reference to `__asan_version_mismatch_check_v8'
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

[alalazo]

Checked with @nazavode. The issue is that without loading the corresponding binutils:

$ spack load binutils

the clang compiler uses system's ld which is too old to understand the ELF format that has been generated:

[...]
/usr/bin/ld: /marconi/home/userinternal/fficarel/src/spack/opt/spack/linux-centos7-broadwell/gcc-9.2.0/llvm-9.0.0-ca65r7inizmvsl26kitwwnaseqi54tpw/lib/clang/9.0.0/lib/linux/libclang_rt.asan-x86_64.a(asan_rtl.cc.o): unrecognized relocation (0x2a) in section `.text._ZN6__asanL15AsanCheckFailedEPKciS1_yy'
/usr/bin/ld: final link failed: Bad value

$ /usr/bin/ld --version
GNU ld version 2.23.52.0.1-55.el7 20130226

We need to check if there's a way to configure llvm so that the default linker is the one of the binutils that are in the DAG.

[alalazo]

For later reference, here's docs on how to customize clang's behavior with configuration files.

[alalazo]

Trying to add the -DLLVM_ENABLE_LLD:BOOL=ON flag to cmake configure. Reference here
@ax3l

[sethrj]

I just hit this myself -- do we need a +binutils variant for LLVM like we have with GCC?