`check-compiler-rt` fails with various errors when using the runtimes build on windows

[zmodem]

At LLVM HEAD from this morning (6909014) in a Visual Studio 2019 x86 development shell:

(The same commands fail in an x64 shell, but in slightly different ways.)

Configuring:

C:\src\llvm-project\build.runtimes>cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS=clang -DLLVM_ENABLE_RUNTIMES=compiler-rt ../llvm
-- The C compiler identification is MSVC 19.29.30145.0
-- The CXX compiler identification is MSVC 19.29.30145.0
[...]
-- Build files have been written to: C:/src/llvm-project/build.runtimes

Let's test it:

C:\src\llvm-project\build.runtimes>ninja check-compiler-rt
[...]
[3728/3739] Performing configure step for 'builtins'
-- The C compiler identification is Clang 16.0.0 with MSVC-like command-line
[...]
-- Build files have been written to: C:/src/llvm-project/build.runtimes/runtimes/builtins-bins
[3729/3739] Performing build step for 'builtins'
[128/128] Linking C static library C:\src\llvm-project\bui...lang\16.0.0\lib\i386-pc-windows-msvc\clang_rt.builtins.lib
[3730/3739] No install step for 'builtins'
[3737/3739] Performing configure step for 'runtimes'
-- The C compiler identification is Clang 16.0.0 with MSVC-like command-line
[...]
[...]
-- Build files have been written to: C:/src/llvm-project/build.runtimes/runtimes/runtimes-bins
[3738/3739] cmd.exe /C "cd /D C:\src\llvm-project\build.ru...untimes-bins/ --target check-compiler-rt --config Release"
[32/279] Building CXX object compiler-rt\lib\interception\...keFiles\RTInterception.i386.dir\interception_linux.cpp.obj
clang-cl: warning: argument unused during compilation: '/experimental:external /external:W0 /external:anglebrackets' [-Wunused-command-line-argument]

(That looks like it's baked three command-line arguments into one which seems like a bug.)

[...]
[101/279] Generating FuzzedDataProviderTestObjects.FuzzedDataProviderUnittest.cpp.i386.o
FAILED: compiler-rt/lib/fuzzer/tests/FuzzedDataProviderTestObjects.FuzzedDataProviderUnittest.cpp.i386.o
cmd.exe /C "cd /D C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\fuzzer\tests && C:\src\llvm-project\build.runtimes\.\bin\clang.exe -DWIN32 -D_WINDOWS -Wno-unknown-warning-option -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -Wno-covered-switch-default -Wno-suggest-override -DGTEST_NO_LLVM_SUPPORT=1 -DGTEST_HAS_RTTI=0 -IC:/src/llvm-project/llvm/utils/unittest/googletest/include -IC:/src/llvm-project/llvm/utils/unittest/googletest -Wno-deprecated-declarations -IC:/src/llvm-project/compiler-rt/lib/fuzzer -fno-rtti -O2 -c -o FuzzedDataProviderTestObjects.FuzzedDataProviderUnittest.cpp.i386.o C:/src/llvm-project/compiler-rt/lib/fuzzer/tests/FuzzedDataProviderUnittest.cpp"
C:/src/llvm-project/compiler-rt/lib/fuzzer/tests/FuzzedDataProviderUnittest.cpp:9:10: fatal error: 'fuzzer/FuzzedDataProvider.h' file not found
#include <fuzzer/FuzzedDataProvider.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[...]
ninja: build stopped: subcommand failed.
FAILED: runtimes/CMakeFiles/check-compiler-rt
cmd.exe /C "cd /D C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins && "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" --build C:/src/llvm-project/build.runtimes/runtimes/runtimes-bins/ --target check-compiler-rt --config Release"
ninja: build stopped: subcommand failed.

That's not good.
Maybe it's a missing build dependency. Let's try building "all" before running the tests.

C:\src\llvm-project\build.runtimes>ninja
[890/1018] Performing build step for 'builtins'
ninja: no work to do.
[898/1018] No install step for 'builtins'
[908/1018] Performing configure step for 'runtimes'
[...]
-- Build files have been written to: C:/src/llvm-project/build.runtimes/runtimes/runtimes-bins
[992/1018] Performing build step for 'runtimes'
[...]
[210/210] Linking CXX shared library C:\src\llvm-project\b...\16.0.0\lib\i386-pc-windows-msvc\clang_rt.asan_dynamic.dll
[1012/1018] No install step for 'runtimes'
[1018/1018] Linking CXX executable bin\opt.exe

Running the tests again:

C:\src\llvm-project\build.runtimes>ninja check-compiler-rt
[0/5] Performing build step for 'builtins'
ninja: no work to do.
[1/5] No install step for 'builtins'
[3/5] Performing configure step for 'runtimes'
[...]
-- Build files have been written to: C:/src/llvm-project/build.runtimes/runtimes/runtimes-bins
[4/5] cmd.exe /C "cd /D C:\src\llvm-project\build.runtimes...untimes-bins/ --target check-compiler-rt --config Release"
[66/103] Generating SANITIZER_TEST_OBJECTS.sanitizer_bitvector_test.cpp.i386.o
C:/src/llvm-project/compiler-rt/lib/sanitizer_common/tests/sanitizer_bitvector_test.cpp:74:29: warning: format specifies type 'size_t' (aka 'unsigned int') but the argument has type 'value_type' (aka 'unsigned long') [-Wformat]
    fprintf(stderr, "%zu ", *it);
                     ~~~    ^~~
                     %lu
1 warning generated.
[85/103] Generating Fuzzer-i386-Test.exe
   Creating library C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\fuzzer\tests\.\Fuzzer-i386-Test.lib and object C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\fuzzer\tests\.\Fuzzer-i386-Test.exp
[88/103] Generating Sanitizer-i386-Test.exe
   Creating library C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\sanitizer_common\tests\.\Sanitizer-i386-Test.lib and object C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\sanitizer_common\tests\.\Sanitizer-i386-Test.exp
[90/103] Generating I386WindowsConfig/Asan-i386-inline-Noinst-Test.exe
   Creating library C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\asan\tests\I386WindowsConfig\.\Asan-i386-inline-Noinst-Test.lib and object C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\asan\tests\I386WindowsConfig\.\Asan-i386-inline-Noinst-Test.exp
[94/103] Generating I386WindowsConfig/Asan-i386-calls-Noinst-Test.exe
   Creating library C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\asan\tests\I386WindowsConfig\.\Asan-i386-calls-Noinst-Test.lib and object C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\asan\tests\I386WindowsConfig\.\Asan-i386-calls-Noinst-Test.exp
[96/103] Generating I386WindowsConfig/Asan-i386-calls-Test.exe
FAILED: compiler-rt/lib/asan/tests/I386WindowsConfig/Asan-i386-calls-Test.exe
cmd.exe /C "cd /D C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins\compiler-rt\lib\asan\tests && C:\src\llvm-project\build.runtimes\.\bin\clang++.exe ASAN_INST_TEST_OBJECTS.gtest-all.cc.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_globals_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_interface_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_internal_interface_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_oob_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_mem_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_str_test.cpp.i386-calls.o ASAN_INST_TEST_OBJECTS.asan_test_main.cpp.i386-calls.o -o C:/src/llvm-project/build.runtimes/runtimes/runtimes-bins/compiler-rt/lib/asan/tests/I386WindowsConfig/./Asan-i386-calls-Test.exe -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -g -fsanitize=address"
LINK : fatal error LNK1181: cannot open input file 'C:\src\llvm-project\build.runtimes\lib\clang\16.0.0\lib\windows\clang_rt.asan-i386.lib'
clang++: error: linker command failed with exit code 1181 (use -v to see invocation)
[99/103] Generating ASAN_DYNAMIC_TEST_OBJECTS.gtest-all.cc.i386-calls.o
ninja: build stopped: subcommand failed.
FAILED: runtimes/CMakeFiles/check-compiler-rt
cmd.exe /C "cd /D C:\src\llvm-project\build.runtimes\runtimes\runtimes-bins && "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" --build C:/src/llvm-project/build.runtimes/runtimes/runtimes-bins/ --target check-compiler-rt --config Release"
ninja: build stopped: subcommand failed.

It doesn't seem to be anywhere:

C:\src\llvm-project\build.runtimes>dir clang_rt.asan-i386.lib /s /b
File Not Found

The build files doesn't seem to know about it either:

C:\src\llvm-project\build.runtimes>ninja lib\clang\16.0.0\lib\windows\clang_rt.asan-i386.lib
ninja: error: unknown target 'lib\clang\16.0.0\lib\windows\clang_rt.asan-i386.lib'

C:\src\llvm-project\build.runtimes>ninja clang_rt.asan-i386.lib
ninja: error: unknown target 'clang_rt.asan-i386.lib'

[davidmrdavid]

@zmodem - did you ever get around these issues?

I'm also unable to get a working build of compiler-rt, on Windows, using the runtimes build mode. The clang_rt.asan* dlls are missing, and possibly other bits as well.

Do you know if the default windows clang distribution is building compiler-rt / asan in the "enable projects" mode instead?

[zmodem]

I don't recall what/when it changed, but check-compiler-rt seems to work for me with the cmake config above these days.

(The warnings about unused /experimental:... flags are still there though.)

Building at the llvmorg-21.1.0 tag, I get the dll in lib\clang\21\lib\windows\clang_rt.asan_dynamic-x86_64.dll

After last year, only the "dynamic" runtime exists. I think the 32-bit runtime may have been dropped at some point too, at least we don't use it.

Do you know if the default windows clang distribution is building compiler-rt / asan in the "enable projects" mode instead?

Yes, it still uses the "enable projects" mode:

llvm-project/llvm/utils/release/build_llvm_release.bat

Line 168 in 52dd4b9

-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;compiler-rt;lldb;openmp"

For Chromium's toolchain we do use the "runtimes build" though: https://source.chromium.org/chromium/chromium/src/+/main:tools/clang/scripts/build.py;l=903-917;drc=81b7e11f66c8848e5353fcbc0ad50f16cc00470a

The "runtimes build" is also used in llvm's presubmit testing after 62232e8

[davidmrdavid]

Thanks for the pointers, I'll do some experiments without "enable projects" and follow up, perhaps in my own ticket, if issues continue. Thanks again!

[davidmrdavid]

Btw I managed to get this working - so the msvc folks will have an easier time testing properly with clang moving forward.

I also see the warnings, I'll stare at them for longer in case I can see a way to fix them.