Reland [asan][windows] Eliminate the static asan runtime on windows

[barcharcraz]

This reapplies 8fa66c6 ([asan][windows] Eliminate the static asan runtime on windows) for a second time.

That PR bounced off the tests because it caused failures in the other sanitizer runtimes, these have been fixed by only building interception, sanitizer_common, and asan with /MD, and continuing to build the rest of the runtimes with /MT. This does mean that any usage of the static ubsan/fuzzer/etc runtimes will mean you're mixing different runtime library linkages in the same app, the interception, sanitizer_common, and asan runtimes are designed for this, however it does result in some linker warnings.

Additionally, it turns out when building in release-mode with LLVM_ENABLE_PDBs the build system forced /OPT:ICF. This totally breaks asan's "new" method of doing "weak" functions on windows, and so /OPT:NOICF was explicitly added to asan's link flags.

[llvm-ci]

LLVM Buildbot has detected a new failure on builder bolt-aarch64-ubuntu-clang running on bolt-worker-aarch64 while building clang,compiler-rt at step 5 "build-clang-bolt".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/128/builds/609

Here is the relevant piece of the build log for the reference

Step 5 (build-clang-bolt) failure: build (failure)
...
BOLT-INFO: padding code to 0x16400000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x190d88d4
BOLT-INFO: clear procedure is 0x190d74d0
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0x190d8840
BOLT-INFO: setting __bolt_runtime_fini to 0x190d88d4
BOLT-INFO: setting __hot_start to 0x8c00000
BOLT-INFO: setting __hot_end to 0x162f67d4
558.554 [2/1/3040] Generating BOLT profile for Clang
-- Testing: 1 tests, 1 workers --
FAIL: Clang Perf Training :: cxx/hello_world.cpp (1 of 1)
******************** TEST 'Clang Perf Training :: cxx/hello_world.cpp' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/home/worker/buildbot-aarch64/bolt-aarch64-ubuntu-clang/build/bin/clang-bolt.inst --driver-mode=g++  -c /home/worker/buildbot-aarch64/llvm-project/clang/utils/perf-training/cxx/hello_world.cpp
# executed command: /home/worker/buildbot-aarch64/bolt-aarch64-ubuntu-clang/build/bin/clang-bolt.inst --driver-mode=g++ -c /home/worker/buildbot-aarch64/llvm-project/clang/utils/perf-training/cxx/hello_world.cpp
# .---command stderr------------
# | free(): invalid pointer
# | PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
# | Stack dump:
# | 0.	Program arguments: /home/worker/buildbot-aarch64/bolt-aarch64-ubuntu-clang/build/bin/clang-bolt.inst --driver-mode=g++ -c /home/worker/buildbot-aarch64/llvm-project/clang/utils/perf-training/cxx/hello_world.cpp
# | 1.	<eof> parser at end of file
# | Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
# | 0  clang-bolt.inst 0x0000aaaacd7cc344 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 68
# | 1  clang-bolt.inst 0x0000aaaacd7c679c llvm::sys::RunSignalHandlers() + 156
# | 2  clang-bolt.inst 0x0000aaaacd61310c
# | 3  linux-vdso.so.1 0x0000ffff8341f7dc __kernel_rt_sigreturn + 0
# | 4  libc.so.6       0x0000ffff82f7f200
# | 5  libc.so.6       0x0000ffff82f3a67c raise + 28
# | 6  libc.so.6       0x0000ffff82f27130 abort + 228
# | 7  libc.so.6       0x0000ffff82f73308
# | 8  libc.so.6       0x0000ffff82f8957c
# | 9  libc.so.6       0x0000ffff82f8b2c4
# | 10 libc.so.6       0x0000ffff82f8dc84 free + 176
# | 11 clang-bolt.inst 0x0000aaaaccfc6264 llvm::MCContext::~MCContext() + 4552
# | 12 clang-bolt.inst 0x0000aaaacb9758e8
# | 13 clang-bolt.inst 0x0000aaaacc9928b0 llvm::PMTopLevelManager::~PMTopLevelManager() + 432
# | 14 clang-bolt.inst 0x0000aaaacc9932a0
# | 15 clang-bolt.inst 0x0000aaaacc98528c llvm::legacy::PassManager::~PassManager() + 132
# | 16 clang-bolt.inst 0x0000aaaaceb9266c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>, clang::BackendConsumer*) + 8236
# | 17 clang-bolt.inst 0x0000aaaaced85e44 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) + 2884
# | 18 clang-bolt.inst 0x0000aaaad37f570c clang::ParseAST(clang::Sema&, bool, bool) + 3212
# | 19 clang-bolt.inst 0x0000aaaacf6d9524 clang::FrontendAction::Execute() + 540
# | 20 clang-bolt.inst 0x0000aaaacf5b56f8 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) + 3128
# | 21 clang-bolt.inst 0x0000aaaacf8d05c8 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) + 1400
# | 22 clang-bolt.inst 0x0000aaaaca61335c cc1_main(llvm::ArrayRef<char const*>, char const*, void*) + 16900

[mstorsjo]

This PR seems to have broken ASAN in mingw configurations. The symptoms seem to be that the ASAN DLL just locks up, hard, when the process is loaded. (Or more precisely, has entered some infinite loop.)

I tried attaching to such a hung process with windbg, and it's showing this:

Break-in sent, waiting 30 seconds...
WARNING: Break-in timed out, suspending.
         This is usually caused by another thread holding the loader lock.

And a backtrace that looks like this:

libclang_rt_asan_dynamic_x86_64!_asan_default_options__dll
libclang_rt_asan_dynamic_x86_64!_asan_default_options__dll+0xa17
libclang_rt_asan_dynamic_x86_64!_sanitizer_register_weak_function+0xd2
stacksmash_asan+0x1dad
stacksmash_asan+0x16b7
stacksmash_asan+0x20e3
ntdll!LdrpCallInitRoutine+0x6f
ntdll!...

(Side note; the PR seems to have added a bunch of new files with CRLF newlines - it'd be nice to clean this up.)

To repro the issue for yourself, you can do the following:

1. Download and unzip https://github.com/mstorsjo/llvm-mingw/releases/download/20240903/llvm-mingw-20240903-ucrt-x86_64.zip, and add the llvm-mingw-<version>-ucrt-x86_64\bin directory to your %PATH% within a terminal
2. Configure a build of compiler-rt with the following parameters: cmake ..\compiler-rt -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER_TARGET=x86_64-w64-windows-gnu -DCOMPILER_RT_DEFAULT_TARGET_ONLY=TRUE -DCOMPILER_RT_USE_BUILTINS_LIBRARY=TRUE -DSANITIZER_CXX_ABI=libc++ -DCMAKE_INSTALL_PREFIX=c:\code\llvm-mingw-20240903-ucrt-x86_64\lib\clang\19
3. Build and install (on top of the newly unpacked toolchain), ninja install
4. Copy the newly built asan DLL into the current directory, copy /y lib\windows\libclang_rt.asan_dynamic-x86_64.dll .
5. Compile a trivial hello world, like https://github.com/mstorsjo/llvm-mingw/blob/master/test/hello.c (really any snippet will do), with asan clang hello.c -o hello.exe -fsanitize=address
6. Try to run hello.exe, which hangs.

It's possible to reproduce the same by building and running the whole compiler-rt testsuite as well, but that's a bit trickier to set up for this configuration.

Surprising side note; when I tried the repro procedure above, building compiler-rt with a slightly older Clang release, and 18.x version, the built ASAN seems to not hang. I'm not sure if this is a functional difference, or if it just so happens to link things slightly differently so whatever issue there is seems to just not happen.

This commit is kinda complex, as it not only removes the static asan configuration (which never was involved in mingw use cases before), but I guess also sets things up so the dynamic asan can be used even when linking the CRT statically?

This issue has caused my nightly builds to start failing: https://github.com/mstorsjo/llvm-mingw/actions/runs/10803005560

[aeubanks]

I think this is also somehow affecting asan tests on macos: https://green.lab.llvm.org/job/llvm.org/job/clang-stage1-RA/2034/

was able to reproduce locally build/cmake/runtimes/runtimes-bins/compiler-rt/test/sanitizer_common/asan-arm64-Darwin/allocator_returns_null.cpp failing at this commit and succeeding at the previous

[aeubanks]

previous passing output

    2: ================================================================= 
    3: ==7382==ERROR: AddressSanitizer: requested allocation size 0x10000000001 (0x10000001008 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0) 
    4:  #0 0x0001032a9268 in _Znwm+0x6c (libclang_rt.asan_osx_dynamic.dylib:arm64+0x61268) 
    5:  #1 0x000102cd775c in main allocator_returns_null.cpp:82 
    6:  #2 0x00018cc07150 (<unknown module>) 
    7:  #3 0x3a08fffffffffffc (<unknown module>) 
    8:  
    9: ==7382==HINT: if you don't care about these errors you may set allocator_may_return_null=1 
   10: SUMMARY: AddressSanitizer: allocation-size-too-big allocator_returns_null.cpp:82 in main 
   11: ==7382==ABORTING 

failing output with this PR

             2: ================================================================= 
             3: ==6635==ERROR: AddressSanitizer: requested allocation size 0x10000000001 (0x10000001008 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0) 
             4:  #0 0x0001029b60ec in malloc+0x70 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x520ec) 
             5:  #1 0x00018cf4abd0 in operator new(unsigned long)+0x1c (libc++abi.dylib:arm64+0x16bd0) 
             6:  #2 0xde678001023f375c (<unknown module>) 
             7:  #3 0x00018cc07150 (<unknown module>) 
             8:  #4 0x9c0d7ffffffffffc (<unknown module>) 
             9:  
            10: ==6635==HINT: if you don't care about these errors you may set allocator_may_return_null=1 
            11: SUMMARY: AddressSanitizer: allocation-size-too-big (libc++abi.dylib:arm64+0x16bd0) in operator new(unsigned long)+0x1c 
            12: ==6635==ABORTING 

[aeubanks]

this line seems to be the problem, reverting it fixes the tests

[barcharcraz]

This should be safe to revert, and I'll make a PR reverting it (and probably just commit it if it doesn't get any reviews before I go to sleep).

[barcharcraz]

I've opened #108329 which should revert this particular edit (which was arguably a driveby change anyway)

[barcharcraz]

This PR seems to have broken ASAN in mingw configurations. The symptoms seem to be that the ASAN DLL just locks up, hard, when the process is loaded. (Or more precisely, has entered some infinite loop.)

I tried attaching to such a hung process with windbg, and it's showing this:

Break-in sent, waiting 30 seconds...
WARNING: Break-in timed out, suspending.
         This is usually caused by another thread holding the loader lock.

And a backtrace that looks like this:

libclang_rt_asan_dynamic_x86_64!_asan_default_options__dll
libclang_rt_asan_dynamic_x86_64!_asan_default_options__dll+0xa17
libclang_rt_asan_dynamic_x86_64!_sanitizer_register_weak_function+0xd2
stacksmash_asan+0x1dad
stacksmash_asan+0x16b7
stacksmash_asan+0x20e3
ntdll!LdrpCallInitRoutine+0x6f
ntdll!...

(Side note; the PR seems to have added a bunch of new files with CRLF newlines - it'd be nice to clean this up.)

To repro the issue for yourself, you can do the following:

1. Download and unzip https://github.com/mstorsjo/llvm-mingw/releases/download/20240903/llvm-mingw-20240903-ucrt-x86_64.zip, and add the llvm-mingw-<version>-ucrt-x86_64\bin directory to your %PATH% within a terminal
2. Configure a build of compiler-rt with the following parameters: cmake ..\compiler-rt -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER_TARGET=x86_64-w64-windows-gnu -DCOMPILER_RT_DEFAULT_TARGET_ONLY=TRUE -DCOMPILER_RT_USE_BUILTINS_LIBRARY=TRUE -DSANITIZER_CXX_ABI=libc++ -DCMAKE_INSTALL_PREFIX=c:\code\llvm-mingw-20240903-ucrt-x86_64\lib\clang\19
3. Build and install (on top of the newly unpacked toolchain), ninja install
4. Copy the newly built asan DLL into the current directory, copy /y lib\windows\libclang_rt.asan_dynamic-x86_64.dll .
5. Compile a trivial hello world, like https://github.com/mstorsjo/llvm-mingw/blob/master/test/hello.c (really any snippet will do), with asan clang hello.c -o hello.exe -fsanitize=address
6. Try to run hello.exe, which hangs.

It's possible to reproduce the same by building and running the whole compiler-rt testsuite as well, but that's a bit trickier to set up for this configuration.

Surprising side note; when I tried the repro procedure above, building compiler-rt with a slightly older Clang release, and 18.x version, the built ASAN seems to not hang. I'm not sure if this is a functional difference, or if it just so happens to link things slightly differently so whatever issue there is seems to just not happen.

This commit is kinda complex, as it not only removes the static asan configuration (which never was involved in mingw use cases before), but I guess also sets things up so the dynamic asan can be used even when linking the CRT statically?

This issue has caused my nightly builds to start failing: https://github.com/mstorsjo/llvm-mingw/actions/runs/10803005560

After investigating it is indeed mangled code due to the UB in register_weak_. I had to step through from the very beginning of process init, as it doesn't seem like the debuginfo generated by -gcodeview points to the right place, but anyway for __asan_default_options_dll the codegen for the weak registration function looks as follows

00007ff6`90501440 4883ec28                 sub     rsp, 28h
00007ff6`90501444 488d0d752c0000           lea     rcx, [hello!.refptr._newmode+0x38 (7ff6905040c0)]
00007ff6`9050144b 488d159e1f0000           lea     rdx, [hello!__asan_default_options__dll (7ff6905033f0)]
00007ff6`90501452 e8e9060000               call    hello!_ZN11__sanitizer13register_weakEPKcy (7ff690501b40)
00007ff6`90501457 89442424                 mov     dword ptr [rsp+24h], eax
00007ff6`9050145b 8b442424                 mov     eax, dword ptr [rsp+24h]
00007ff6`9050145f 4883c428                 add     rsp, 28h
00007ff6`90501463 c3                       ret     

Note that there is no comparison, the call to __sanitizer::register_weak is made unconditionally, even though in this case the local function should be the same as the default implementation from the dll. Worse, it seems like it's not even loading the correct pointer for the local function, so it ends up "intercepting" some random memory!

I've fixed this in #108327

[zmodem]

After this change we're running into #34485 ("ASan strtol interceptor breaks errno on Windows"):

C:\src\chromium\src>type a.cc
#include <stdio.h>
#include <stdlib.h>

int main() {
  errno = 0;
  long result = strtol("2147483648", nullptr, 10);
  printf("errno: %d\n", errno);
  printf("result: %ld\n", result);
  return 0;
}

C:\src\chromium\src>"third_party\llvm-build\Release+Asserts\bin\clang-cl.exe" a.cc /Feout\Release\a.exe && out\Release\a.exe
errno: 34
result: 2147483647

C:\src\chromium\src>"third_party\llvm-build\Release+Asserts\bin\clang-cl.exe" -fsanitize=address a.cc /Feout\Release\a.exe && out\Release\a.exe
errno: 0
result: 2147483647

That's an old bug, but it wasn't a problem in Chromium until this runtime change.

It sounds like Visual Studio users also hit this before: https://developercommunity.visualstudio.com/t/Cannot-detect-strtol-range-errors-with-A/10412869 Do you know what the fix for that was?