Can't run VS address sanitizer on Windows-2022 runner

[Reedbeta]

Description

(Initially posted in the community discussions, but then I found this repo and thought an issue here might be better.)

I'm having trouble running an executable compiled with Visual Studio with Address Sanitizer enabled, on the Windows runners.

Here's a minimal repro: https://github.com/Reedbeta/gha-asan-test

This is VS's canned hello-world app, which I've modified by enabling ASan in the Debug configuration.

Now, running ASan requires clang_rt.asan_dynamic-x86_64.dll to be on the PATH. This will be the case when running through a VS developer command prompt, so in my GHA setup I've used the setup-msvc-dev action to do this.

I've verified that the clang ASan DLL is present in the expected directory on the runner, i.e. C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64, and I've verified that directory is on the PATH, e.g. I can invoke cl.exe and it's found.

However, running my hello-world app with ASan on the Windows runner fails immediately with no output and an exit code -1073741502, which maps to 0xc0000142 which is STATUS_DLL_INIT_FAILED.

I cannot reproduce this on my own machine (Windows 11 10.0.22621.2715, VS Community 17.7.3, same toolchain version 14.37.32822). For me, the app runs without error.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image: windows-2022
Version: 20231115.2.0

Failed run: https://github.com/Reedbeta/gha-asan-test/actions/runs/6997736159/job/19035042249

Is it regression?

Unknown

Expected behavior

Executables compiled with ASan should run successfully.

Actual behavior

The executable fails with exit code -1073741502.

Repro steps

Fork https://github.com/Reedbeta/gha-asan-test and run the "Build and Test" action.

[ilia-shipitsin]

hello, @Reedbeta , I'm a bit confused with the whole idea. You use https://github.com/marketplace/actions/setup-msvc-developer-command-prompt and probably issue should be reported there. What do you think ?

[ilia-shipitsin]

meanwhile, I checked whether we install appropriate workload, it's there

[ilia-shipitsin]

also, I've added

    - name: check PATH
      run: |
        ${env:PATH}.Split(';').Where({ $_ -like '*Hostx64*' }) | gci -filter '*asan*'
      shell: pwsh

and libs are in the PATH:

https://github.com/ilia-shipitsin/gha-asan-test/actions/runs/7004748447/job/19053131076

[Reedbeta]

Hi @ilia-shipitsin, thanks for taking a look. I don't think the setup-msvc-dev action has anything to do with this, I just used it to get the proper directory on PATH. We can try another method of adding the directory to the PATH (not using the action) if you like - but, as you verified, the ASan component is installed and DLLs are indeed present on the PATH. Also note the error code for a missing DLL is a different one, STATUS_DLL_NOT_FOUND, while here I'm getting STATUS_DLL_INIT_FAILED.

So this seems to be some other issue or incompatibility with the runner configuration. After some googling I found this thread which sounds like similar symptoms, but it was marked fixed more than two years ago. I think this would need someone to connect a debugger on the runner to see more about what's going on.

[ilia-shipitsin]

I did a check (above) that shown that asan lib present in PATH.
is there any other requirement to check ?

[ilia-shipitsin]

maybe you will find useful to upload a binary via artifacts and investigate it locally

[ilia-shipitsin]

also, we installed VS 17.8.0 (the latest one available at the time image created), are you running the same locally ?

this week 17.8.1 will be deployed

[ilia-shipitsin]

@Reedbeta

I enabled user dump collection

https://github.com/ilia-shipitsin/gha-asan-test/blob/main/.github/workflows/build.yml#L22-L54

here's dump: https://github.com/ilia-shipitsin/gha-asan-test/actions/runs/7008887175

windbg says there's something wrong with ASAN. looks like VS regression (?!)

*** WARNING: Check Image - Checksum mismatch - Dump: 0x37ae17, File: 0x37f596 - C:\ProgramData\Dbg\sym\KERNELBASE.dll\7472090C36f000\KERNELBASE.dll
*** WARNING: Unable to verify checksum for ConsoleApplication1.exe

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 250

    Key  : Analysis.Elapsed.mSec
    Value: 3820

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 3

    Key  : Analysis.IO.Write.Mb
    Value: 21

    Key  : Analysis.Init.CPU.mSec
    Value: 93

    Key  : Analysis.Init.Elapsed.mSec
    Value: 11130

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 76

    Key  : Failure.Bucket
    Value: BREAKPOINT_80000003_clang_rt.asan_dynamic-x86_64.dll!__interception::InterceptionFailed

    Key  : Failure.Hash
    Value: {41579315-1333-eda9-0c4d-516f537fd2e8}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 349

    Key  : Timeline.Process.Start.DeltaSec
    Value: 3

    Key  : WER.OS.Branch
    Value: fe_release

    Key  : WER.OS.Version
    Value: 10.0.20348.1


FILE_IN_CAB:  ConsoleApplication1.exe.8100.dmp

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
rax=0000000000000000 rbx=00007ff846b7f801 rcx=f0d1ac4a12e10000
rdx=0000000000000000 rsi=00007ff846b7f850 rdi=0000000000000000
rip=00007ff87be09f62 rsp=00000042e5deea18 rbp=0000000000000000
 r8=000000000000005a  r9=000000000000005b r10=0000000000000069
r11=00000042e5deea00 r12=00007ff84136347c r13=00007ff8414035a0
r14=00007ff842206a48 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
KERNELBASE!wil::details::DebugBreak+0x2:
00007ff8`7be09f62 cc              int     3
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ff87be09f62 (KERNELBASE!wil::details::DebugBreak+0x0000000000000002)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0000000000000000

PROCESS_NAME:  ConsoleApplication1.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

EXCEPTION_PARAMETER1:  0000000000000000

STACK_TEXT:  
00000042`e5deea18 00007ff8`4136a3d3     : 00007ff8`46b7f853 00007ff8`46b7f850 00000000`00000000 00007ff8`7c99c340 : KERNELBASE!wil::details::DebugBreak+0x2
00000042`e5deea20 00007ff8`4136a040     : 00000000`00000000 00007ff8`46b7f853 00000000`00000003 00007ff8`7e5100f3 : clang_rt_asan_dynamic_x86_64!__interception::InterceptionFailed+0x63
00000042`e5deea50 00007ff8`4136af1b     : 00000000`00000003 00007ff8`413698b8 00007ff8`7e3a6eba 00000000`00000008 : clang_rt_asan_dynamic_x86_64!__interception::GetInstructionSize+0x760
00000042`e5deea80 00007ff8`4136abb6     : 00000000`00000006 00007ff8`46b7f850 00007ff8`46b7f850 00007ff8`7e3a6ec0 : clang_rt_asan_dynamic_x86_64!__interception::RoundUpToInstrBoundary+0x2b
00000042`e5deeab0 00007ff8`4136a8ee     : 00007ff8`42206a48 00000000`00000000 00000000`00000000 00007ff8`46b7f850 : clang_rt_asan_dynamic_x86_64!__interception::OverrideFunctionWithTrampoline+0x36
00000042`e5deeb00 00007ff8`4136a824     : 00000000`00000001 00000042`e5deef00 00007ff8`414c13c0 00007ff8`7bd5ddd6 : clang_rt_asan_dynamic_x86_64!__interception::OverrideFunction+0x4e
00000042`e5deeb30 00007ff8`413adb4c     : 00007ff8`413f0248 00000042`e5deefb0 00000042`e5def680 00000000`0000000f : clang_rt_asan_dynamic_x86_64!__interception::OverrideFunction+0xa4
00000042`e5deeb90 00007ff8`413ad50f     : 00007ff8`413f0248 00000042`e5deefb0 00000042`e5def680 00000042`e5def680 : clang_rt_asan_dynamic_x86_64!InitializeCommonInterceptors+0x2ec
00000042`e5deebd0 00007ff8`413ce05a     : 00007ff8`413f0248 00000042`e5def680 00008952`00000002 00007ff8`00000002 : clang_rt_asan_dynamic_x86_64!__asan::InitializeAsanInterceptors+0x1f
00000042`e5deec10 00007ff8`7c0a2c8d     : 00007ff8`413f0220 00000000`7ffe0300 00000042`e5def680 00007ff8`7e38179a : clang_rt_asan_dynamic_x86_64!__asan::AsanInitInternal+0xea
00000042`e5deec70 00007ff8`413e8fba     : 00000000`7ffe0300 00007ff8`7e327390 00000042`e5deefb0 00007ff8`7e327390 : ucrtbase!initterm+0x2d
00000042`e5deeca0 00007ff8`413e9198     : 00000000`7ffe0385 00000042`e5def680 00000000`7ffe0300 00000000`00000001 : clang_rt_asan_dynamic_x86_64!dllmain_crt_process_attach+0x9a
00000042`e5deecd0 00007ff8`7e327387     : 00007ff8`41360000 00000000`00000001 00000042`e5def680 00000000`7ffe0385 : clang_rt_asan_dynamic_x86_64!dllmain_dispatch+0x74
00000042`e5deed30 00007ff8`7e36fa8a     : 0000023a`0b6c69e0 00007ff8`41360000 00007ff8`00000001 00000000`00000000 : ntdll!LdrpCallInitRoutine+0x6b
00000042`e5deeda0 00007ff8`7e36f836     : 0000023a`0b6c6b20 0000023a`0b6c6b00 00000042`00000000 0000023a`0b6c6a28 : ntdll!LdrpInitializeNode+0x1ca
00000042`e5deeee0 00007ff8`7e36f8a0     : 00007ff8`7e42bd00 0000023a`0b6be2c0 00000042`e5deefb0 0000023a`0b6c9180 : ntdll!LdrpInitializeGraphRecurse+0x42
00000042`e5deef20 00007ff8`7e3d80d1     : 00007ff8`7e430700 00007ff8`7e4307d8 00000000`00000000 00000042`e5fcf000 : ntdll!LdrpInitializeGraphRecurse+0xac
00000042`e5deef60 00007ff8`7e378828     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000001 : ntdll!LdrpInitializeProcess+0x1ca5
00000042`e5def330 00007ff8`7e39e35d     : 00000000`00000000 00007ff8`7e300000 00000042`e5fd0000 00000042`e5fcf050 : ntdll!LdrpInitialize+0x16c
00000042`e5def3b0 00007ff8`7e37862e     : 00000042`e5def680 00000000`00000000 00000042`e5def680 00000000`00000000 : ntdll!LdrpInitializeInternal+0xed
00000042`e5def630 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe


STACK_COMMAND:  ~0s; .ecxr ; kb

SYMBOL_NAME:  clang_rt_asan_dynamic_x86_64!__interception::InterceptionFailed+63

MODULE_NAME: clang_rt_asan_dynamic_x86_64

IMAGE_NAME:  clang_rt.asan_dynamic-x86_64.dll

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_clang_rt.asan_dynamic-x86_64.dll!__interception::InterceptionFailed

OS_VERSION:  10.0.20348.1

BUILDLAB_STR:  fe_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {41579315-1333-eda9-0c4d-516f537fd2e8}

Followup:     MachineOwner
---------

[ilia-shipitsin]

I'm not sure we did something wrong on image level. Also, we do not run e2e asan tests assuming those are run on the VS side

[Reedbeta]

Thanks for getting the crash dump and callstack, that's helpful. It looks like this is a very similar failure to the ones reported previously here and here. I suspect an incompatibility with the version of ASan and the version of Windows on the runner, or some other software on the runner. It probably needs to be addressed with either VS or LLVM.