Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ASAN tests are sporadically failing #2908

Closed
StephanTLavavej opened this issue Jul 25, 2022 · 6 comments
Closed

ASAN tests are sporadically failing #2908

StephanTLavavej opened this issue Jul 25, 2022 · 6 comments
Assignees
@StephanTLavavej
Labels
ASan Address Sanitizer high priority Important! not reproducible We can’t reproduce the described behavior test Related to test code

Comments

@StephanTLavavej
Copy link
Member

StephanTLavavej commented Jul 25, 2022 •
edited by CaseyCarter
Loading

We're seeing sporadic failures in GH_002030_asan_annotate_string and GH_002030_asan_annotate_vector. They've all happened for x64 so far:

Date PR Test Run
2022-07-14 #2855 https://dev.azure.com/vclibs/STL/_build/results?buildId=11694&view=results
2022-07-19 #2883 https://dev.azure.com/vclibs/STL/_build/results?buildId=11723&view=results
2022-07-24 #2880 https://dev.azure.com/vclibs/STL/_build/results?buildId=11750&view=results
2022-07-25 #2806 https://dev.azure.com/vclibs/STL/_build/results?buildId=11775&view=results
2022-07-26 #2947 https://dev.azure.com/vclibs/STL/_build/results?buildId=11801&view=results
2022-07-29 #2958 https://dev.azure.com/vclibs/STL/_build/results?buildId=11888&view=results
2022-08-03 #2964 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=11964&view=results
2022-08-03 #2994 https://dev.azure.com/vclibs/STL/_build/results?buildId=11969&view=results
2022-08-07 #3002 https://dev.azure.com/vclibs/STL/_build/results?buildId=12011&view=results
2022-08-08 #3011 https://dev.azure.com/vclibs/STL/_build/results?buildId=12024&view=results
2022-08-13 #2169 https://dev.azure.com/vclibs/STL/_build/results?buildId=12118&view=results
2022-08-17 #3038 https://dev.azure.com/vclibs/STL/_build/results?buildId=12173&view=results
2022-08-23 #3049 https://dev.azure.com/vclibs/STL/_build/results?buildId=12236&view=results
2022-09-06 #3080 https://dev.azure.com/vclibs/STL/_build/results?buildId=12398&view=results
2022-09-08 #3083 https://dev.azure.com/vclibs/STL/_build/results?buildId=12429&view=results
2022-09-16 #3077 https://dev.azure.com/vclibs/STL/_build/results?buildId=12492&view=results
2022-09-20 #3117 https://dev.azure.com/vclibs/STL/_build/results?buildId=12530&view=results
2022-10-24 #3157 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=12730&view=results
2022-10-26 #3174 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=12753&view=results
2022-12-06 #3258 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=12914&view=results
2023-01-12 #3344 https://dev.azure.com/vclibs/STL/_build/results?buildId=13117&view=results
2023-01-21 #3145 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13167&view=results
2023-01-26 #3322 https://dev.azure.com/vclibs/STL/_build/results?buildId=13207&view=results
2023-01-29 #3367 https://dev.azure.com/vclibs/STL/_build/results?buildId=13231&view=results
2023-01-31 #3361 https://dev.azure.com/vclibs/STL/_build/results?buildId=13266&view=results
2023-02-06 #3389 https://dev.azure.com/vclibs/STL/_build/results?buildId=13321&view=results
2023-02-09 #3361 https://dev.azure.com/vclibs/STL/_build/results?buildId=13341&view=results
2023-02-10 #3380 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13346&view=results
2023-02-12 #3453 https://dev.azure.com/vclibs/STL/_build/results?buildId=13378&view=results
2023-02-13 #3390 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13411&view=results
2023-02-15 #3470 https://dev.azure.com/vclibs/STL/_build/results?buildId=13447&view=results
2023-02-15 #3469 https://dev.azure.com/vclibs/STL/_build/results?buildId=13449&view=results
2023-02-19 #3475 https://dev.azure.com/vclibs/STL/_build/results?buildId=13498&view=results
2023-02-20 #3477 https://dev.azure.com/vclibs/STL/_build/results?buildId=13519&view=results
2023-02-22 #3492 https://dev.azure.com/vclibs/STL/_build/results?buildId=13543&view=results
2023-02-22 #3469 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13553&view=results
2023-02-22 #3475 https://dev.azure.com/vclibs/STL/_build/results?buildId=13548&view=results
2023-02-25 #3503 https://dev.azure.com/vclibs/STL/_build/results?buildId=13596&view=results
2023-02-26 #3506 https://dev.azure.com/vclibs/STL/_build/results?buildId=13608&view=results
2023-03-06 #3523 https://dev.azure.com/vclibs/STL/_build/results?buildId=13695&view=results
2023-03-07 #3547 https://dev.azure.com/vclibs/STL/_build/results?buildId=13697&view=results
2023-03-07 #3543 https://dev.azure.com/vclibs/STL/_build/results?buildId=13708&view=results
2023-03-07 #3523 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13710&view=results
2023-03-13 #3551 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13788&view=results
2023-03-13 #3569 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13794&view=results
2023-03-28 #3587 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13895&view=results
2023-03-28 #3591 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=13897&view=results
2023-03-29 #2116 https://dev.azure.com/vclibs/STL/_build/results?buildId=13904&view=results
2023-04-04 #3619 https://dev.azure.com/vclibs/STL/_build/results?buildId=13930&view=results
2023-04-06 #3630 https://dev.azure.com/vclibs/STL/_build/results?buildId=13950&view=results
2023-04-08 #3641 https://dev.azure.com/vclibs/STL/_build/results?buildId=13986&view=results
2023-05-04 #3677 (CI) https://dev.azure.com/vclibs/STL/_build/results?buildId=14078&view=results
2023-05-25 #3725 https://dev.azure.com/vclibs/STL/_build/results?buildId=14201&view=results

Example:

Build setup steps:
Build steps:
Command: "C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.33.31627\bin\HostX64\x64\cl.EXE" "C:\a\1\s\tests\std\tests\GH_002030_asan_annotate_vector\test.cpp" "-ID:\build\out\inc" "-IC:\a\1\s\llvm-project\libcxx\test\support" "-IC:\a\1\s\tests\std\include" "/nologo" "/Od" "/W4" "/w14061" "/w14242" "/w14265" "/w14582" "/w14583" "/w14587" "/w14588" "/w14749" "/w14841" "/w14842" "/w15038" "/w15214" "/w15215" "/w15216" "/w15217" "/sdl" "/WX" "/Zc:strictStrings" "/D_ENABLE_STL_INTERNAL_CHECK" "/bigobj" "/FIforce_include.hpp" "/w14365" "/D_ENFORCE_FACET_SPECIALIZATIONS=1" "/D_STL_CALL_ABORT_INSTEAD_OF_INVALID_PARAMETER" "/Zi" "/wd4611" "/w14640" "/Zc:threadSafeInit-" "-fsanitize=address" "/EHsc" "/MTd" "/std:c++latest" "/permissive" "/fno-sanitize-address-vcasan-lib" "-FeD:\build\tests\std\tests\GH_002030_asan_annotate_vector\Output\16\GH_002030_asan_annotate_vector.exe" "-link" "-LIBPATH:D:\build\out\lib\amd64" "-LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.33.31627\lib\x64" "/MANIFEST:EMBED" "/debug"
Exit Code: 0
Standard Output:
--
test.cpp
   Creating library D:\build\tests\std\tests\GH_002030_asan_annotate_vector\Output\16\GH_002030_asan_annotate_vector.lib and object D:\build\tests\std\tests\GH_002030_asan_annotate_vector\Output\16\GH_002030_asan_annotate_vector.exp
--

Intellisense response file steps:
Test setup steps:
Test steps:
Test step failed unexpectedly.
Command: "D:\build\tests\std\tests\GH_002030_asan_annotate_vector\Output\16\GH_002030_asan_annotate_vector.exe"
Exit Code: 3221226356

@fsb4000 noted that this is STATUS_HEAP_CORRUPTION.

This bug is VSO-1813166 / AB#1813166 in the MSVC internal bug database.
@StephanTLavavej StephanTLavavej added the test Related to test code label Jul 25, 2022
@CaseyCarter

This comment was marked as resolved.

Sign in to view
@StephanTLavavej StephanTLavavej added the high priority Important! label Jul 26, 2022
@StephanTLavavej
Copy link
Member Author

Marking as high priority as this is a potentially scary issue that we should find the root cause of. (In the past, sporadic failures in the STL's GitHub repo have found real bugs, at least twice.)

@StephanTLavavej StephanTLavavej self-assigned this Jul 26, 2022
@CaseyCarter CaseyCarter mentioned this issue Aug 8, 2022
Merged
@StephanTLavavej
Copy link
Member Author

In the 1ES Hosted Pool that I'm experimenting with, I got a bit more info:

Command: "D:\build\tests\std\tests\GH_002030_asan_annotate_vector\Output\42\GH_002030_asan_annotate_vector.exe"
Exit Code: 57005
Standard Error:
--
=================================================================
==2160==ERROR: AddressSanitizer: access-violation on unknown address 0xffffffffffffffff (pc 0x7ff9b3d1836c bp 0x021524d40000 sp 0x0036f2efe570 T0)
==2160==The signal is caused by a READ memory access.
--

@fsb4000 fsb4000 mentioned this issue Sep 7, 2022
Closed
@StephanTLavavej
Copy link
Member Author

Interestingly, in https://dev.azure.com/vclibs/STL/_build/results?buildId=12753&view=results (a CI run) there was a failure in GH_002030_asan_annotate_string despite the fact that string annotations are currently disabled. (I think this is the first time I've seen this.) This indicates that the mere presence of -fsanitize=address is sufficient, and that the annotations themselves are not involved.

Build setup steps:
Build steps:
Command: "C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31921\bin\HostX64\x64\cl.EXE" "D:\a\_work\1\s\tests\std\tests\GH_002030_asan_annotate_string\test.cpp" "-ID:\build\out\inc" "-ID:\a\_work\1\s\llvm-project\libcxx\test\support" "-ID:\a\_work\1\s\tests\std\include" "/nologo" "/Od" "/W4" "/w14061" "/w14242" "/w14265" "/w14582" "/w14583" "/w14587" "/w14588" "/w14749" "/w14841" "/w14842" "/w15038" "/w15214" "/w15215" "/w15216" "/w15217" "/w15262" "/sdl" "/WX" "/Zc:strictStrings" "/D_ENABLE_STL_INTERNAL_CHECK" "/bigobj" "/FIforce_include.hpp" "/w14365" "/D_ENFORCE_FACET_SPECIALIZATIONS=1" "/D_STL_CALL_ABORT_INSTEAD_OF_INVALID_PARAMETER" "/Zi" "/wd4611" "/w14640" "/Zc:threadSafeInit-" "-fsanitize=address" "/EHsc" "/MT" "/std:c++latest" "/permissive-" "/fno-sanitize-address-vcasan-lib" "-FeD:\build\tests\std\tests\GH_002030_asan_annotate_string\Output\15\GH_002030_asan_annotate_string.exe" "-link" "-LIBPATH:D:\build\out\lib\amd64" "-LIBPATH:C:\Program Files\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.34.31921\lib\x64" "/MANIFEST:EMBED" "/debug"
Exit Code: 0
Standard Output:
--
test.cpp
   Creating library D:\build\tests\std\tests\GH_002030_asan_annotate_string\Output\15\GH_002030_asan_annotate_string.lib and object D:\build\tests\std\tests\GH_002030_asan_annotate_string\Output\15\GH_002030_asan_annotate_string.exp
--

Intellisense response file steps:
Test setup steps:
Test steps:
Test step failed unexpectedly.
Command: "D:\build\tests\std\tests\GH_002030_asan_annotate_string\Output\15\GH_002030_asan_annotate_string.exe"
Exit Code: 57005
Standard Error:
--
=================================================================
==8924==ERROR: AddressSanitizer: access-violation on unknown address 0xffffffffffffffff (pc 0x7fff36c7784c bp 0x018d954f0000 sp 0x0007258fe050 T0)
==8924==The signal is caused by a READ memory access.
--

@StephanTLavavej StephanTLavavej added the ASan Address Sanitizer label Nov 11, 2022
@StephanTLavavej StephanTLavavej mentioned this issue Feb 16, 2023
Closed
@shankarseal shankarseal mentioned this issue May 1, 2023
Closed
@shpalani
Copy link

shpalani commented May 1, 2023

@StephanTLavavej ,
We are observing a build failure run with AddressSanitizer, as stated in microsoft/ebpf-for-windows#2404.
Is this a known issue? If yes, is there a workaround or fix at the earliest? Thanks.

00 000000b8758fbe38 00007ffded2a1e88 ntdll!NtWaitForMultipleObjects+0x14
01 000000b8758fbe40 00007ffded2a145e ntdll!WerReportExceptionWorker+0x568
02 000000b8758fbec0 00007ffded2a0c1b ntdll!RtlReportExceptionEx+0x80e
03 000000b8758fbf90 00007ffded2c4014 ntdll!RtlReportException+0x9b
04 000000b8758fc010 00007ffdb868eb5e ntdll!RtlIsZeroMemory+0x154
05 000000b8758fc040 00007ffded2643af ucrtbased!_C_specific_handler+0x16e
06 000000b8758fc0f0 00007ffded1f170e ntdll!_chkstk+0x12f
07 000000b8758fc120 00007ffded20f2c3 ntdll!RtlVirtualUnwind2+0x35e
08 000000b8758fc860 00007ffded2c3fd9 ntdll!RtlRaiseException+0x163
09 000000b8758fd680 00007ffded2c3fa3 ntdll!RtlIsZeroMemory+0x119
0a 000000b8758fd6d0 00007ffded2ccdaa ntdll!RtlIsZeroMemory+0xe3
0b 000000b8758fd7c0 00007ffded2cd08a ntdll!_misaligned_access+0x41a
0c 000000b8758fd7f0 00007ffded2d7709 ntdll!_misaligned_access+0x6fa
0d 000000b8758fd820 00007ffded26a46d ntdll!_misaligned_access+0xad79
0e 000000b8758fd850 00007ffded1d78b1 ntdll!memset+0x432d
0f 000000b8758fd8a0 00007ffded1db550 ntdll!RtlEnterCriticalSection+0x431
10 000000b8758fda00 00007ffded1da8c1 ntdll!RtlGetCurrentServiceSessionId+0xbf0
11 000000b8758fdac0 00007ffdb69807d9 ntdll!RtlFreeHeap+0x51
12 000000b8758fdb00 00007ffded22994a clang_rt_asan_dbg_dynamic_x86_64!_asan_wrap_GlobalSize+0x4d7bd

@StephanTLavavej StephanTLavavej added the not reproducible We can’t reproduce the described behavior label Sep 21, 2023
@StephanTLavavej
Copy link
Member Author

This stopped reproing after May 2023, so it appears to have been fixed, possibly by VS 2022 17.7 Preview 2 (#3769) or a later toolset update.

@StephanTLavavej StephanTLavavej closed this as not planned Won't fix, can't repro, duplicate, stale Sep 21, 2023
@StephanTLavavej StephanTLavavej mentioned this issue Mar 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@StephanTLavavej StephanTLavavej

Labels
ASan Address Sanitizer high priority Important! not reproducible We can’t reproduce the described behavior test Related to test code
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@CaseyCarter @StephanTLavavej @shpalani