VS 17.8 MS testhost.exe crashing on .NET 8 with Internal CLR error. (0x80131506) #4752
Comments
|
Looks like this is running under code coverage. Could you try disabling that? |
|
I disabled codeCoverage in the yml task and removed the runsettings file for it, and still no crash after 10 test runs. It looks like this might be the culprit. |
|
@jakubch1 please have a look here. This is working after CC is disabled. |
|
@extedosse to find the root cause we need to reproduce issue locally. Could you please provide all binaries needed to run: Extedo.NGP.Connector.MessageQueue.Tests project? You can open issue on: https://developercommunity.visualstudio.com/home and provide those binaries in private way. We can also try to tweak your runsettings to find workaround for this issue. Could you please provide you runsettings and result .coverage file (it should be generated even if your test run failed)? |
|
Hello, |
|
Closing here. |
Description
I migrated a solution to .NET 8 from .NET 7 on wednesday and for that installed VS 2022 17.8 on the build system.
The PR build executes a few thousand tests in an Azure Devops pipeline and I noticed that around 30-40% of the time, testrunner.exe crashed.
Thus I opened an issue with a minidump on the runtime repo dotnet/runtime#94770
@jkotas suggested that the crash is due to unmanaged memory corruption and suggested that if it happened after upgrade to VS 17.8 it might be an issue in the respective vstest runner.
To continue with that, I modified my PR Build to manually install a specific test runner via the VisualStudioTestPlatformInstaller task:
The result of that is:
With the result of these tests, I closed the bug in the runtime repo and opened it again here.
Something keeps crashing test runner for my tests (completely unrelated tests), I tried with runInIsolation, without runInParallel and the runner still crashes.
Steps to reproduce
not yet certain
Expected behavior
No crash
Actual behavior
testhost.exe crashes
Diagnostic logs
************* Path validation summary ************** Response Time (ms) Location Deferred srv* Symbol search path is: srv* Executable search path is: Windows 10 Version 14393 MP (4 procs) Free x64 Product: Server, suite: TerminalServer DataCenter SingleUserTS Edition build lab: 10.0.14393.6343 (rs1_release.230913-1727) Debug session time: Wed Nov 15 13:18:13.000 2023 (UTC + 1:00) System Uptime: not available Process Uptime: 0 days 0:00:46.000 ................................................................ ................................................................ ..... For analysis of this file, run !analyze -v *** WARNING: Unable to verify timestamp for ntdll.dll *** WARNING: Unable to verify timestamp for KERNELBASE.dll ntdll!NtWaitForMultipleObjects+0x14: 00007ffd`aa2b6974 c3 ret 0:000> !analyze -v ******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* Unable to load image C:\Windows\System32\ole32.dll, Win32 error 0n2 *** WARNING: Unable to verify timestamp for ole32.dll Unable to load image C:\Windows\System32\combase.dll, Win32 error 0n2 *** WARNING: Unable to verify timestamp for combase.dll KEY_VALUES_STRING: 1 Key : Analysis.CPU.mSec Value: 3171 Key : Analysis.Elapsed.mSec Value: 7855 Key : Analysis.IO.Other.Mb Value: 1 Key : Analysis.IO.Read.Mb Value: 1 Key : Analysis.IO.Write.Mb Value: 2 Key : Analysis.Init.CPU.mSec Value: 234 Key : Analysis.Init.Elapsed.mSec Value: 8380 Key : Analysis.Memory.CommitPeak.Mb Value: 226 Key : CLR.Engine Value: CORECLR Key : CLR.Version Value: 8.0.23.53103 Key : Failure.Bucket Value: BREAKPOINT_80000003_ntdll.dll!NtWaitForMultipleObjects Key : Failure.Hash Value: {49536040-de3f-62da-08a0-ba8e5a2a4aba} Key : Timeline.Process.Start.DeltaSec Value: 46 Key : WER.OS.Branch Value: rs1_release Key : WER.OS.Version Value: 10.0.14393.6343 Key : WER.Process.Version Value: 17.800.23.55801 FILE_IN_CAB: testhost.exe_8160_1700050648_crashdump.dmp NTGLOBALFLAG: 0 PROCESS_BAM_CURRENT_THROTTLED: 0 PROCESS_BAM_PREVIOUS_THROTTLED: 0 APPLICATION_VERIFIER_FLAGS: 0 EXCEPTION_RECORD: (.exr -1) ExceptionAddress: 0000000000000000 ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 0 FAULTING_THREAD: 00001c2c PROCESS_NAME: testhost.exe ERROR_CODE: (NTSTATUS) 0x80000003 - {AUSNAHME} Haltepunkt Im Quellprogramm wurde ein Haltepunkt erreicht. EXCEPTION_CODE_STR: 80000003 STACK_TEXT: 000000c5`83b7da78 00007ffd`a6b64d6f : 00007ffd`2a4f4000 00000000`0000367a 00007ffd`2a4f4000 00007ffd`8097f586 : ntdll!NtWaitForMultipleObjects+0x14 000000c5`83b7da80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjectsEx+0xef STACK_COMMAND: ~0s; .ecxr ; kb SYMBOL_NAME: ntdll!NtWaitForMultipleObjects+14 MODULE_NAME: ntdll IMAGE_NAME: ntdll.dll FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!NtWaitForMultipleObjects OS_VERSION: 10.0.14393.6343 BUILDLAB_STR: rs1_release OSPLATFORM_TYPE: x64 OSNAME: Windows 10 IMAGE_VERSION: 6.2.14393.6343 FAILURE_ID_HASH: {49536040-de3f-62da-08a0-ba8e5a2a4aba} Followup: MachineOwner --------- 0:000> !DumpStack OS Thread Id: 0x1c2c (0) Current frame: ntdll!NtWaitForMultipleObjects + 0x14 Child-SP RetAddr Caller, Callee 000000C583B7DCA0 00007ffd8a185aa0 coreclr!StgPoolReadOnly::GetString + 0x20 [D:\a\_work\1\s\src\coreclr\inc\stgpool.h:215], calling coreclr!_guard_dispatch_icall_nop [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7DD00 00007ffd8a0dd5ba coreclr!MethodDesc::GetSig + 0xb2 [D:\a\_work\1\s\src\coreclr\vm\method.cpp:410], calling coreclr!_guard_dispatch_icall_nop [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7DD40 00007ffd8a156a32 coreclr!Thread::GetFinalApartment + 0x12 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:4868], calling coreclr!Thread::GetApartment [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:4754] 000000C583B7DE50 00007ffd8a156635 coreclr!Thread::DoAppropriateWait + 0x85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:3182], calling coreclr!Thread::DoAppropriateWaitWorker [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:3386] 000000C583B7DED0 00007ffd8a155df0 coreclr!SyncBlock::Wait + 0x1c8 [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp:2874], calling coreclr!Thread::DoAppropriateWait [D:\a\_work\1\s\src\coreclr\vm\threads.cpp:3156] 000000C583B7DFF0 00007ffd8a1559ad coreclr!ObjectNative::WaitTimeout + 0xcd [D:\a\_work\1\s\src\coreclr\classlibnative\bcltype\objectnative.cpp:307], calling coreclr!SyncBlock::Wait [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp:2786] 000000C583B7E118 00007ffd8a15595b coreclr!ObjectNative::WaitTimeout + 0x7b [D:\a\_work\1\s\src\coreclr\classlibnative\bcltype\objectnative.cpp:301], calling coreclr!LazyMachStateCaptureState [D:\a\_work\1\s\src\coreclr\vm\amd64\getstate.asm:55] 000000C583B7E170 00007ffd8003df9e (MethodDesc 00007ffd2a6d2620 + 0x1e System.Threading.Monitor.Wait(System.Object, Int32)), calling 00007ffd8a1558e0 (stub for System.Threading.Monitor.ObjWait(Int32, System.Object)) 000000C583B7E1A0 00007ffd80048f43 (MethodDesc 00007ffd2a8fb8f0 + 0x1c3 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)), calling 00007ffd2a64a148 000000C583B7E240 00007ffd80061e69 (MethodDesc 00007ffd2a839850 + 0xa9 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)), calling 00007ffd2a9101f8 000000C583B7E2C0 00007ffd80061c56 (MethodDesc 00007ffd2a839838 + 0x76 System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken)), calling System_Private_CoreLib + 0x66ed20 000000C583B7E340 00007ffd8006185a (MethodDesc 00007ffd2a839760 + 0x3a System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)), calling System_Private_CoreLib + 0x66ed20 000000C583B7E370 00007ffd80061732 (MethodDesc 00007ffd2a8396e8 + 0x12 System.Threading.Tasks.Task.Wait()), calling System_Private_CoreLib + 0x66ed20 000000C583B7E4A0 00007ffd2a654409 (MethodDesc 00007ffd2a7004d8 + 0x89 testhost.dll!Unknown) 000000C583B7E590 00007ffd8a1baa43 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:100] 000000C583B7E5D0 00007ffd8a0ea42e coreclr!MethodDescCallSite::CallTargetWorker + 0x296 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:570], calling coreclr!CallDescrWorkerInternal [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:37] 000000C583B7E660 00007ffd8a187bfb coreclr!MethodDesc::IsVoid + 0x2f [D:\a\_work\1\s\src\coreclr\vm\method.cpp:987], calling coreclr!SigPointer::PeekElemTypeClosed [D:\a\_work\1\s\src\coreclr\vm\siginfo.cpp:2429] 000000C583B7E710 00007ffd8a187988 coreclr!RunMainInternal + 0x11c [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1304], calling coreclr!MethodDescCallSite::CallTargetWorker [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:260] 000000C583B7E830 00007ffd8a114d5e coreclr!RunMain + 0xd2 [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1375], calling coreclr!RunMainInternal [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1275] 000000C583B7E8E0 00007ffd8a1148fb coreclr!Assembly::ExecuteMainMethod + 0x1bf [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1504], calling coreclr!RunMain [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1324] 000000C583B7E960 00007ffd8a1bbfea coreclr!DelayLoad_Helper + 0x7a [D:\a\_work\1\s\src\coreclr\vm\amd64\ExternalMethodFixupThunk.asm:61], calling coreclr!DynamicHelperWorker [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp:3776] 000000C583B7EA80 00007ffd8a1baa43 coreclr!CallDescrWorkerInternal + 0x83 [D:\a\_work\1\s\src\coreclr\vm\amd64\CallDescrWorkerAMD64.asm:100] 000000C583B7EAC0 00007ffd8a071996 coreclr!DispatchCallSimple + 0x72 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp:222], calling coreclr!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45] 000000C583B7EAE0 00007ffd8a0e6f9f coreclr!MethodDesc::TryGetMultiCallableAddrOfCode + 0x87 [D:\a\_work\1\s\src\coreclr\vm\method.cpp:2071], calling coreclr!MethodDesc::GetMethodEntryPoint [D:\a\_work\1\s\src\coreclr\vm\method.cpp:477] 000000C583B7EB20 00007ffd8a13e2b2 coreclr!`anonymous namespace'::GetConfigDWORD + 0x32 [D:\a\_work\1\s\src\coreclr\utilcode\clrconfig.cpp:247], calling coreclr!`anonymous namespace'::EnvGetString [D:\a\_work\1\s\src\coreclr\utilcode\clrconfig.cpp:134] 000000C583B7EBB0 00007ffd8a113d5e coreclr!CorHost2::ExecuteAssembly + 0x15e [D:\a\_work\1\s\src\coreclr\vm\corhost.cpp:340], calling coreclr!Assembly::ExecuteMainMethod [D:\a\_work\1\s\src\coreclr\vm\assembly.cpp:1440] 000000C583B7ECF0 00007ffd8a13bb68 coreclr!coreclr_execute_assembly + 0xd8 [D:\a\_work\1\s\src\coreclr\dlls\mscoree\exports.cpp:504], calling coreclr!_guard_dispatch_icall_nop [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7ED10 00007ffd8b5d24ba hostpolicy!breadcrumb_writer_t::begin_write + 0x1ba [D:\a\_work\1\s\src\native\corehost\hostpolicy\breadcrumbs.cpp:39], calling hostpolicy!trace::verbose [D:\a\_work\1\s\src\native\corehost\hostmisc\trace.cpp:131] 000000C583B7ED30 00007ffda7653531 ucrtbase!is_stream_flushable_or_commitable + 0x9, calling ucrtbase!is_stream_flushable 000000C583B7ED60 00007ffda76557e6 ucrtbase!fflush + 0x16, calling ucrtbase!is_stream_flushable_or_commitable 000000C583B7ED90 00007ffd8b5f2972 hostpolicy!run_app_for_context + 0x512 [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:256], calling hostpolicy!_guard_dispatch_icall_nop [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7EE60 00007ffd8b60c795 hostpolicy!mtx_do_lock + 0x69 [D:\a\_work\1\s\src\vctools\crt\github\stl\src\mutex.cpp:161], calling hostpolicy!__security_check_cookie [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45] 000000C583B7EE90 00007ffd8b60c987 hostpolicy!_Mtx_unlock + 0x1f [D:\a\_work\1\s\src\vctools\crt\github\stl\src\mutex.cpp:171], calling hostpolicy!_guard_dispatch_icall_nop [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7EF20 00007ffd8b5f2c5c hostpolicy!run_app + 0x3c [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:285], calling hostpolicy!run_app_for_context [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:208] 000000C583B7EF60 00007ffd8b5f3759 hostpolicy!corehost_main + 0x199 [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:426], calling hostpolicy!run_app [D:\a\_work\1\s\src\native\corehost\hostpolicy\hostpolicy.cpp:280] 000000C583B7F060 00007ffd8d39b63a hostfxr!execute_app + 0x2fa [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:145], calling hostfxr!_guard_dispatch_icall_nop [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7F160 00007ffd8d39e1e6 hostfxr!`anonymous namespace'::read_config_and_execute + 0xa6 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:532], calling hostfxr!execute_app [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:87] 000000C583B7F1E0 00007ffd8d3a1e67 hostfxr!std::vector<wchar_t const *,std::allocator<wchar_t const *> >::_Insert_counted_range<wchar_t const * *> + 0x1b7 [C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\include\vector:1199], calling hostfxr!memcpy [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\string\amd64\memcpy.asm:68] 000000C583B7F250 00007ffd8d3a04b6 hostfxr!fx_muxer_t::handle_exec_host_command + 0x166 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:1007], calling hostfxr!`anonymous namespace'::read_config_and_execute [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:515] 000000C583B7F300 00007ffd8d39e7c4 hostfxr!fx_muxer_t::execute + 0x494 [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:578], calling hostfxr!fx_muxer_t::handle_exec_host_command [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:990] 000000C583B7F440 00007ffd8d398660 hostfxr!hostfxr_main_startupinfo + 0xa0 [D:\a\_work\1\s\src\native\corehost\fxr\hostfxr.cpp:62], calling hostfxr!fx_muxer_t::execute [D:\a\_work\1\s\src\native\corehost\fxr\fx_muxer.cpp:555] 000000C583B7F510 00007ff610c4cc0d testhost_exe!trace::flush + 0x4d [D:\a\_work\1\s\src\corehost\common\trace.cpp:192], calling testhost_exe!pal::mutex_t::unlock [D:\a\_work\1\s\src\corehost\common\pal.windows.cpp:768] 000000C583B7F540 00007ff610c52d47 testhost_exe!exe_start + 0x997 [D:\a\_work\1\s\src\corehost\corehost.cpp:220], calling testhost_exe!_guard_dispatch_icall_nop [d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\misc\amd64\guard_dispatch.asm:53] 000000C583B7F558 00007ffdaa371460 ntdll!LdrpInvertedFunctionTable + 0x40 000000C583B7F6E0 00007ff610c4ce1c testhost_exe!trace::setup + 0x11c [D:\a\_work\1\s\src\corehost\common\trace.cpp:41], calling testhost_exe!__security_check_cookie [d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\gs\amd64\amdsecgs.asm:45] 000000C583B7F710 00007ff610c54771 testhost_exe!atexit + 0x9 [d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\utility\utility.cpp:275], calling testhost_exe!_onexit [d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\utility\utility.cpp:256] 000000C583B7F760 00007ff610c532c8 testhost_exe!wmain + 0x88 [D:\a\_work\1\s\src\corehost\corehost.cpp:287], calling testhost_exe!exe_start [D:\a\_work\1\s\src\corehost\corehost.cpp:90] 000000C583B7F790 00007ff610c54978 testhost_exe!__scrt_common_main_seh + 0x10c [d:\a01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288], calling testhost_exe!wmain [D:\a\_work\1\s\src\corehost\corehost.cpp:269] 000000C583B7F7D0 00007ffda95284d4 kernel32!BaseThreadInitThunk + 0x14, calling kernel32!guard_dispatch_icall_nop 0:000> !verifyheap 98.251 objects verified, 0 errors. No heap corruption detected.testhost.exe_8160_1700050648_crashdump.zip
Environment
Windows Server 2016 DataCenter
Visual Studio 2022 17.8 (from tuesday)
The text was updated successfully, but these errors were encountered: