Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

ericstj · 2022-05-25T23:16:34Z

Failures 7/5-8/5 (incl. PRs) based on Runfo in last 30 days:

Day	Run	Failure
8/4	PR #73174 - 1923945	Crash dump
8/4	Rolling run 1923908	Crash dump
~~8/4~~	~~PR #73073 - 1922939~~	fail: Application has finished with exit code 1 but 0 was expected - likely infra hiccup
8/3	PR #73265 - 1921135	Crash dump
8/2	PR #72443 - 1918306	Crash dump
8/2	PR #72851 - 1917737	Crash dump
8/2	PR #72143 - 1917321	Crash dump
~~7/30~~	~~PR #72934 - 1913487~~	No clear error - likely infra hiccup
~~7/29~~	~~PR #49576 - 1912700~~	2x Segmentation fault - most likely caused by the PR
7/29	PR #73009 - 1912477	Crash dump
~~7/29~~	~~PR #49576 - 1912185~~	2x Segmentation fault - most likely caused by the PR
~~7/29~~	~~PR #49576 - 1910538~~	2x Segmentation fault - most likely caused by the PR
7/28	PR #67272 - 1909374	Unhandled Exception
7/28	PR #72762 - 1908782	Unhandled Exception
7/28	PR #69542 - 1908613	Crash dump
7/26	PR #71661 - 1905330	Unhandled Exception
7/26	Rolling run 1904930	Crash dump
7/24	PR #72448 - 1900699	Crash dump
7/24	PR #72732 - 1900156	Crash dump
7/22	PR #72672 - 1897479	Unhandled Exception
7/20	PR #72482 - 1893721	Crash dump
7/20	PR #72529 - 1892945	Crash dump
7/19	PR #72394 - 1891289	Crash dump
7/19	PR #72452 - 1891146	Unhandled Exception
7/16	PR #71827 - 1885040	Crash dump
7/16	PR #72275 - 1885005	Crash dump
7/15	PR #72021 - 1882959	Crash dump
4/5-7/14	Logs not available

Unhandled Exception

Example: Console log

16:13:46.844 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation_REF op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation_REF op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
   at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Crash dump

Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1789473

Console output: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/console.b7c2b95a.log?helixlogtype=result

Relevant log snippet:

22:49:29.737 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fa4c2a98722 - Unknown
	0x7fa4c2a39a2e - Unknown
	0x7fa4c2996d71 - Unknown
	0x7fa4c4e52980 - Unknown
	0x7fa4c29e5494 - Unknown
	0x7fa4bed63df3 - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 26773]
[New LWP 26774]
[New LWP 26775]
[New LWP 26776]
[New LWP 26781]
[New LWP 26785]
[New LWP 26786]
[New LWP 26787]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/datadisks/disk1/work/B4770A02/p/libcoreclr.so.dbg" does not match "/datadisks/disk1/work/B4770A02/p/libcoreclr.so" (CRC mismatch).

0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7fa4c5479740 (LWP 26772) "corerun" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7fa4c13ff700 (LWP 26773) "SGen worker" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7fa4be6cc700 (LWP 26774) "corerun" 0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fa4be4cb700 (LWP 26775) "Finalizer" 0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7fa4b63fe700 (LWP 26776) "corerun" 0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0x7fa4a8194700 (LWP 26781) ".NET Sockets" 0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7fa4a7d8d700 (LWP 26785) ".NET ThreadPool" 0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0x7fa4a7b8c700 (LWP 26786) ".NET ThreadPool" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7fa4a7b4b700 (LWP 26787) ".NET ThreadPool" 0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Thread 9 (Thread 0x7fa4a7b4b700 (LWP 26787)):
#0  0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fa4c2a98867 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2a39a2e in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2996d71 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  <signal handler called>
#5  0x00007fa4c29e5494 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4bed63df3 in System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_INT_SetExistingTaskResult_System_Threading_Tasks_Task_1_TResult_INT_TResult_INT () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00005627026707f8 in ?? ()
#8  0x00007fa4c143f198 in ?? ()
#9  0x00007fa4c143f620 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fa4a7b8c700 (LWP 26786)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa49c002c40, cond=0x7fa49c002c68) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa49c002c68, mutex=0x7fa49c002c40) at pthread_cond_wait.c:655
#3  0x00007fa4bcf3d1fc in SystemNative_LowLevelMonitor_Wait (monitor=0x7fa49c002c40) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x00007fa4bf0ee1e6 in wrapper_managed_to_native_Interop_Sys_LowLevelMonitor_Wait_intptr () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#5  0x00007fa4bea252df in System_Threading_WaitSubsystem_ThreadWaitInfo_Wait_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4bea2688e in System_Threading_WaitSubsystem_WaitableObject_Wait_Locked_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4bea265fb in System_Threading_WaitSubsystem_WaitableObject_Wait_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#8  0x00007fa4a7b8bbb0 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fa4a7d8d700 (LWP 26785)):
#0  0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7fa4a7d8c7a8, mutex=0x7fa4a000b800, cond=0x7fa4a7d8c800) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7fa4a7d8c800, mutex=0x7fa4a000b800, abstime=0x7fa4a7d8c7a8) at pthread_cond_wait.c:667
#3  0x00007fa4c292a474 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c292fb74 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4bef1e863 in wrapper_managed_to_native_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal_intptr_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4be9fb9cd in System_Threading_LowLevelLifoSemaphore_WaitForSignal_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4c1441d08 in ?? ()
#8  0x0000000000000002 in ?? ()
#9  0xfffeffffffffffff in ?? ()
#10 0x0000000000004e20 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#13 0x00007fa4be9fb6db in System_Threading_LowLevelLifoSemaphore_Wait_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#14 0x00007fa4c1441998 in ?? ()
#15 0x00007fa4a7d8c9d0 in ?? ()
#16 0x00007fa4a7d8ca08 in ?? ()
#17 0x00007fa4a7d8cdc8 in ?? ()
#18 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#19 0x00007fa4a7d8ca80 in ?? ()
#20 0x00007fa4bea20b1d in System_Threading_PortableThreadPool_WorkerThread_WorkerThreadStart () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0x7fa4a8194700 (LWP 26781)):
#0  0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007fa4bcf37f9e in WaitForSocketEventsInner (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:2717
#2  0x00007fa4bcf37ebf in SystemNative_WaitForSocketEvents (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3017
#3  0x00007fa4a8eef485 in wrapper_managed_to_native_Interop_Sys_WaitForSocketEvents_intptr_Interop_Sys_SocketEvent__int_ (param0=13, param1=94725539321392, param2=140345171589588) at /_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:2264
#4  0x00007fa4a8e30f1c in System_Net_Sockets_SocketAsyncEngine_EventLoop () at /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:184
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fa4b63fe700 (LWP 26776)):
#0  0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x00007fa4bcf3c60e in SignalHandlerLoop (arg=0x562702331280) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x00007fa4c4e476db in start_thread (arg=0x7fa4b63fe700) at pthread_create.c:463
#3  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa4be4cb700 (LWP 26775)):
#0  0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fa4c4e508b8 in __new_sem_wait_slow (sem=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fa4c29011c6 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28da2ba in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28da149 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c4e476db in start_thread (arg=0x7fa4be4cb700) at pthread_create.c:463
#7  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa4be6cc700 (LWP 26774)):
#0  0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fa4c2b5f84a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2b5cd45 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2b5b5a9 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c2b5ec61 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c4e476db in start_thread (arg=0x7fa4be6cc700) at pthread_create.c:463
#6  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa4c13ff700 (LWP 26773)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa4c40d9578, cond=0x7fa4c40d95a0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa4c40d95a0, mutex=0x7fa4c40d9578) at pthread_cond_wait.c:655
#3  0x00007fa4c297ee73 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c4e476db in start_thread (arg=0x7fa4c13ff700) at pthread_create.c:463
#5  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa4c5479740 (LWP 26772)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562701fd5138, cond=0x562701fd5160) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562701fd5160, mutex=0x562701fd5138) at pthread_cond_wait.c:655
#3  0x00007fa4c292a490 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28df52a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28df3a8 in mono_w32handle_wait_one () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c2902bb4 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#7  0x00007fa4c2892b93 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#8  0x00007fa4bef1d79d in wrapper_managed_to_native_System_Threading_Monitor_Monitor_wait_object_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#9  0x00007fa4be9f5986 in System_Threading_Monitor_ObjWait_int_object () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#10 0x0000000000000001 in ?? ()
#11 0x00007fa4c1440a30 in ?? ()
#12 0x0000000000000000 in ?? ()

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa4c29e5494):0x7fa4c29e5484  57 01 48 8b 40 38 48 8b 80 98 00 00 00 48 8b 38  W.H.@8H......H.8
0x7fa4c29e5494  41 8b 46 10 8d 34 c5 08 00 00 00 e8 6c 91 f1 ff  A.F..4......l...
0x7fa4c29e54a4  48 89 04 24 41 83 7e 10 00 0f 8e 80 00 00 00 48  H..$A.~........H
0x7fa4c29e54b4  63 c5 48 89 44 24 10 f7 d8 48 89 44 24 08 48 8d  c.H.D$...H.D$.H.

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
JIT/Methodical/Methodical_others/Methodical_others.sh: line 382: 26772 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Crashdump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/core.1000.26772?helixlogtype=result

Runfo Tracking Issue: methodical_others work item

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2022-05-25T23:16:37Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2022-05-25T23:17:55Z

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1789473

Console output: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/console.b7c2b95a.log?helixlogtype=result

Relevant log snippet:

22:49:29.737 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7fa4c2a98722 - Unknown
	0x7fa4c2a39a2e - Unknown
	0x7fa4c2996d71 - Unknown
	0x7fa4c4e52980 - Unknown
	0x7fa4c29e5494 - Unknown
	0x7fa4bed63df3 - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 26773]
[New LWP 26774]
[New LWP 26775]
[New LWP 26776]
[New LWP 26781]
[New LWP 26785]
[New LWP 26786]
[New LWP 26787]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/datadisks/disk1/work/B4770A02/p/libcoreclr.so.dbg" does not match "/datadisks/disk1/work/B4770A02/p/libcoreclr.so" (CRC mismatch).

0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7fa4c5479740 (LWP 26772) "corerun" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7fa4c13ff700 (LWP 26773) "SGen worker" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7fa4be6cc700 (LWP 26774) "corerun" 0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7fa4be4cb700 (LWP 26775) "Finalizer" 0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7fa4b63fe700 (LWP 26776) "corerun" 0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0x7fa4a8194700 (LWP 26781) ".NET Sockets" 0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7fa4a7d8d700 (LWP 26785) ".NET ThreadPool" 0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0x7fa4a7b8c700 (LWP 26786) ".NET ThreadPool" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7fa4a7b4b700 (LWP 26787) ".NET ThreadPool" 0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Thread 9 (Thread 0x7fa4a7b4b700 (LWP 26787)):
#0  0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fa4c2a98867 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2a39a2e in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2996d71 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  <signal handler called>
#5  0x00007fa4c29e5494 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4bed63df3 in System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_INT_SetExistingTaskResult_System_Threading_Tasks_Task_1_TResult_INT_TResult_INT () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00005627026707f8 in ?? ()
#8  0x00007fa4c143f198 in ?? ()
#9  0x00007fa4c143f620 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fa4a7b8c700 (LWP 26786)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa49c002c40, cond=0x7fa49c002c68) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa49c002c68, mutex=0x7fa49c002c40) at pthread_cond_wait.c:655
#3  0x00007fa4bcf3d1fc in SystemNative_LowLevelMonitor_Wait (monitor=0x7fa49c002c40) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x00007fa4bf0ee1e6 in wrapper_managed_to_native_Interop_Sys_LowLevelMonitor_Wait_intptr () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#5  0x00007fa4bea252df in System_Threading_WaitSubsystem_ThreadWaitInfo_Wait_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4bea2688e in System_Threading_WaitSubsystem_WaitableObject_Wait_Locked_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4bea265fb in System_Threading_WaitSubsystem_WaitableObject_Wait_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#8  0x00007fa4a7b8bbb0 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fa4a7d8d700 (LWP 26785)):
#0  0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7fa4a7d8c7a8, mutex=0x7fa4a000b800, cond=0x7fa4a7d8c800) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7fa4a7d8c800, mutex=0x7fa4a000b800, abstime=0x7fa4a7d8c7a8) at pthread_cond_wait.c:667
#3  0x00007fa4c292a474 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c292fb74 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4bef1e863 in wrapper_managed_to_native_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal_intptr_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6  0x00007fa4be9fb9cd in System_Threading_LowLevelLifoSemaphore_WaitForSignal_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7  0x00007fa4c1441d08 in ?? ()
#8  0x0000000000000002 in ?? ()
#9  0xfffeffffffffffff in ?? ()
#10 0x0000000000004e20 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#13 0x00007fa4be9fb6db in System_Threading_LowLevelLifoSemaphore_Wait_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#14 0x00007fa4c1441998 in ?? ()
#15 0x00007fa4a7d8c9d0 in ?? ()
#16 0x00007fa4a7d8ca08 in ?? ()
#17 0x00007fa4a7d8cdc8 in ?? ()
#18 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#19 0x00007fa4a7d8ca80 in ?? ()
#20 0x00007fa4bea20b1d in System_Threading_PortableThreadPool_WorkerThread_WorkerThreadStart () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0x7fa4a8194700 (LWP 26781)):
#0  0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007fa4bcf37f9e in WaitForSocketEventsInner (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:2717
#2  0x00007fa4bcf37ebf in SystemNative_WaitForSocketEvents (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3017
#3  0x00007fa4a8eef485 in wrapper_managed_to_native_Interop_Sys_WaitForSocketEvents_intptr_Interop_Sys_SocketEvent__int_ (param0=13, param1=94725539321392, param2=140345171589588) at /_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:2264
#4  0x00007fa4a8e30f1c in System_Net_Sockets_SocketAsyncEngine_EventLoop () at /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:184
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fa4b63fe700 (LWP 26776)):
#0  0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x00007fa4bcf3c60e in SignalHandlerLoop (arg=0x562702331280) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x00007fa4c4e476db in start_thread (arg=0x7fa4b63fe700) at pthread_create.c:463
#3  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa4be4cb700 (LWP 26775)):
#0  0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fa4c4e508b8 in __new_sem_wait_slow (sem=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fa4c29011c6 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28da2ba in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28da149 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c4e476db in start_thread (arg=0x7fa4be4cb700) at pthread_create.c:463
#7  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa4be6cc700 (LWP 26774)):
#0  0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007fa4c2b5f84a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2  0x00007fa4c2b5cd45 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3  0x00007fa4c2b5b5a9 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c2b5ec61 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c4e476db in start_thread (arg=0x7fa4be6cc700) at pthread_create.c:463
#6  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa4c13ff700 (LWP 26773)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa4c40d9578, cond=0x7fa4c40d95a0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa4c40d95a0, mutex=0x7fa4c40d9578) at pthread_cond_wait.c:655
#3  0x00007fa4c297ee73 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c4e476db in start_thread (arg=0x7fa4c13ff700) at pthread_create.c:463
#5  0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa4c5479740 (LWP 26772)):
#0  0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x562701fd5138, cond=0x562701fd5160) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x562701fd5160, mutex=0x562701fd5138) at pthread_cond_wait.c:655
#3  0x00007fa4c292a490 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4  0x00007fa4c28df52a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5  0x00007fa4c28df3a8 in mono_w32handle_wait_one () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6  0x00007fa4c2902bb4 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#7  0x00007fa4c2892b93 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#8  0x00007fa4bef1d79d in wrapper_managed_to_native_System_Threading_Monitor_Monitor_wait_object_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#9  0x00007fa4be9f5986 in System_Threading_Monitor_ObjWait_int_object () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#10 0x0000000000000001 in ?? ()
#11 0x00007fa4c1440a30 in ?? ()
#12 0x0000000000000000 in ?? ()

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa4c29e5494):0x7fa4c29e5484  57 01 48 8b 40 38 48 8b 80 98 00 00 00 48 8b 38  W.H.@8H......H.8
0x7fa4c29e5494  41 8b 46 10 8d 34 c5 08 00 00 00 e8 6c 91 f1 ff  A.F..4......l...
0x7fa4c29e54a4  48 89 04 24 41 83 7e 10 00 0f 8e 80 00 00 00 48  H..$A.~........H
0x7fa4c29e54b4  63 c5 48 89 44 24 10 f7 d8 48 89 44 24 08 48 8d  c.H.D$...H.D$.H.

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
JIT/Methodical/Methodical_others/Methodical_others.sh: line 382: 26772 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Crashdump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-69813-merge-212c7ffc453c4eb2a8/Methodical_others/1/core.1000.26772?helixlogtype=result

Author:	ericstj
Assignees:	-
Labels:	`area-CodeGen-coreclr`, `blocking-clean-ci`, `untriaged`
Milestone:	-

ericstj · 2022-05-25T23:19:43Z

Changing the tag as this happened in llvmfullaot

eerhardt · 2022-07-06T00:45:35Z

Occurred here again:

https://dev.azure.com/dnceng/public/_build/results?buildId=1861609&view=logs&j=ac444259-14dc-5e9e-4846-f4aabcc22cf0&t=8b5dcf28-4222-5bb1-452d-824912c72244&l=29

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71501-merge-6f012c604d60484ca0/Methodical_others/1/console.d462ba5d.log?helixlogtype=result

karelz · 2022-08-05T18:16:00Z

This happens more than 1x per day. We should disable the test ASAP. @vargaz can you please do it? Thanks!

Re: dotnet#69832.

…#73511) Re: #69832.

karelz · 2022-08-09T12:26:26Z

Test disabled in PR #73511 - not blocking CI anymore.
Note: Disabled only on Mono interpreter by accident.

SamMonoRT · 2022-08-10T16:41:56Z

Assigning to @jandupej and moving to 8.0.0 We will triage the fix and consider backporting to re-enable the tests.

karelz · 2022-08-26T09:43:39Z

Test disabled on all Mono configs in main (8.0) in PR #74265 and in 7.0 (RC2) in PR #74636

Note: It happened in release/7.0 rolling run 1967536 on 8/25 - therefore backporting the disabling test into 7.0 branch.

ghost added the untriaged New issue has not been triaged by the area owner label May 25, 2022

ericstj added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels May 25, 2022

ericstj mentioned this issue May 25, 2022

Update 'main' branding to release/7.0-preview6 #69813

Merged

ericstj added area-Codegen-AOT-mono and removed area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI labels May 25, 2022

runfoapp bot mentioned this issue May 25, 2022

Infrastructure - Status/Health #702

Closed

jakobbotsch mentioned this issue Jun 9, 2022

methodical_others test failing/timing out on multiple runtimes, multiple OSes #70305

Closed

lewing assigned vargaz Jun 9, 2022

eerhardt mentioned this issue Jul 6, 2022

Use crypto.subtle for AES on Browser WASM #71501

Merged

marek-safar removed the untriaged New issue has not been triaged by the area owner label Jul 20, 2022

marek-safar added this to the 7.0.0 milestone Jul 20, 2022

vargaz added a commit to vargaz/runtime that referenced this issue Aug 6, 2022

[mono] Disable JIT/Methodical/Coverage/copy_prop_byref_to_native_int.

08bb397

Re: dotnet#69832.

vargaz mentioned this issue Aug 6, 2022

[mono] Disable JIT/Methodical/Coverage/copy_prop_byref_to_native_int. #73511

Merged

vargaz removed their assignment Aug 6, 2022

vargaz added a commit that referenced this issue Aug 7, 2022

[mono] Disable JIT/Methodical/Coverage/copy_prop_byref_to_native_int. (…

9929753

…#73511) Re: #69832.

karelz added disabled-test The test is disabled in source code against the issue and removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels Aug 9, 2022

radical mentioned this issue Aug 9, 2022

CI: Add support in yml for detecting darc dependency changes in eng/Version.Details.xml #73435

Merged

SamMonoRT assigned jandupej Aug 10, 2022

SamMonoRT modified the milestones: 7.0.0, 8.0.0 Aug 10, 2022

jakobbotsch mentioned this issue Aug 17, 2022

crash in JIT/Methodical/Coverage/copy_prop_byref_to_native_int on Mono #74049

Closed

This was referenced Aug 26, 2022

[mono] Disable JIT/Methodical/Coverage/copy_prop_byref_to_native_int on all configs #74265

Merged

[release/7.0] [mono] Disable JIT/Methodical/Coverage/copy_prop_byref_to_native_int on all configs #74636

Merged

jandupej modified the milestones: 8.0.0, 9.0.0 Jul 12, 2023

SamMonoRT assigned matouskozak and unassigned jandupej Oct 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

ericstj commented May 25, 2022 •

edited by runfoapp bot

Loading

dotnet-issue-labeler bot commented May 25, 2022

ghost commented May 25, 2022

ericstj commented May 25, 2022

eerhardt commented Jul 6, 2022

karelz commented Aug 5, 2022

karelz commented Aug 9, 2022 •

edited

Loading

SamMonoRT commented Aug 10, 2022

karelz commented Aug 26, 2022

Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

Comments

ericstj commented May 25, 2022 • edited by runfoapp bot Loading

Unhandled Exception

Crash dump

dotnet-issue-labeler bot commented May 25, 2022

ghost commented May 25, 2022

ericstj commented May 25, 2022

eerhardt commented Jul 6, 2022

karelz commented Aug 5, 2022

karelz commented Aug 9, 2022 • edited Loading

SamMonoRT commented Aug 10, 2022

karelz commented Aug 26, 2022

ericstj commented May 25, 2022 •

edited by runfoapp bot

Loading

karelz commented Aug 9, 2022 •

edited

Loading