Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MulticastDelegate.CombineImpl throws "double free or corruption" or "pointer being freed was not allocated" errors #11488

Closed
akoeplinger opened this issue Oct 31, 2018 · 7 comments

Comments

@akoeplinger
Copy link
Member

@akoeplinger akoeplinger commented Oct 31, 2018

Steps to Reproduce

  1. Run System.ServiceModel or System.ServiceModel.Web tests (seen it happen both on Linux ARMv5 and OSX x64, see stack traces below)
  2. It occasionally crashes with the mentioned malloc errors

Current Behavior

From https://jenkins.mono-project.com/job/test-mono-pull-request-armel/17583/:

*** Error in `/home/builder/jenkins/workspace/test-mono-pull-request-armel/mono/mini/mono': double free or corruption (fasttop): 0xf2d05f40 ***
Stacktrace:

  at <unknown> <0xffffffff>
  at System.MulticastDelegate.CombineImpl (System.Delegate) [0x0000c] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/corlib/System/MulticastDelegate.cs:162
  at System.Delegate.Combine (System.Delegate,System.Delegate) [0x0004f] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/corlib/System/Delegate.cs:552
  at System.ServiceModel.Channels.CommunicationObject.add_Closed (System.EventHandler) [0x0000b] in <3b115f8fd6d1402a8d7938d244ec3533>:0
  at System.ServiceModel.Dispatcher.ListenerLoopManager/<>c__DisplayClass23_0.<ChannelAccepted>b__0 (object,System.EventArgs) [0x00017] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:545
  at System.ServiceModel.Channels.CommunicationObject.OnOpened () [0x0000f] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:269
  at System.ServiceModel.Channels.CommunicationObject.ProcessOpened () [0x00011] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:257
  at System.ServiceModel.Channels.CommunicationObject.Open (System.TimeSpan) [0x0000d] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:171
  at System.ServiceModel.Channels.CommunicationObject.Open () [0x00000] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:164
  at System.ServiceModel.Dispatcher.ListenerLoopManager.ChannelAccepted (System.ServiceModel.Channels.IChannel) [0x00080] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:552
  at System.ServiceModel.Dispatcher.ListenerLoopManager/<>c__DisplayClass15_0`1<TChannel_REF>.<CreateAcceptor>b__0 (System.IAsyncResult) [0x00000] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:385
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x00065] in <53fa4e330ec54f158d9e1e8bf9454259>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Runtime.Remoting.Messaging.AsyncResult.Invoke (System.Runtime.Remoting.Messaging.AsyncResult) <0x0003f>
  at System.Runtime.Remoting.Messaging.AsyncResult.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/corlib/System.Runtime.Remoting.Messaging/AsyncResult.cs:184
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:872
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1227
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x00028] in <53fa4e330ec54f158d9e1e8bf9454259>:0
/proc/self/maps:
00787000-00b5b000 r-xp 00000000 08:02 673365                             /home/builder/jenkins/workspace/test-mono-pull-request-armel/mono/mini/mono-sgen
00b6b000-00b6f000 r--p 003d4000 08:02 673365                             /home/builder/jenkins/workspace/test-mono-pull-request-armel/mono/mini/mono-sgen
00b6f000-00b73000 rw-p 003d8000 08:02 673365                             /home/builder/jenkins/workspace/test-mono-pull-request-armel/mono/mini/mono-sgen
00b73000-00bf9000 rw-p 00000000 00:00 0 
01034000-015b1000 rw-p 00000000 00:00 0                                  [heap]
f2900000-f2921000 rw-p 00000000 00:00 0 
f2921000-f2a00000 ---p 00000000 00:00 0 
f2a00000-f2a21000 rw-p 00000000 00:00 0 
f2a21000-f2b00000 ---p 00000000 00:00 0 
f2b00000-f2b01000 ---p 00000000 00:00 0 
f2b01000-f2c00000 rwxp 00000000 00:00 0 
f2c00000-f2c21000 rw-p 00000000 00:00 0 
f2c21000-f2d00000 ---p 00000000 00:00 0 
f2d00000-f2d21000 rw-p 00000000 00:00 0 
f2d21000-f2e00000 ---p 00000000 00:00 0 
f2e00000-f2e01000 ---p 00000000 00:00 0 
f2e01000-f2f00000 rwxp 00000000 00:00 0 
f2f00000-f2f01000 ---p 00000000 00:00 0 
f2f01000-f3000000 rwxp 00000000 00:00 0 
f3000000-f3021000 rw-p 00000000 00:00 0 
f3021000-f3100000 ---p 00000000 00:00 0 
f3100000-f3121000 rw-p 00000000 00:00 0 
f3121000-f3200000 ---p 00000000 00:00 0 
f3200000-f3201000 ---p 00000000 00:00 0 
f3201000-f3300000 rwxp 00000000 00:00 0 
Memory around native instruction pointer (0xf79a5fec):
0xf79a5fdc  00 20 a0 e3 08 30 a0 e3 af 70 a0 e3 00 00 00 ef  . ...0...p......
0xf79a5fec  04 00 a0 e1 01 dc 8d e2 b0 40 bd e8 1e ff 2f e1  .........@..../.
0xf79a5ffc  c3 aa ff eb 14 30 9f e5 03 30 9f e7 00 20 64 e2  .....0...0... d.
0xf79a600c  03 20 80 e7 00 40 e0 e3 ee ff ff ea c8 6c 0f 00  . ...@.......l..

Native stacktrace:

Pkilling 0xf36ff450 from 0xf2fff450
Pkilling 0xf2bff450 from 0xf2fff450
Pkilling 0xf7c274b0 from 0xf2fff450
Pkilling 0xf54c4450 from 0xf2fff450
Pkilling 0xf3ef1450 from 0xf2fff450
Pkilling 0xf33ff450 from 0xf2fff450
Pkilling 0xf32ff450 from 0xf2fff450
Pkilling 0xf3bff450 from 0xf2fff450
Pkilling 0xf3aff450 from 0xf2fff450
Pkilling 0xf2eff450 from 0xf2fff450
Pkilling 0xf37ff450 from 0xf2fff450
Entering thread summarizer pause from 0xf2fff450
Finished thread summarizer pause from 0xf2fff450.

Waiting for dumping threads to resume


Debug info from gdb:

[New LWP 11714]
[New LWP 11715]
[New LWP 11731]
[New LWP 11742]
[New LWP 11743]
[New LWP 11772]
[New LWP 11773]
[New LWP 11775]
[New LWP 11779]
[New LWP 11781]
[New LWP 11782]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabi/libthread_db.so.1".
0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  Id   Target Id         Frame 
* 1    Thread 0xf7c274b0 (LWP 11695) "mono" 0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  2    Thread 0xf73ff450 (LWP 11714) "SGen worker" 0xf7b01760 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  3    Thread 0xf54c4450 (LWP 11715) "Finalizer" 0xf7b04268 in do_futex_wait.constprop () from /lib/arm-linux-gnueabi/libpthread.so.0
  4    Thread 0xf3ef1450 (LWP 11731) "Thread Pool I/O" 0xf7a47a00 in poll () from /lib/arm-linux-gnueabi/libc.so.6
  5    Thread 0xf37ff450 (LWP 11742) "mono" 0xf7b01760 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  6    Thread 0xf36ff450 (LWP 11743) "mono" 0xf7b01760 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  7    Thread 0xf33ff450 (LWP 11772) "mono" 0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  8    Thread 0xf32ff450 (LWP 11773) "Thread Pool Wor" 0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
  9    Thread 0xf2fff450 (LWP 11775) "Thread Pool Wor" 0xf7b068ec in waitpid () from /lib/arm-linux-gnueabi/libpthread.so.0
  10   Thread 0xf3bff450 (LWP 11779) "Thread Pool Wor" 0xf7b044cc in do_futex_wait () from /lib/arm-linux-gnueabi/libpthread.so.0
  11   Thread 0xf2eff450 (LWP 11781) "Thread Pool Wor" 0xf7b044cc in do_futex_wait () from /lib/arm-linux-gnueabi/libpthread.so.0
  12   Thread 0xf2bff450 (LWP 11782) "Thread Pool Wor" 0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0

Thread 12 (Thread 0xf2bff450 (LWP 11782)):
#0  0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0x00a7bd64 in mono_os_cond_timedwait (cond=cond@entry=0x1034d60, mutex=mutex@entry=0x1034d48, timeout_ms=timeout_ms@entry=600000) at mono-os-mutex.c:75
#2  0x009d3100 in mono_coop_cond_timedwait (timeout_ms=600000, mutex=0x1034d48, cond=0x1034d60) at ../../mono/utils/mono-coop-mutex.h:102
#3  mono_w32handle_timedwait_signal_naked (alerted=0xf2bfde50, poll=0, timeout=600000, mutex=0x1034d48, cond=0x1034d60) at w32handle.c:650
#4  mono_w32handle_timedwait_signal_handle (handle_data=0x1034d38, timeout=600000, alerted=alerted@entry=0xf2bfde50, poll=0) at w32handle.c:765
#5  0x009d45cc in mono_w32handle_wait_one (handle=<optimized out>, timeout=600000, timeout@entry=4072660724, alertable=alertable@entry=1) at w32handle.c:883
#6  0x009d4e78 in mono_w32handle_wait_multiple (handles=handles@entry=0xf2bfe3b4, nhandles=nhandles@entry=1, waitall=waitall@entry=0, timeout=4072660724, timeout@entry=600000, alertable=alertable@entry=1, error=0xf2bfe290, error@entry=0xf2bfe288) at w32handle.c:982
#7  0x009b69b4 in ves_icall_System_Threading_WaitHandle_Wait_internal (handles=handles@entry=0xf2bfe3b4, numhandles=numhandles@entry=1, waitall=waitall@entry=0 '\000', timeout=timeout@entry=600000, error=0xf2bfe290) at threads.c:2153
#8  0x0094efe0 in ves_icall_System_Threading_WaitHandle_Wait_internal_raw (a0=0xf2bfe3b4, a1=1, a2=<optimized out>, a3=600000, error=0xf2bfe290) at ../../mono/metadata/icall-def.h:1128
#9  0xf3efb1dc in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 11 (Thread 0xf2eff450 (LWP 11781)):
#0  0xf7b044cc in do_futex_wait () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0xf7b047a8 in __new_sem_wait_slow () from /lib/arm-linux-gnueabi/libpthread.so.0
#2  0x00a1c158 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xb76208 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#3  mono_coop_sem_timedwait (sem=0xb76208 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#4  worker_park () at threadpool-worker-default.c:391
#5  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:490
#6  0x009b9790 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#7  start_wrapper (data=0xf34014d8) at threads.c:1234
#8  0xf7afa2c4 in start_thread () from /lib/arm-linux-gnueabi/libpthread.so.0
#9  0xf7a5101c in ?? () from /lib/arm-linux-gnueabi/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 10 (Thread 0xf3bff450 (LWP 11779)):
#0  0xf7b044cc in do_futex_wait () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0xf7b047a8 in __new_sem_wait_slow () from /lib/arm-linux-gnueabi/libpthread.so.0
#2  0x00a1c158 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xb76208 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#3  mono_coop_sem_timedwait (sem=0xb76208 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#4  worker_park () at threadpool-worker-default.c:391
#5  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:490
#6  0x009b9790 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#7  start_wrapper (data=0xf3804200) at threads.c:1234
#8  0xf7afa2c4 in start_thread () from /lib/arm-linux-gnueabi/libpthread.so.0
#9  0xf7a5101c in ?? () from /lib/arm-linux-gnueabi/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xf2fff450 (LWP 11775)):
#0  0xf7b068ec in waitpid () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0x0088b154 in dump_native_stacktrace (ctx=0xf2ffdad8, ctx@entry=0xab07b4, signal=0xab04a0 "SIGABRT") at mini-posix.c:1083
#2  0x0088b468 in mono_dump_native_crash_info (signal=signal@entry=0xab04a0 "SIGABRT", ctx=0xab07b4, ctx@entry=0xf2ffdad8, info=info@entry=0xf2ffda58) at mini-posix.c:1112
#3  0x00843a2c in mono_handle_native_crash (signal=0xab04a0 "SIGABRT", ctx=ctx@entry=0xf2ffdad8, info=info@entry=0xf2ffda58) at mini-exceptions.c:3196
#4  0x0088a57c in sigabrt_signal_handler (_dummy=6, _info=0xf2ffda58, context=0xf2ffdad8) at mini-posix.c:223
#5  <signal handler called>
#6  0xf79a5fec in raise () from /lib/arm-linux-gnueabi/libc.so.6
#7  0xf79a7488 in abort () from /lib/arm-linux-gnueabi/libc.so.6
#8  0x00000020 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 8 (Thread 0xf32ff450 (LWP 11773)):
#0  0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0x00a7bd64 in mono_os_cond_timedwait (cond=cond@entry=0x1034580, mutex=mutex@entry=0x1034568, timeout_ms=timeout_ms@entry=599999) at mono-os-mutex.c:75
#2  0x009d3100 in mono_coop_cond_timedwait (timeout_ms=599999, mutex=0x1034568, cond=0x1034580) at ../../mono/utils/mono-coop-mutex.h:102
#3  mono_w32handle_timedwait_signal_naked (alerted=0xf32fdf88, poll=0, timeout=599999, mutex=0x1034568, cond=0x1034580) at w32handle.c:650
#4  mono_w32handle_timedwait_signal_handle (handle_data=0x1034558, timeout=599999, alerted=alerted@entry=0xf32fdf88, poll=0) at w32handle.c:765
#5  0x009d45cc in mono_w32handle_wait_one (handle=<optimized out>, timeout=599999, timeout@entry=0, alertable=alertable@entry=1) at w32handle.c:883
#6  0x009d4e78 in mono_w32handle_wait_multiple (handles=handles@entry=0xf32fe4ec, nhandles=nhandles@entry=1, waitall=waitall@entry=0, timeout=0, timeout@entry=599999, alertable=alertable@entry=1, error=0xf32fe3c8, error@entry=0xf32fe3c0) at w32handle.c:982
#7  0x009b69b4 in ves_icall_System_Threading_WaitHandle_Wait_internal (handles=handles@entry=0xf32fe4ec, numhandles=numhandles@entry=1, waitall=waitall@entry=0 '\000', timeout=timeout@entry=599999, error=0xf32fe3c8) at threads.c:2153
#8  0x0094efe0 in ves_icall_System_Threading_WaitHandle_Wait_internal_raw (a0=0xf32fe4ec, a1=1, a2=<optimized out>, a3=599999, error=0xf32fe3c8) at ../../mono/metadata/icall-def.h:1128
#9  0xf3efb1dc in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xf33ff450 (LWP 11772)):
#0  0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0x00a7bd64 in mono_os_cond_timedwait (cond=0xbf8120 <sleep_cond>, mutex=0xbf8108 <sleep_mutex>, timeout_ms=500, timeout_ms@entry=3475441331) at mono-os-mutex.c:75
#2  0x00a85acc in mono_coop_cond_timedwait (cond=0xbf8120 <sleep_cond>, mutex=0xbf8108 <sleep_mutex>, timeout_ms=3475441331) at ../../mono/utils/mono-coop-mutex.h:102
#3  sleep_interruptable (alerted=0xf33fedc4, ms=500) at mono-threads.c:1572
#4  mono_thread_info_sleep (ms=ms@entry=500, alerted=alerted@entry=0xf33fedc4) at mono-threads.c:1604
#5  0x00a1d0cc in monitor_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:708
#6  0x009b9790 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#7  start_wrapper (data=0xf3910b80) at threads.c:1234
#8  0xf7afa2c4 in start_thread () from /lib/arm-linux-gnueabi/libpthread.so.0
Backtrace stopped: Cannot access memory at address 0x16

Thread 6 (Thread 0xf36ff450 (LWP 11743)):
#0  0xf7b01760 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0xf7b013cc in pthread_cond_destroy@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#2  0x01034c78 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (Thread 0xf37ff450 (LWP 11742)):
#0  0xf7b01760 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0xf7b013cc in pthread_cond_destroy@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#2  0x00000046 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (Thread 0xf3ef1450 (LWP 11731)):
#0  0xf7a47a00 in poll () from /lib/arm-linux-gnueabi/libc.so.6
#1  0xf7a479f4 in poll () from /lib/arm-linux-gnueabi/libc.so.6
#2  0xf3ef0d84 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (Thread 0xf54c4450 (LWP 11715)):
#0  0xf7b04268 in do_futex_wait.constprop () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0xf7b043e4 in __new_sem_wait_slow.constprop.0 () from /lib/arm-linux-gnueabi/libpthread.so.0
#2  0x00a05f30 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0xb75cd0 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:209
#3  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0xb75cd0 <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:43
#4  finalizer_thread (unused=unused@entry=0x0) at gc.c:920
#5  0x009b9790 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#6  start_wrapper (data=0x10b5a08) at threads.c:1234
#7  0xf7afa2c4 in start_thread () from /lib/arm-linux-gnueabi/libpthread.so.0
#8  0xf7a5101c in ?? () from /lib/arm-linux-gnueabi/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (Thread 0xf73ff450 (LWP 11714)):
#0  0xf7b01760 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0xf7b013cc in pthread_cond_destroy@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#2  0x00000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0xf7c274b0 (LWP 11695)):
#0  0xf7b01b6c in pthread_cond_timedwait@@GLIBC_2.4 () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0x00a7bd64 in mono_os_cond_timedwait (cond=cond@entry=0x1034b20, mutex=mutex@entry=0x1034b08, timeout_ms=timeout_ms@entry=59894) at mono-os-mutex.c:75
#2  0x009d3100 in mono_coop_cond_timedwait (timeout_ms=59894, mutex=0x1034b08, cond=0x1034b20) at ../../mono/utils/mono-coop-mutex.h:102
#3  mono_w32handle_timedwait_signal_naked (alerted=0xffc55ce8, poll=0, timeout=59894, mutex=0x1034b08, cond=0x1034b20) at w32handle.c:650
#4  mono_w32handle_timedwait_signal_handle (handle_data=0x1034af8, timeout=59894, alerted=alerted@entry=0xffc55ce8, poll=0) at w32handle.c:765
#5  0x009d45cc in mono_w32handle_wait_one (handle=<optimized out>, timeout=59894, timeout@entry=4291124660, alertable=alertable@entry=1) at w32handle.c:883
#6  0x009d4e78 in mono_w32handle_wait_multiple (handles=handles@entry=0xffc5624c, nhandles=nhandles@entry=1, waitall=32, waitall@entry=0, timeout=4291124660, timeout@entry=59894, alertable=alertable@entry=1, error=0xffc56128, error@entry=0xffc56120) at w32handle.c:982
#7  0x009b69b4 in ves_icall_System_Threading_WaitHandle_Wait_internal (handles=handles@entry=0xffc5624c, numhandles=numhandles@entry=1, waitall=waitall@entry=0 '\000', timeout=timeout@entry=59894, error=0xffc56128) at threads.c:2153
#8  0x0094efe0 in ves_icall_System_Threading_WaitHandle_Wait_internal_raw (a0=0xffc5624c, a1=1, a2=<optimized out>, a3=59894, error=0xffc56128) at ../../mono/metadata/icall-def.h:1128
#9  0xf3efb1dc in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

Aborted (core dumped)
make[2]: *** [run-test-lib] Error 1
../../build/tests.make:239: recipe for target 'run-test-lib' failed
make[2]: Leaving directory '/home/builder/jenkins/workspace/test-mono-pull-request-armel/mcs/class/System.ServiceModel.Web'
../../build/rules.make:209: recipe for target 'do-run-test' failed
make[1]: *** [do-run-test] Error 1

From https://jenkins.mono-project.com/job/test-mono-pull-request-amd64-osx/16049

***** MonoTests.Features.Serialization.AsyncPatternTest.TestAsync
mono(4692,0x700005dcc000) malloc: *** error for object 0x7ff0a4ca0fa0: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Stacktrace:

  at <unknown> <0xffffffff>
  at System.MulticastDelegate.CombineImpl (System.Delegate) [0x0000c] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/corlib/System/MulticastDelegate.cs:162
  at System.Delegate.Combine (System.Delegate,System.Delegate) [0x0004f] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/corlib/System/Delegate.cs:552
  at System.ServiceModel.Channels.CommunicationObject.add_Closed (System.EventHandler) [0x0000b] in <34cfdab9c094499f8f3668eba4bd6933>:0
  at System.ServiceModel.Dispatcher.ListenerLoopManager/<>c__DisplayClass23_0.<ChannelAccepted>b__0 (object,System.EventArgs) [0x00017] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:545
  at System.ServiceModel.Channels.CommunicationObject.OnOpened () [0x0000f] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:269
  at System.ServiceModel.Channels.CommunicationObject.ProcessOpened () [0x00011] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:257
  at System.ServiceModel.Channels.CommunicationObject.Open (System.TimeSpan) [0x0000d] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:171
  at System.ServiceModel.Channels.CommunicationObject.Open () [0x00000] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Channels/CommunicationObject.cs:164
  at System.ServiceModel.Dispatcher.ListenerLoopManager.ChannelAccepted (System.ServiceModel.Channels.IChannel) [0x00080] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:552
  at System.ServiceModel.Dispatcher.ListenerLoopManager/<>c__DisplayClass15_0`1<TChannel_REF>.<CreateAcceptor>b__0 (System.IAsyncResult) [0x00000] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:385
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002e] in <5f2055e796d548799c057e9227372de5>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Runtime.Remoting.Messaging.AsyncResult.Invoke (System.Runtime.Remoting.Messaging.AsyncResult) [0x00013] in <5f2055e796d548799c057e9227372de5>:0
  at System.Runtime.Remoting.Messaging.AsyncResult.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/corlib/System.Runtime.Remoting.Messaging/AsyncResult.cs:210
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:872
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1227
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002b] in <5f2055e796d548799c057e9227372de5>:0
Memory around native instruction pointer (0x7fff978dad42):
0x7fff978dad32  ff ff c3 90 90 90 b8 48 01 00 02 49 89 ca 0f 05  .......H...I....
0x7fff978dad42  73 08 48 89 c7 e9 63 8f ff ff c3 90 90 90 b8 4c  s.H...c........L
0x7fff978dad52  01 00 02 49 89 ca 0f 05 73 08 48 89 c7 e9 4b 8f  ...I....s.H...K.
0x7fff978dad62  ff ff c3 90 90 90 b8 49 01 00 02 49 89 ca 0f 05  .......I...I....

Native stacktrace:

	0   mono                                0x0000000108d397e2 mono_dump_native_crash_info + 898
	1   mono                                0x0000000108cde8f3 mono_handle_native_crash + 195
	2   libsystem_platform.dylib            0x00007fff979bbb3a _sigtramp + 26
	3   mono                                0x0000000108f38956 monoeg_malloc + 22
	4   libsystem_c.dylib                   0x00007fff97840420 abort + 129
	5   libsystem_malloc.dylib              0x00007fff9792ffe7 szone_size + 0
	6   mono                                0x0000000108c2ffe5 mini_method_compile + 3173
	7   mono                                0x0000000108c3315c mono_jit_compile_method_inner + 76
	8   mono                                0x0000000108c36b39 mono_jit_compile_method_with_opt + 2665
	9   mono                                0x0000000108c367a6 mono_jit_compile_method_with_opt + 1750
	10  mono                                0x0000000108ce1754 common_call_trampoline + 1364
	11  mono                                0x0000000108ce1199 mono_magic_trampoline + 105
	12  mono                                0x0000000108ce1e64 mono_aot_trampoline + 196
	13  ???                                 0x000000010920fb13 0x0 + 4448123667
	14  mscorlib.dll.dylib                  0x000000010ada51f7 System_MulticastDelegate_CombineImpl_System_Delegate + 103
	15  ???                                 0x000000010e6f712b 0x0 + 4537151787
	16  ???                                 0x000000010e8c2953 0x0 + 4539033939
	17  ???                                 0x00000001092e9c44 0x0 + 4449016900
	18  mono                                0x0000000108c3b0d4 mono_jit_runtime_invoke + 1668
	19  mono                                0x0000000108e488b4 do_runtime_invoke + 84
	20  mono                                0x0000000108e5165a ves_icall_System_Runtime_Remoting_Messaging_AsyncResult_Invoke + 378
	21  ???                                 0x000000010e8bc4c1 0x0 + 4539008193
	22  mscorlib.dll.dylib                  0x000000010ade423d System_Threading__ThreadPoolWaitCallback_PerformWaitCallback + 29
	23  mono                                0x0000000108c3b0d4 mono_jit_runtime_invoke + 1668
	24  mono                                0x0000000108e488b4 do_runtime_invoke + 84
	25  mono                                0x0000000108e70d12 worker_callback + 914
	26  mono                                0x0000000108daae7c worker_thread + 348
	27  mono                                0x0000000108e6e2b3 start_wrapper + 707
	28  libsystem_pthread.dylib             0x00007fff979c593b _pthread_body + 180
	29  libsystem_pthread.dylib             0x00007fff979c5887 _pthread_body + 0
	30  libsystem_pthread.dylib             0x00007fff979c508d thread_start + 13
Pkilling 0x7000061d2000 from 0x700005dcc000
Pkilling 0x700005fcf000 from 0x700005dcc000
Pkilling 0x700005bc9000 from 0x700005dcc000
Pkilling 0x7000059c6000 from 0x700005dcc000
Pkilling 0x7fffa071e3c0 from 0x700005dcc000
Pkilling 0x7000057c3000 from 0x700005dcc000
Pkilling 0x7000069de000 from 0x700005dcc000
Pkilling 0x7000055c0000 from 0x700005dcc000
Pkilling 0x7000067db000 from 0x700005dcc000
Pkilling 0x7000053bd000 from 0x700005dcc000
Pkilling 0x7000065d8000 from 0x700005dcc000
Pkilling 0x700004e2e000 from 0x700005dcc000
Pkilling 0x7000051ba000 from 0x700005dcc000
Pkilling 0x7000063d5000 from 0x700005dcc000
Entering thread summarizer pause from 0x700005dcc000
Finished thread summarizer pause from 0x700005dcc000.
/Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/scripts/ci/babysitter: Command `make` timed out
/Users/builder/jenkins/workspace/test-mono-pull-request-amd64-osx/scripts/ci/babysitter: Saw timeout in test case MonoTests.Features.Serialization.AsyncPatternTest.TestAsync (never allowed). Will halt testing.
*** end(900): System.ServiceModel: \e[43mUnstable\e[0m

Expected Behavior

No crash.

On which platforms did you notice this

[ X ] macOS
[ X ] Linux
[ ] Windows

Version Used: master

@akoeplinger

This comment has been minimized.

Copy link
Member Author

@akoeplinger akoeplinger commented Oct 31, 2018

Hmm looks like I see this in the Helix environment where I'm working too, though not in MulticastDelegate. Maybe it's more remoting related?

X86:

*** Error in `/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen': double free or corruption (fasttop): 0x583c6b28 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6738a)[0xf75ad38a]
/lib/i386-linux-gnu/libc.so.6(+0x6dfc7)[0xf75b3fc7]
/lib/i386-linux-gnu/libc.so.6(+0x6e806)[0xf75b4806]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(monoeg_g_free+0x20)[0x568b4f70]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(mono_bitset_free+0x26)[0x5689f966]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x11d367)[0x566be367]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x11f45e)[0x566c045e]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x2ea72)[0x565cfa72]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x2ee20)[0x565cfe20]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0xbfde0)[0x56660de0]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0xc0738)[0x56661738]
[0xf73ab188]
[0xf454dee0]
[0xf7099e24]
[0xf7099c1c]
[0xf7099afc]
[0xf709989c]
[0xf7099634]
[0xf70994dd]
[0xf709948e]
[0xf7099374]
[0xf70c0d40]
[0xf70bb5f0]
[0xf70b9f6c]
[0xf70b9d84]
[0xf70b9d14]
[0xf70b9cdc]
[0xf70b9bac]
[0xf70b97c8]
[0xf70b2b3c]
[0xf70b2610]
[0xf70b26bb]
/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x30470)[0x565d1470]
======= Memory map: ========
565a1000-56a9d000 r-xp 00000000 08:01 134596                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen
56a9e000-56aa2000 r--p 004fc000 08:01 134596                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen
56aa2000-56aa5000 rw-p 00500000 08:01 134596                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen
56aa5000-56b2c000 rw-p 00000000 00:00 0 
58250000-58413000 rw-p 00000000 00:00 0                                  [heap]
f43ff000-f4408000 ---p 00000000 00:00 0 
f4408000-f4500000 rw-p 00000000 00:00 0 
f4500000-f45cb000 rw-p 00000000 00:00 0 
f45cb000-f4600000 ---p 00000000 00:00 0 
f46ff000-f4708000 ---p 00000000 00:00 0 
f4708000-f4800000 rw-p 00000000 00:00 0 
f4800000-f4821000 rw-p 00000000 00:00 0 
f4821000-f4900000 ---p 00000000 00:00 0 
f498d000-f4996000 ---p 00000000 00:00 0 
f4996000-f4a8e000 rw-p 00000000 00:00 0 
f4a8e000-f4d00000 r--p 00000000 08:01 134511                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/System.dll
f4d00000-f4d21000 rw-p 00000000 00:00 0 
f4d21000-f4e00000 ---p 00000000 00:00 0 
f4ef9000-f579f000 r-xp 00000000 08:01 134613                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll.so
f579f000-f57a0000 r--p 008a5000 08:01 134613                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll.so
f57a0000-f57a1000 rw-p 008a6000 08:01 134613                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll.so
f57a1000-f57b5000 rw-p 00000000 00:00 0 
f57b5000-f5bff000 r--p 00000000 08:01 134612                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll
f5bff000-f63ff000 rw-p 00000000 00:00 0 
f63ff000-f6400000 ---p 00000000 00:00 0 
f6400000-f7000000 rw-p 00000000 00:00 0 
f705b000-f7077000 r-xp 00000000 08:01 1705490                            /lib/i386-linux-gnu/libgcc_s.so.1
f7077000-f7078000 r--p 0001b000 08:01 1705490                            /lib/i386-linux-gnu/libgcc_s.so.1
f7078000-f7079000 rw-p 0001c000 08:01 1705490                            /lib/i386-linux-gnu/libgcc_s.so.1
f7082000-f70c2000 rwxp 00000000 00:00 0 
f70c2000-f70ce000 rw-p 00000000 00:00 0 
f70ce000-f71d3000 r--p 00000000 08:01 134341                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/System.Core.dll
f71d3000-f71d7000 rw-p 00000000 00:00 0 
f71d7000-f71e0000 ---p 00000000 00:00 0 
f71e0000-f735d000 rw-p 00000000 00:00 0 
f735d000-f7360000 rw-p 00000000 00:00 0 
f7360000-f7370000 rwxp 00000000 00:00 0 
f7370000-f7372000 r--p 00000000 08:01 141529                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/tests/runtime/appdomain-threadpool-unload.exe
f7372000-f737b000 rw-p 00000000 00:00 0 
f737b000-f73aa000 ---p 00000000 00:00 0 
f73aa000-f73ab000 rw-s 00000000 00:14 126132                             /dev/shm/mono.7222
f73ab000-f73bb000 rwxp 00000000 00:00 0 
f73bb000-f740c000 r--p 00000000 08:01 1313918                            /usr/lib/locale/aa_DJ.utf8/LC_CTYPE
f740c000-f740d000 r--p 00000000 08:01 1444928                            /usr/lib/locale/aa_ET/LC_NUMERIC
f740d000-f740e000 r--p 00000000 08:01 1573167                            /usr/lib/locale/en_US.utf8/LC_TIME
f740e000-f753e000 r--p 00000000 08:01 1313917                            /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
f753e000-f753f000 r--p 00000000 08:01 1445549                            /usr/lib/locale/chr_US/LC_MONETARY
f753f000-f7540000 r--p 00000000 08:01 1572954                            /usr/lib/locale/en_AG/LC_MESSAGES/SYS_LC_MESSAGES
f7540000-f7541000 r--p 00000000 08:01 1445551                            /usr/lib/locale/chr_US/LC_PAPER
f7541000-f7542000 r--p 00000000 08:01 1445550                            /usr/lib/locale/chr_US/LC_NAME
f7542000-f7543000 r--p 00000000 08:01 1573164                            /usr/lib/locale/en_US.utf8/LC_ADDRESS
f7543000-f7544000 r--p 00000000 08:01 1445552                            /usr/lib/locale/chr_US/LC_TELEPHONE
f7544000-f7546000 rw-p 00000000 00:00 0 
f7546000-f76f7000 r-xp 00000000 08:01 1705495                            /lib/i386-linux-gnu/libc-2.24.so
f76f7000-f76f9000 r--p 001b0000 08:01 1705495                            /lib/i386-linux-gnu/libc-2.24.so
f76f9000-f76fa000 rw-p 001b2000 08:01 1705495                            /lib/i386-linux-gnu/libc-2.24.so
f76fa000-f76fd000 rw-p 00000000 00:00 0 
f76fd000-f7716000 r-xp 00000000 08:01 1705509                            /lib/i386-linux-gnu/libpthread-2.24.so
f7716000-f7717000 r--p 00018000 08:01 1705509                            /lib/i386-linux-gnu/libpthread-2.24.so
f7717000-f7718000 rw-p 00019000 08:01 1705509                            /lib/i386-linux-gnu/libpthread-2.24.so
f7718000-f771a000 rw-p 00000000 00:00 0 
f771a000-f771d000 r-xp 00000000 08:01 1705498                            /lib/i386-linux-gnu/libdl-2.24.so
f771d000-f771e000 r--p 00002000 08:01 1705498                            /lib/i386-linux-gnu/libdl-2.24.so
f771e000-f771f000 rw-p 00003000 08:01 1705498                            /lib/i386-linux-gnu/libdl-2.24.so
f771f000-f7726000 r-xp 00000000 08:01 1705511                            /lib/i386-linux-gnu/librt-2.24.so
f7726000-f7727000 r--p 00006000 08:01 1705511                            /lib/i386-linux-gnu/librt-2.24.so
f7727000-f7728000 rw-p 00007000 08:01 1705511                            /lib/i386-linux-gnu/librt-2.24.so
f7728000-f777b000 r-xp 00000000 08:01 1705499                            /lib/i386-linux-gnu/libm-2.24.so
f777b000-f777c000 r--p 00052000 08:01 1705499                            /lib/i386-linux-gnu/libm-2.24.so
f777c000-f777d000 rw-p 00053000 08:01 1705499                            /lib/i386-linux-gnu/libm-2.24.so
f777d000-f777e000 r--p 00000000 08:01 1445546                            /usr/lib/locale/chr_US/LC_MEASUREMENT
f777e000-f7785000 r--s 00000000 08:01 142663                             /usr/lib/i386-linux-gnu/gconv/gconv-modules.cache
f7785000-f7786000 r--p 00000000 08:01 1573165                            /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
f7786000-f7789000 rw-p 00000000 00:00 0 
f7789000-f778b000 r--p 00000000 00:00 0                                  [vvar]
f778b000-f778d000 r-xp 00000000 00:00 0                                  [vdso]
f778d000-f77b0000 r-xp 00000000 08:01 1705491                            /lib/i386-linux-gnu/ld-2.24.so
f77b0000-f77b1000 r--p 00022000 08:01 1705491                            /lib/i386-linux-gnu/ld-2.24.so
f77b1000-f77b2000 rw-p 00023000 08:01 1705491                            /lib/i386-linux-gnu/ld-2.24.so
ff330000-ff338000 ---p 00000000 00:00 0 
ffb0e000-ffb2f000 rw-p 00000000 00:00 0                                  [stack]
Stacktrace:

  at <unknown> <0xffffffff>
  at System.IO.MonoIO.Read (System.Runtime.InteropServices.SafeHandle,byte[],int,int,System.IO.MonoIOError&) [0x00010] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.IO.FileStream.ReadData (System.Runtime.InteropServices.SafeHandle,byte[],int,int) [0x00002] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.IO.FileStream.RefillBuffer () [0x00006] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.IO.FileStream.ReadInternal (byte[],int,int) [0x00049] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.IO.FileStream.Read (byte[],int,int) [0x000a1] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.IO.StreamReader.ReadBuffer () [0x00028] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.IO.StreamReader.Peek () [0x00030] in <0d27703b1d7d413998da387624d5a7e4>:0
  at Mono.Xml.SmallXmlParser.Peek () [0x00000] in <0d27703b1d7d413998da387624d5a7e4>:0
  at Mono.Xml.SmallXmlParser.Parse (System.IO.TextReader,Mono.Xml.SmallXmlParser/IContentHandler) [0x0001d] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Security.Cryptography.CryptoConfig.LoadConfig (string,System.Collections.Generic.IDictionary`2<string, System.Type>,System.Collections.Generic.IDictionary`2<string, string>) [0x0001e] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Security.Cryptography.CryptoConfig.Initialize () [0x00875] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Security.Cryptography.CryptoConfig.CreateFromName (string,object[]) [0x00025] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Security.Cryptography.CryptoConfig.CreateFromName (string) [0x00000] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Security.Cryptography.RandomNumberGenerator.Create (string) [0x00000] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Security.Cryptography.RandomNumberGenerator.Create () [0x00000] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Guid.NewGuid () [0x0001f] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Runtime.Remoting.RemotingServices.NewUri () [0x0001e] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.Runtime.Remoting.RemotingServices.Marshal (System.MarshalByRefObject,string,System.Type) [0x000a3] in <0d27703b1d7d413998da387624d5a7e4>:0
  at System.AppDomain.GetMarshalledDomainObjRef () [0x00005] in <0d27703b1d7d413998da387624d5a7e4>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_object__this__ (object,intptr,intptr,intptr) [0x00027] in <0d27703b1d7d413998da387624d5a7e4>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) <0x00012>
  at System.AppDomain.InvokeInDomain (System.AppDomain,System.Reflection.MethodInfo,object,object[]) <0x000a5>
  at System.Runtime.Remoting.RemotingServices.GetDomainProxy (System.AppDomain) <0x00073>
  at System.AppDomain.CreateDomain (string,System.Security.Policy.Evidence,System.AppDomainSetup) <0x001cb>
  at System.AppDomain.CreateDomain (string) <0x00040>
  at Driver/<>c.<Main>b__1_0 (int) <0x0004b>
  at System.Linq.Parallel.SelectQueryOperator`2/SelectQueryOperatorEnumerator`1<int, int, int>.MoveNext (int&,int&) <0x0005b>
  at System.Linq.Parallel.SelectQueryOperator`2/SelectQueryOperatorEnumerator`1<int, int, int>.MoveNext (int&,int&) <0x00041>
  at System.Linq.Parallel.PipelineSpoolingTask`2<int, int>.SpoolingWork () <0x000bd>
  at System.Linq.Parallel.SpoolingTaskBase.Work () <0x00030>
  at System.Linq.Parallel.QueryTask.BaseWork (object) <0x00046>
  at System.Linq.Parallel.QueryTask/<>c.<.cctor>b__10_0 (object) <0x0004f>
  at System.Threading.Tasks.Task.InnerInvoke () <0x000a4>
  at System.Threading.Tasks.Task.Execute () <0x00037>
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) <0x00052>
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x0018f>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00046>
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) <0x000e0>
  at System.Threading.Tasks.Task.ExecuteEntry (bool) <0x000d7>
  at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x00038>
  at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x00246>
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x0002a>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) <0x00064>
/proc/self/maps:
565a1000-56a9d000 r-xp 00000000 08:01 134596                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen
56a9e000-56aa2000 r--p 004fc000 08:01 134596                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen
56aa2000-56aa5000 rw-p 00500000 08:01 134596                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen
56aa5000-56b2c000 rw-p 00000000 00:00 0 
58250000-58413000 rw-p 00000000 00:00 0                                  [heap]
f43ff000-f4408000 ---p 00000000 00:00 0 
f4408000-f4500000 rw-p 00000000 00:00 0 
f4500000-f45cb000 rw-p 00000000 00:00 0 
f45cb000-f4600000 ---p 00000000 00:00 0 
f46ff000-f4708000 ---p 00000000 00:00 0 
f4708000-f4800000 rw-p 00000000 00:00 0 
f4800000-f4821000 rw-p 00000000 00:00 0 
f4821000-f4900000 ---p 00000000 00:00 0 
f498d000-f4996000 ---p 00000000 00:00 0 
f4996000-f4a8e000 rw-p 00000000 00:00 0 
f4a8e000-f4d00000 r--p 00000000 08:01 134511                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/System.dll
f4d00000-f4d21000 rw-p 00000000 00:00 0 
f4d21000-f4e00000 ---p 00000000 00:00 0 
f4ef9000-f579f000 r-xp 00000000 08:01 134613                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll.so
f579f000-f57a0000 r--p 008a5000 08:01 134613                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll.so
f57a0000-f57a1000 rw-p 008a6000 08:01 134613                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll.so
f57a1000-f57b5000 rw-p 00000000 00:00 0 
f57b5000-f5bff000 r--p 00000000 08:01 134612                             /home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mscorlib.dll
f5bff000-f63ff000 rw-p 00000000 00:00 0 
f63ff000-f6400000 ---p 00000000 00:00 0 
Memory around native instruction pointer (0xf778bc89):
0xf778bc79  90 90 90 90 90 90 90 51 52 55 89 e5 0f 34 cd 80  .......QRU...4..
0xf778bc89  5d 5a 59 c3 90 90 90 90 eb 0d 90 90 90 90 90 90  ]ZY.............
0xf778bc99  90 90 90 90 90 90 90 58 b8 77 00 00 00 cd 80 90  .......X.w......
0xf778bca9  8d b4 26 00 00 00 00 b8 ad 00 00 00 cd 80 90 48  ..&............H

Native stacktrace:

	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x117756) [0x566b8756]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x117bb7) [0x566b8bb7]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0xbc3b9) [0x5665d3b9]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x116d34) [0x566b7d34]
	linux-gate.so.1(__kernel_rt_sigreturn+0) [0xf778bcb0]
	linux-gate.so.1(__kernel_vsyscall+0x9) [0xf778bc89]
	/lib/i386-linux-gnu/libc.so.6(gsignal+0xb0) [0xf7571dd0]
	/lib/i386-linux-gnu/libc.so.6(abort+0x157) [0xf7573297]
	/lib/i386-linux-gnu/libc.so.6(+0x6738f) [0xf75ad38f]
	/lib/i386-linux-gnu/libc.so.6(+0x6dfc7) [0xf75b3fc7]
	/lib/i386-linux-gnu/libc.so.6(+0x6e806) [0xf75b4806]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(monoeg_g_free+0x20) [0x568b4f70]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(mono_bitset_free+0x26) [0x5689f966]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x11d367) [0x566be367]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x11f45e) [0x566c045e]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x2ea72) [0x565cfa72]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x2ee20) [0x565cfe20]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0xbfde0) [0x56660de0]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0xc0738) [0x56661738]
	[0xf73ab188]
	[0xf454dee0]
	[0xf7099e24]
	[0xf7099c1c]
	[0xf7099afc]
	[0xf709989c]
	[0xf7099634]
	[0xf70994dd]
	[0xf709948e]
	[0xf7099374]
	[0xf70c0d40]
	[0xf70bb5f0]
	[0xf70b9f6c]
	[0xf70b9d84]
	[0xf70b9d14]
	[0xf70b9cdc]
	[0xf70b9bac]
	[0xf70b97c8]
	[0xf70b2b3c]
	[0xf70b2610]
	[0xf70b26bb]
	/home/helixbot/dotnetbuild/work/f19b7c56-9d26-4816-a344-30fc818d34bc/Payload/net_4_x/mono-sgen(+0x30470) [0x565d1470]

Waiting for dumping threads to resume


Debug info from gdb:

[New LWP 7223]
[New LWP 7224]
[New LWP 7225]
[New LWP 7226]
[New LWP 7227]
[New LWP 7229]
[New LWP 7230]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0xf778bc89 in __kernel_vsyscall ()
  Id   Target Id         Frame 
* 1    Thread 0xf7544940 (LWP 7222) "mono-sgen" 0xf778bc89 in __kernel_vsyscall ()
  2    Thread 0xf6bffb40 (LWP 7223) "SGen worker" 0xf778bc89 in __kernel_vsyscall ()
  3    Thread 0xf72d7b40 (LWP 7224) "Finalizer" 0xf778bc89 in __kernel_vsyscall ()
  4    Thread 0xf4a8db40 (LWP 7225) "mono-sgen" 0xf778bc89 in __kernel_vsyscall ()
  5    Thread 0xf47ffb40 (LWP 7226) "Thread Pool Wor" 0xf778bc89 in __kernel_vsyscall ()
  6    Thread 0xf44ffb40 (LWP 7227) "Thread Pool Wor" 0xf778bc89 in __kernel_vsyscall ()
  7    Thread 0xf3efeb40 (LWP 7229) "Domain unloader" 0xf778bc89 in __kernel_vsyscall ()
  8    Thread 0xf3fffb40 (LWP 7230) "Thread Pool Wor" 0xf778bc89 in __kernel_vsyscall ()

Thread 8 (Thread 0xf3fffb40 (LWP 7230)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf770b3ea in do_futex_wait () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xf770b591 in __new_sem_wait_slow () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x5684c96d in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0x56aa8d48 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0x56aa8d48 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=<optimized out>) at threadpool-worker-default.c:490
#7  0x567ea45c in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#8  start_wrapper (data=0xf4808748) at threads.c:1234
#9  0xf770327a in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#10 0xf762dae6 in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 7 (Thread 0xf3efeb40 (LWP 7229)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf770b116 in do_futex_wait.constprop () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xf770b207 in __new_sem_wait_slow.constprop.1 () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x568a90be in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=4294967295, sem=0x56b2aec0 <suspend_semaphore>) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_threads_wait_pending_operations () at mono-threads.c:311
#6  0x5685bad7 in sgen_unified_suspend_stop_world () at sgen-stw.c:297
#7  sgen_client_stop_world (generation=1, serial_collection=1) at sgen-stw.c:124
#8  0x5686bfb9 in sgen_stop_world (generation=1, serial_collection=1) at sgen-gc.c:3821
#9  0x5686c185 in sgen_perform_collection_inner (requested_size=requested_size@entry=0, generation_to_collect=generation_to_collect@entry=1, reason=reason@entry=0x569792ac "user request", forced_serial=1, stw=1) at sgen-gc.c:2530
#10 0x5686c95d in sgen_perform_collection (stw=1, forced_serial=1, reason=0x569792ac "user request", generation_to_collect=1, requested_size=0) at sgen-gc.c:2635
#11 sgen_gc_collect (generation=1) at sgen-gc.c:3087
#12 0x56845158 in mono_gc_collect (generation=1) at sgen-mono.c:2441
#13 0x5683756e in mono_domain_finalize (domain=0x582ec7e8, timeout=4294967295) at gc.c:455
#14 0x56728a2f in unload_thread_main (arg=0xf45ef210) at appdomain.c:2853
#15 0x567ea45c in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#16 start_wrapper (data=0xf4253838) at threads.c:1234
#17 0xf770327a in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#18 0xf762dae6 in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 6 (Thread 0xf44ffb40 (LWP 7227)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf7708b9b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x56896110 in mono_os_cond_wait (mutex=0x56ab0818 <signal_mutex>, cond=0xf44fe9b0) at mono-os-mutex.h:173
#3  mono_os_event_wait_multiple (events=<optimized out>, nevents=<optimized out>, waitall=<optimized out>, timeout=<optimized out>, alertable=<optimized out>) at os-event-unix.c:190
#4  0x56896385 in mono_os_event_wait_one (event=<optimized out>, timeout=4294967295, alertable=1) at os-event-unix.c:94
#5  0x568ab7c4 in mono_thread_info_wait_one_handle (thread_handle=0xf4d01098, timeout=4294967295, alertable=1) at mono-threads.c:1936
#6  0x5672e1f7 in guarded_wait (timeout=4294967295, alertable=1, thread_handle=0xf4d01098) at appdomain.c:2931
#7  mono_domain_try_unload (domain=0x582ec7e8, exc=0xf44feb0c) at appdomain.c:3033
#8  0x5672e3a6 in ves_icall_System_AppDomain_InternalUnload (domain_id=2, error=0xf44feb70) at appdomain.c:2559
#9  0x56773443 in ves_icall_System_AppDomain_InternalUnload_raw (a0=2, error=0xf44feb70) at ../../mono/metadata/icall-def.h:185
#10 0xf4ebcb38 in ?? ()
#11 0xf50414b0 in System_AppDomain_Unload_System_AppDomain (domain=...) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/mcs/class/corlib/System/AppDomain.cs:1200
#12 0x58361208 in ?? ()
#13 0xf736a728 in ?? ()
#14 0xf736a1ec in ?? ()
#15 0xf736a1d2 in ?? ()
#16 0xf736a15e in ?? ()
#17 0xf7369ed1 in ?? ()
#18 0xf7369d37 in ?? ()
#19 0xf7369c08 in ?? ()
#20 0xf50c1af5 in System_Threading_Tasks_Task_InnerInvoke (this=...) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2482
#21 0xf6c05b78 in ?? ()
#22 0xf50c1818 in System_Threading_Tasks_Task_Execute (this=...) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2319
#23 0x56aa2000 in ?? ()
#24 0xf50c1a43 in System_Threading_Tasks_Task_ExecutionContextCallback_object (obj=0xf57a0180) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:2463
#25 0x568ae7f6 in mono_tls_get_thread () at mono-tls.c:308
#26 0x00000000 in ?? ()

Thread 5 (Thread 0xf47ffb40 (LWP 7226)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf770ce0f in waitpid () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x566b8890 in dump_native_stacktrace (ctx=ctx@entry=0xf47fd44c, signal=0x568d9061 "SIGABRT") at mini-posix.c:1083
#3  0x566b8bb7 in mono_dump_native_crash_info (signal=0x568d9061 "SIGABRT", ctx=0xf47fd44c, info=0xf47fd3cc) at mini-posix.c:1112
#4  0x5665d3b9 in mono_handle_native_crash (signal=0x568d9061 "SIGABRT", ctx=0xf47fd44c, info=0xf47fd3cc) at mini-exceptions.c:3196
#5  0x566b7d34 in sigabrt_signal_handler (_dummy=6, _info=0xf47fd3cc, context=0xf47fd44c) at mini-posix.c:223
#6  <signal handler called>
#7  0xf778bc89 in __kernel_vsyscall ()
#8  0xf7571dd0 in raise () from /lib/i386-linux-gnu/libc.so.6
#9  0xf7573297 in abort () from /lib/i386-linux-gnu/libc.so.6
#10 0xf75ad38f in ?? () from /lib/i386-linux-gnu/libc.so.6
#11 0xf75b3fc7 in ?? () from /lib/i386-linux-gnu/libc.so.6
#12 0xf75b4806 in ?? () from /lib/i386-linux-gnu/libc.so.6
#13 0x568b4f70 in monoeg_g_free (ptr=0x583c6b28) at gmem.c:71
#14 0x5689f966 in mono_bitset_free (set=0x583c6b28) at monobitset.c:77
#15 0x566be367 in mono_compile_create_vars (cfg=<optimized out>) at mini.c:2085
#16 mini_method_compile (method=<optimized out>, opts=<optimized out>, domain=<optimized out>, flags=<optimized out>, parts=<optimized out>, aot_method_index=<optimized out>) at mini.c:3478
#17 0x566c045e in mono_jit_compile_method_inner (method=0x583ed558, target_domain=0xf4523120, opt=382822911, error=0xf47fe07c) at mini.c:4071
#18 0x565cfa72 in mono_jit_compile_method_with_opt (method=0x583ed558, opt=382822911, jit_only=jit_only@entry=1, error=0xf47fe07c) at mini-runtime.c:2436
#19 0x565d0aca in mono_jit_compile_method_jit_only (method=<optimized out>, error=0xf47fe07c) at mini-runtime.c:2499
#20 0x565cfe20 in compile_special (error=0xf47fe07c, target_domain=0xf4523120, method=0xf454dee0) at mini-runtime.c:2182
#21 mono_jit_compile_method_with_opt (method=0xf454dee0, opt=382822911, jit_only=jit_only@entry=0, error=0xf47fe07c) at mini-runtime.c:2413
#22 0x565d0a87 in mono_jit_compile_method (method=<optimized out>, error=0xf47fe07c) at mini-runtime.c:2485
#23 0x56660de0 in common_call_trampoline (regs=regs@entry=0xf47fe0f0, code=code@entry=0xf7099f08 "\211E\360\307", <incomplete sequence \354>, m=m@entry=0xf454dee0, vt=0x0, vtable_slot=<optimized out>, error=0xf47fe07c) at mini-trampolines.c:751
#24 0x56661738 in mono_magic_trampoline (regs=0xf47fe0f0, code=0xf7099f08 "\211E\360\307", <incomplete sequence \354>, arg=0xf454dee0, tramp=0x0) at mini-trampolines.c:891
#25 0xf73ab188 in ?? ()
#26 0xf454dee0 in ?? ()
#27 0xf7099e24 in ?? ()
#28 0xf7099c1c in ?? ()
#29 0xf7099afc in ?? ()
#30 0xf709989c in ?? ()
#31 0xf7099634 in ?? ()
#32 0xf70994dd in ?? ()
#33 0xf709948e in ?? ()
#34 0xf7099374 in ?? ()
#35 0xf70c0d40 in ?? ()
#36 0xf70bb5f0 in ?? ()
#37 0xf70b9f6c in ?? ()
#38 0xf70b9d84 in ?? ()
#39 0xf70b9d14 in ?? ()
#40 0xf70b9cdc in ?? ()
#41 0xf70b9bac in ?? ()
#42 0xf70b97c8 in ?? ()
#43 0xf70b2b3c in ?? ()
#44 0xf70b2610 in ?? ()
#45 0xf70b26bb in ?? ()
#46 0x565d1470 in mono_jit_runtime_invoke (method=0xf45295c8, obj=0xf6c0f550, params=0x0, exc=<optimized out>, error=0xf47fea2c) at mini-runtime.c:3177
#47 0x567be6ed in do_runtime_invoke (method=0xf45295c8, obj=<optimized out>, params=<optimized out>, exc=0x0, error=0xf47fea2c) at object.c:2978
#48 0x567cc4d4 in mono_runtime_try_invoke_array (method=0xf45295c8, obj=0xf6c0f550, params=0x0, exc=0x0, error=0xf47fea2c) at object.c:5464
#49 0x567cc7a1 in mono_runtime_invoke_array_checked (method=0xf45295c8, obj=0xf6c0f550, params=0x0, error=0xf47fea2c) at object.c:5342
#50 0x5676755b in ves_icall_InternalInvoke (method=0xf6c0faf8, this_arg=0xf6c0f550, params=0x0, exc=0xf47feb00) at icall.c:3427
#51 0xf70b2580 in ?? ()
#52 0xf5040896 in System_AppDomain_InvokeInDomain_System_AppDomain_System_Reflection_MethodInfo_object_object__ (domain=..., method=..., obj=0xf47feb20, args=0xf47feb68) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/mcs/class/corlib/System/AppDomain.cs:944
#53 0xf6c0f550 in ?? ()
#54 0xf51abb84 in System_Runtime_Remoting_RemotingServices_GetDomainProxy_System_AppDomain (domain=...) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/mcs/class/corlib/System.Runtime.Remoting/RemotingServices.cs:837
#55 0xf6c0faf8 in ?? ()
#56 0xf5040d4c in System_AppDomain_CreateDomain_string_System_Security_Policy_Evidence_System_AppDomainSetup (friendlyName=0x5835e368, securityInfo=..., info=...) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/mcs/class/corlib/System/AppDomain.cs:1039
#57 0xf6c07b58 in ?? ()
#58 0xf5040b21 in System_AppDomain_CreateDomain_string (friendlyName=0x0) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/mcs/class/corlib/System/AppDomain.cs:989
#59 0x00000000 in ?? ()

Thread 4 (Thread 0xf4a8db40 (LWP 7225)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf7708f65 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x568a1455 in mono_os_cond_timedwait (cond=0x56b2ae40 <sleep_cond>, mutex=0x56b2ae70 <sleep_mutex>, timeout_ms=500) at mono-os-mutex.c:75
#3  0x568ab173 in mono_coop_cond_timedwait (cond=0x56b2ae40 <sleep_cond>, mutex=0x56b2ae70 <sleep_mutex>, timeout_ms=500) at ../../mono/utils/mono-coop-mutex.h:102
#4  sleep_interruptable (alerted=0xf4a8d27c, ms=500) at mono-threads.c:1572
#5  mono_thread_info_sleep (ms=500, alerted=0xf4a8d27c) at mono-threads.c:1604
#6  0x5684d3e1 in monitor_thread (unused=<optimized out>) at threadpool-worker-default.c:708
#7  0x567ea45c in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#8  start_wrapper (data=0x58357688) at threads.c:1234
#9  0xf770327a in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#10 0xf762dae6 in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 3 (Thread 0xf72d7b40 (LWP 7224)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf770b116 in do_futex_wait.constprop () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xf770b207 in __new_sem_wait_slow.constprop.1 () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x568368d5 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x56aa872c <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x56aa872c <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:43
#5  finalizer_thread (unused=<optimized out>) at gc.c:920
#6  0x567ea45c in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#7  start_wrapper (data=0x582cf2a0) at threads.c:1234
#8  0xf770327a in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xf762dae6 in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 2 (Thread 0xf6bffb40 (LWP 7223)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf7708b9b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x568931fa in mono_os_cond_wait (mutex=0x56ab06f0 <lock>, cond=0x56ab06c0 <work_cond>) at ../../mono/utils/mono-os-mutex.h:173
#3  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=0) at sgen-thread-pool.c:165
#4  thread_func (data=0x0) at sgen-thread-pool.c:196
#5  0xf770327a in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#6  0xf762dae6 in clone () from /lib/i386-linux-gnu/libc.so.6

Thread 1 (Thread 0xf7544940 (LWP 7222)):
#0  0xf778bc89 in __kernel_vsyscall ()
#1  0xf7708b9b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x568a14a2 in mono_os_cond_wait (mutex=0x58252ab8, cond=0x58252ad0) at mono-os-mutex.h:173
#3  mono_os_cond_timedwait (cond=0x58252ad0, mutex=0x58252ab8, timeout_ms=4294967295) at mono-os-mutex.c:32
#4  0x5680378f in mono_coop_cond_timedwait (timeout_ms=4294967295, mutex=0x58252ab8, cond=0x58252ad0) at ../../mono/utils/mono-coop-mutex.h:102
#5  mono_w32handle_timedwait_signal_naked (alerted=0xffb2d010, poll=0, timeout=4294967295, mutex=0x58252ab8, cond=0x58252ad0) at w32handle.c:650
#6  mono_w32handle_timedwait_signal_handle (handle_data=0x58252aa8, timeout=timeout@entry=4294967295, alerted=alerted@entry=0xffb2d010, poll=0) at w32handle.c:765
#7  0x56804b40 in mono_w32handle_wait_one (handle=0x58252aa8, timeout=4294967295, alertable=1) at w32handle.c:873
#8  0x56840410 in ves_icall_System_Threading_Monitor_Monitor_wait (obj=0xf6c04f08, ms=4294967295) at monitor.c:1390
#9  0xf7369e1c in ?? ()
#10 0xf5090cf8 in System_Threading_Monitor_ObjWait_bool_int_object (exitContext=false, millisecondsTimeout=-1, obj=0x5828b4e1) at /mnt/jenkins/workspace/test-mono-mainline-linux-helix/label/debian-9-i386/mcs/class/corlib/System.Threading/Monitor.cs:83
#11 0xffffffff in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

X64:

*** Error in `/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen': double free or corruption (fasttop): 0x00007ff2bc0689b0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7ff2cfac4bfb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fc6)[0x7ff2cfacafc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7780e)[0x7ff2cfacb80e]
/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x12fc7a)[0x560ad1a37c7a]
/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x131df4)[0x560ad1a39df4]
/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x3fc07)[0x560ad1947c07]
/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x3ff91)[0x560ad1947f91]
/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0xc4675)[0x560ad19cc675]
/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0xc529f)[0x560ad19cd29f]
[0x41066513]
======= Memory map: ========
41065000-41075000 rwxp 00000000 00:00 0 
4138f000-413cf000 rwxp 00000000 00:00 0 
41601000-41641000 rwxp 00000000 00:00 0 
41f82000-41fa2000 rwxp 00000000 00:00 0 
560ad1908000-560ad1d66000 r-xp 00000000 08:01 133517                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen
560ad1f65000-560ad1f6c000 r--p 0045d000 08:01 133517                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen
560ad1f6c000-560ad1f71000 rw-p 00464000 08:01 133517                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen
560ad1f71000-560ad2002000 rw-p 00000000 00:00 0 
560ad25be000-560ad275e000 rw-p 00000000 00:00 0                          [heap]
7ff2b0000000-7ff2b0021000 rw-p 00000000 00:00 0 
7ff2b0021000-7ff2b4000000 ---p 00000000 00:00 0 
7ff2b8000000-7ff2b81c2000 rw-p 00000000 00:00 0 
7ff2b81c2000-7ff2bc000000 ---p 00000000 00:00 0 
7ff2bc000000-7ff2bc152000 rw-p 00000000 00:00 0 
7ff2bc152000-7ff2c0000000 ---p 00000000 00:00 0 
7ff2c0000000-7ff2c0021000 rw-p 00000000 00:00 0 
7ff2c0021000-7ff2c4000000 ---p 00000000 00:00 0 
7ff2c4000000-7ff2c4021000 rw-p 00000000 00:00 0 
7ff2c4021000-7ff2c8000000 ---p 00000000 00:00 0 
7ff2c8000000-7ff2c8021000 rw-p 00000000 00:00 0 
7ff2c8021000-7ff2cc000000 ---p 00000000 00:00 0 
7ff2cc055000-7ff2cc0a2000 r--p 00000000 08:01 133232                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/Mono.Security.dll
7ff2cc0a2000-7ff2cc0ab000 ---p 00000000 00:00 0 
7ff2cc0ab000-7ff2cc2a3000 rw-p 00000000 00:00 0 
7ff2cc2a3000-7ff2cc2ac000 ---p 00000000 00:00 0 
7ff2cc2ac000-7ff2cc4a4000 rw-p 00000000 00:00 0 
7ff2cc4a4000-7ff2cc4ad000 ---p 00000000 00:00 0 
7ff2cc4ad000-7ff2cc6a5000 rw-p 00000000 00:00 0 
7ff2cc6a5000-7ff2cc6ae000 ---p 00000000 00:00 0 
7ff2cc6ae000-7ff2cc8a6000 rw-p 00000000 00:00 0 
7ff2cc8a6000-7ff2ccb18000 r--p 00000000 08:01 133432                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/System.dll
7ff2ccb18000-7ff2ccb21000 ---p 00000000 00:00 0 
7ff2ccb21000-7ff2ccd19000 rw-p 00000000 00:00 0 
7ff2ccd19000-7ff2cd58c000 r-xp 00000000 08:01 133534                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mscorlib.dll.so
7ff2cd58c000-7ff2cd78b000 ---p 00873000 08:01 133534                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mscorlib.dll.so
7ff2cd78b000-7ff2cd78c000 r--p 00872000 08:01 133534                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mscorlib.dll.so
7ff2cd78c000-7ff2cd78d000 rw-p 00873000 08:01 133534                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mscorlib.dll.so
7ff2cd78d000-7ff2cd7b5000 rw-p 00000000 00:00 0 
7ff2cd7b5000-7ff2cdbff000 r--p 00000000 08:01 133533                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mscorlib.dll
7ff2cdbff000-7ff2cebff000 rw-p 00000000 00:00 0 
7ff2cebff000-7ff2cec00000 ---p 00000000 00:00 0 
7ff2cec00000-7ff2cf800000 rw-p 00000000 00:00 0 
7ff2cf840000-7ff2cf853000 rw-p 00000000 00:00 0 
7ff2cf853000-7ff2cf958000 r--p 00000000 08:01 133262                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/System.Core.dll
7ff2cf958000-7ff2cfa50000 rw-p 00000000 00:00 0 
7ff2cfa50000-7ff2cfa52000 r--p 00000000 08:01 140450                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/tests/runtime/appdomain-threadpool-unload.exe
7ff2cfa52000-7ff2cfa54000 rw-p 00000000 00:00 0 
7ff2cfa54000-7ff2cfbe9000 r-xp 00000000 08:01 1703959                    /lib/x86_64-linux-gnu/libc-2.24.so
7ff2cfbe9000-7ff2cfde9000 ---p 00195000 08:01 1703959                    /lib/x86_64-linux-gnu/libc-2.24.so
7ff2cfde9000-7ff2cfded000 r--p 00195000 08:01 1703959                    /lib/x86_64-linux-gnu/libc-2.24.so
7ff2cfded000-7ff2cfdef000 rw-p 00199000 08:01 1703959                    /lib/x86_64-linux-gnu/libc-2.24.so
7ff2cfdef000-7ff2cfdf3000 rw-p 00000000 00:00 0 
7ff2cfdf3000-7ff2cfe09000 r-xp 00000000 08:01 1703952                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff2cfe09000-7ff2d0008000 ---p 00016000 08:01 1703952                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff2d0008000-7ff2d0009000 r--p 00015000 08:01 1703952                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff2d0009000-7ff2d000a000 rw-p 00016000 08:01 1703952                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff2d000a000-7ff2d0022000 r-xp 00000000 08:01 1703976                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7ff2d0022000-7ff2d0221000 ---p 00018000 08:01 1703976                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7ff2d0221000-7ff2d0222000 r--p 00017000 08:01 1703976                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7ff2d0222000-7ff2d0223000 rw-p 00018000 08:01 1703976                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7ff2d0223000-7ff2d0227000 rw-p 00000000 00:00 0 
7ff2d0227000-7ff2d022a000 r-xp 00000000 08:01 1703964                    /lib/x86_64-linux-gnu/libdl-2.24.so
7ff2d022a000-7ff2d0429000 ---p 00003000 08:01 1703964                    /lib/x86_64-linux-gnu/libdl-2.24.so
7ff2d0429000-7ff2d042a000 r--p 00002000 08:01 1703964                    /lib/x86_64-linux-gnu/libdl-2.24.so
7ff2d042a000-7ff2d042b000 rw-p 00003000 08:01 1703964                    /lib/x86_64-linux-gnu/libdl-2.24.so
7ff2d042b000-7ff2d0432000 r-xp 00000000 08:01 1703978                    /lib/x86_64-linux-gnu/librt-2.24.so
7ff2d0432000-7ff2d0631000 ---p 00007000 08:01 1703978                    /lib/x86_64-linux-gnu/librt-2.24.so
7ff2d0631000-7ff2d0632000 r--p 00006000 08:01 1703978                    /lib/x86_64-linux-gnu/librt-2.24.so
7ff2d0632000-7ff2d0633000 rw-p 00007000 08:01 1703978                    /lib/x86_64-linux-gnu/librt-2.24.so
7ff2d0633000-7ff2d0736000 r-xp 00000000 08:01 1703965                    /lib/x86_64-linux-gnu/libm-2.24.so
7ff2d0736000-7ff2d0935000 ---p 00103000 08:01 1703965                    /lib/x86_64-linux-gnu/libm-2.24.so
7ff2d0935000-7ff2d0936000 r--p 00102000 08:01 1703965                    /lib/x86_64-linux-gnu/libm-2.24.so
7ff2d0936000-7ff2d0937000 rw-p 00103000 08:01 1703965                    /lib/x86_64-linux-gnu/libm-2.24.so
7ff2d0937000-7ff2d095a000 r-xp 00000000 08:01 1703953                    /lib/x86_64-linux-gnu/ld-2.24.so
7ff2d095a000-7ff2d0960000 rw-p 00000000 00:00 0 
7ff2d0960000-7ff2d09bf000 ---p 00000000 00:00 0 
7ff2d09bf000-7ff2d09c0000 rw-s 00000000 00:14 22606                      /dev/shm/mono.2405
7ff2d09c0000-7ff2d0a11000 r--p 00000000 08:01 1313918                    /usr/lib/locale/aa_DJ.utf8/LC_CTYPE
7ff2d0a11000-7ff2d0a12000 r--p 00000000 08:01 1444928                    /usr/lib/locale/aa_ET/LC_NUMERIC
7ff2d0a12000-7ff2d0a13000 r--p 00000000 08:01 1573167                    /usr/lib/locale/en_US.utf8/LC_TIME
7ff2d0a13000-7ff2d0b43000 r--p 00000000 08:01 1313917                    /usr/lib/locale/aa_DJ.utf8/LC_COLLATE
7ff2d0b43000-7ff2d0b44000 r--p 00000000 08:01 1445549                    /usr/lib/locale/chr_US/LC_MONETARY
7ff2d0b44000-7ff2d0b4b000 r--s 00000000 08:01 1313106                    /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7ff2d0b4b000-7ff2d0b50000 rw-p 00000000 00:00 0 
7ff2d0b50000-7ff2d0b51000 r--p 00000000 08:01 1572954                    /usr/lib/locale/en_AG/LC_MESSAGES/SYS_LC_MESSAGES
7ff2d0b51000-7ff2d0b52000 r--p 00000000 08:01 1445551                    /usr/lib/locale/chr_US/LC_PAPER
7ff2d0b52000-7ff2d0b53000 r--p 00000000 08:01 1445550                    /usr/lib/locale/chr_US/LC_NAME
7ff2d0b53000-7ff2d0b54000 r--p 00000000 08:01 1573164                    /usr/lib/locale/en_US.utf8/LC_ADDRESS
7ff2d0b54000-7ff2d0b55000 r--p 00000000 08:01 1445552                    /usr/lib/locale/chr_US/LC_TELEPHONE
7ff2d0b55000-7ff2d0b56000 r--p 00000000 08:01 1445546                    /usr/lib/locale/chr_US/LC_MEASUREMENT
7ff2d0b56000-7ff2d0b57000 r--p 00000000 08:01 1573165                    /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
7ff2d0b57000-7ff2d0b5a000 rw-p 00000000 00:00 0 
7ff2d0b5a000-7ff2d0b5b000 r--p 00023000 08:01 1703953                    /lib/x86_64-linux-gnu/ld-2.24.so
7ff2d0b5b000-7ff2d0b5c000 rw-p 00024000 08:01 1703953                    /lib/x86_64-linux-gnu/ld-2.24.so
7ff2d0b5c000-7ff2d0b5d000 rw-p 00000000 00:00 0 
7ffdc5f63000-7ffdc5f6b000 ---p 00000000 00:00 0 
7ffdc6741000-7ffdc6762000 rw-p 00000000 00:00 0                          [stack]
7ffdc67ec000-7ffdc67ee000 r--p 00000000 00:00 0                          [vvar]
7ffdc67ee000-7ffdc67f0000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Stacktrace:

  at <unknown> <0xffffffff>
  at System.Runtime.Remoting.TypeInfo..ctor (System.Type) [0x000b6] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at System.Runtime.Remoting.ServerIdentity.CreateObjRef (System.Type) [0x00036] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at System.MarshalByRefObject.CreateObjRef (System.Type) [0x00018] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at System.Runtime.Remoting.RemotingServices.Marshal (System.MarshalByRefObject,string,System.Type) [0x000ff] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at System.AppDomain.GetMarshalledDomainObjRef () [0x00005] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_object__this__ (object,intptr,intptr,intptr) [0x0002a] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) [0x0000c] in <59b4eb5b3aca4652a503912ca44d4bc9>:0
  at System.AppDomain.InvokeInDomain (System.AppDomain,System.Reflection.MethodInfo,object,object[]) <0x000be>
  at System.Runtime.Remoting.RemotingServices.GetDomainProxy (System.AppDomain) <0x0005f>
  at System.AppDomain.CreateDomain (string,System.Security.Policy.Evidence,System.AppDomainSetup) <0x00223>
  at System.AppDomain.CreateDomain (string) <0x00028>
  at Driver/<>c.<Main>b__1_0 (int) <0x00072>
  at System.Linq.Parallel.SelectQueryOperator`2/SelectQueryOperatorEnumerator`1<int, int, int>.MoveNext (int&,int&) <0x0006e>
  at System.Linq.Parallel.SelectQueryOperator`2/SelectQueryOperatorEnumerator`1<int, int, int>.MoveNext (int&,int&) <0x00052>
  at System.Linq.Parallel.PipelineSpoolingTask`2<int, int>.SpoolingWork () <0x000e2>
  at System.Linq.Parallel.SpoolingTaskBase.Work () <0x00051>
  at System.Linq.Parallel.QueryTask.BaseWork (object) <0x0005c>
  at System.Linq.Parallel.QueryTask/<>c.<.cctor>b__10_0 (object) <0x0005b>
  at System.Threading.Tasks.Task.InnerInvoke () <0x0009b>
  at System.Threading.Tasks.Task.Execute () <0x00037>
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) <0x0005b>
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00192>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00042>
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) <0x000f6>
  at System.Threading.Tasks.Task.ExecuteEntry (bool) <0x000dc>
  at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x00026>
  at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x00279>
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x0001c>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) <0x00086>
/proc/self/maps:
41065000-41075000 rwxp 00000000 00:00 0 
4138f000-413cf000 rwxp 00000000 00:00 0 
41601000-41641000 rwxp 00000000 00:00 0 
41f82000-41fa2000 rwxp 00000000 00:00 0 
560ad1908000-560ad1d66000 r-xp 00000000 08:01 133517                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen
560ad1f65000-560ad1f6c000 r--p 0045d000 08:01 133517                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen
560ad1f6c000-560ad1f71000 rw-p 00464000 08:01 133517                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen
560ad1f71000-560ad2002000 rw-p 00000000 00:00 0 
560ad25be000-560ad275e000 rw-p 00000000 00:00 0                          [heap]
7ff2b0000000-7ff2b0021000 rw-p 00000000 00:00 0 
7ff2b0021000-7ff2b4000000 ---p 00000000 00:00 0 
7ff2b8000000-7ff2b81c2000 rw-p 00000000 00:00 0 
7ff2b81c2000-7ff2bc000000 ---p 00000000 00:00 0 
7ff2bc000000-7ff2bc152000 rw-p 00000000 00:00 0 
7ff2bc152000-7ff2c0000000 ---p 00000000 00:00 0 
7ff2c0000000-7ff2c0021000 rw-p 00000000 00:00 0 
7ff2c0021000-7ff2c4000000 ---p 00000000 00:00 0 
7ff2c4000000-7ff2c4021000 rw-p 00000000 00:00 0 
7ff2c4021000-7ff2c8000000 ---p 00000000 00:00 0 
7ff2c8000000-7ff2c8021000 rw-p 00000000 00:00 0 
7ff2c8021000-7ff2cc000000 ---p 00000000 00:00 0 
7ff2cc055000-7ff2cc0a2000 r--p 00000000 08:01 133232                     /home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/Mono.Security.dll
7ff2cc0a2000-7ff2cc0ab000 ---p 00000000 00:00 0 
7ff2cc0ab000-7ff2cc2a3000 rw-p 00000000 00:00 0 
7ff2cc2a3000-7ff2cc2ac000 ---p 00000000 00:00 0 
Memory around native instruction pointer (0x7ff2cfa86fff):
0x7ff2cfa86fef  d2 4c 89 ce bf 02 00 00 00 b8 0e 00 00 00 0f 05  .L..............
0x7ff2cfa86fff  48 81 c4 88 00 00 00 44 89 c0 5b c3 0f 1f 44 00  H......D..[...D.
0x7ff2cfa8700f  00 48 8b 15 61 5e 36 00 f7 d8 64 89 02 eb ca 66  .H..a^6...d....f
0x7ff2cfa8701f  90 85 ff 78 0c f7 df e9 e5 02 00 00 0f 1f 44 00  ...x..........D.

Native stacktrace:

	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x12a52a) [0x560ad1a3252a]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x12a810) [0x560ad1a32810]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0xc0ee5) [0x560ad19c8ee5]
	/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7ff2d001b0c0]
	/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7ff2cfa86fff]
	/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7ff2cfa8842a]
	/lib/x86_64-linux-gnu/libc.so.6(+0x70c00) [0x7ff2cfac4c00]
	/lib/x86_64-linux-gnu/libc.so.6(+0x76fc6) [0x7ff2cfacafc6]
	/lib/x86_64-linux-gnu/libc.so.6(+0x7780e) [0x7ff2cfacb80e]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x12fc7a) [0x560ad1a37c7a]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x131df4) [0x560ad1a39df4]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x3fc07) [0x560ad1947c07]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0x3ff91) [0x560ad1947f91]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0xc4675) [0x560ad19cc675]
	/home/helixbot/dotnetbuild/work/9cc05752-4ecc-4c52-98dd-84817e4e68a8/Payload/net_4_x/mono-sgen(+0xc529f) [0x560ad19cd29f]
	[0x41066513]

Waiting for dumping threads to resume


Debug info from gdb:

[New LWP 2406]
[New LWP 2407]
[New LWP 2408]
[New LWP 2409]
[New LWP 2410]
[New LWP 2411]
[New LWP 2413]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185	../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7ff2d0b4b740 (LWP 2405) "mono-sgen" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  2    Thread 0x7ff2cf3ff700 (LWP 2406) "SGen worker" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7ff2ccd18700 (LWP 2407) "Finalizer" 0x00007ff2d0019536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7ff2c8000920) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7ff2cc8a5700 (LWP 2408) "mono-sgen" 0x00007ff2d0019536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7ff2c0000920) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7ff2cc6a4700 (LWP 2409) "Thread Pool Wor" 0x00007ff2d0019700 in futex_abstimed_wait_cancelable (private=0, abstime=0x7ff2cc6a3de0, expected=0, futex_word=0x560ad1f77a48 <worker+72>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  6    Thread 0x7ff2cc4a3700 (LWP 2410) "Thread Pool Wor" 0x00007ff2d001ab3a in __waitpid (pid=pid@entry=2414, stat_loc=stat_loc@entry=0x7ff2cc4a01c4, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  7    Thread 0x7ff2cc2a2700 (LWP 2411) "Thread Pool Wor" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7ff2b7bfe700 (LWP 2413) "Domain unloader" 0x00007ff2d0019536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x560ad2000ba0 <suspend_semaphore>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205

Thread 8 (Thread 0x7ff2b7bfe700 (LWP 2413)):
#0  0x00007ff2d0019536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x560ad2000ba0 <suspend_semaphore>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x560ad2000ba0 <suspend_semaphore>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007ff2d00195e4 in __new_sem_wait_slow (sem=0x560ad2000ba0 <suspend_semaphore>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007ff2d0019679 in __new_sem_wait (sem=sem@entry=0x560ad2000ba0 <suspend_semaphore>) at sem_wait.c:29
#4  0x0000560ad1c10dd1 in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#5  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=4294967295, sem=0x560ad2000ba0 <suspend_semaphore>) at ../../mono/utils/mono-os-semaphore.h:242
#6  mono_threads_wait_pending_operations () at mono-threads.c:311
#7  0x0000560ad1bc283b in sgen_unified_suspend_stop_world () at sgen-stw.c:297
#8  sgen_client_stop_world (generation=generation@entry=1, serial_collection=serial_collection@entry=1) at sgen-stw.c:124
#9  0x0000560ad1bd276c in sgen_stop_world (generation=1, serial_collection=1) at sgen-gc.c:3821
#10 0x0000560ad1bd28e9 in sgen_perform_collection_inner (requested_size=requested_size@entry=0, generation_to_collect=generation_to_collect@entry=1, reason=0x560ad1ce374c "user request", forced_serial=1, stw=1) at sgen-gc.c:2530
#11 0x0000560ad1bd2f99 in sgen_perform_collection (stw=1, forced_serial=1, reason=0x560ad1ce374c "user request", generation_to_collect=1, requested_size=0) at sgen-gc.c:2635
#12 sgen_gc_collect (generation=generation@entry=1) at sgen-gc.c:3087
#13 0x0000560ad1bad09e in mono_gc_collect (generation=1) at sgen-mono.c:2441
#14 0x0000560ad1b9ffbc in mono_domain_finalize (domain=domain@entry=0x7ff2bc0026d0, timeout=timeout@entry=4294967295) at gc.c:455
#15 0x0000560ad1a9d001 in unload_thread_main (arg=arg@entry=0x7ff2b0153aa0) at appdomain.c:2853
#16 0x0000560ad1b54cfe in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#17 start_wrapper (data=0x7ff2b0153a40) at threads.c:1234
#18 0x00007ff2d0011494 in start_thread (arg=0x7ff2b7bfe700) at pthread_create.c:333
#19 0x00007ff2cfb3cacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 7 (Thread 0x7ff2cc2a2700 (LWP 2411)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ad1bfdd7f in mono_os_cond_wait (mutex=0x560ad1f860c0 <signal_mutex>, cond=0x7ff2cc2a1320) at mono-os-mutex.h:173
#2  mono_os_event_wait_multiple (events=events@entry=0x7ff2cc2a1398, nevents=nevents@entry=1, waitall=waitall@entry=1, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at os-event-unix.c:190
#3  0x0000560ad1bfe002 in mono_os_event_wait_one (event=<optimized out>, event@entry=0x7ff2a4000f60, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at os-event-unix.c:94
#4  0x0000560ad1c1348d in mono_thread_info_wait_one_handle (thread_handle=thread_handle@entry=0x7ff2a4000f50, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at mono-threads.c:1936
#5  0x0000560ad1aa20a7 in guarded_wait (timeout=4294967295, alertable=1, thread_handle=0x7ff2a4000f50) at appdomain.c:2931
#6  mono_domain_try_unload (domain=domain@entry=0x7ff2bc0026d0, exc=exc@entry=0x7ff2cc2a14a8) at appdomain.c:3033
#7  0x0000560ad1aa2216 in ves_icall_System_AppDomain_InternalUnload (domain_id=domain_id@entry=2, error=error@entry=0x7ff2cc2a1540) at appdomain.c:2559
#8  0x0000560ad1ae31de in ves_icall_System_AppDomain_InternalUnload_raw (a0=2, error=0x7ff2cc2a1540) at ../../mono/metadata/icall-def.h:185
#9  0x0000000041fb37c5 in ?? ()
#10 0x00007ff2cfa002c0 in ?? ()
#11 0x00007ff2cf408630 in ?? ()
#12 0x00007ff2cf403eb8 in ?? ()
#13 0x00007ff2cf403f50 in ?? ()
#14 0x00007ff2cf4a7e30 in ?? ()
#15 0x00007ff2bc000fd0 in ?? ()
#16 0x00007ff2cc2a1770 in ?? ()
#17 0x00007ff2cc2a1500 in ?? ()
#18 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7ff2cc4a3700 (LWP 2410)):
#0  0x00007ff2d001ab3a in __waitpid (pid=pid@entry=2414, stat_loc=stat_loc@entry=0x7ff2cc4a01c4, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x0000560ad1a3266c in dump_native_stacktrace (ctx=ctx@entry=0x7ff2cc4a0dc0, signal=0x560ad1c40b2e "SIGABRT") at mini-posix.c:1083
#2  0x0000560ad1a32810 in mono_dump_native_crash_info (signal=signal@entry=0x560ad1c40b2e "SIGABRT", ctx=ctx@entry=0x7ff2cc4a0dc0, info=info@entry=0x7ff2cc4a0ef0) at mini-posix.c:1112
#3  0x0000560ad19c8ee5 in mono_handle_native_crash (signal=0x560ad1c40b2e "SIGABRT", ctx=0x7ff2cc4a0dc0, info=0x7ff2cc4a0ef0) at mini-exceptions.c:3196
#4  <signal handler called>
#5  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#6  0x00007ff2cfa8842a in __GI_abort () at abort.c:89
#7  0x00007ff2cfac4c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ff2cfbb9d98 "*** Error in `%s': %s: 0x%s ***
") at ../sysdeps/posix/libc_fatal.c:175
#8  0x00007ff2cfacafc6 in malloc_printerr (action=3, str=0x7ff2cfbb9ef0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
#9  0x00007ff2cfacb80e in _int_free (av=0x7ff2bc000020, p=0x7ff2bc0689a0, have_lock=0) at malloc.c:3905
#10 0x0000560ad1a37c7a in mono_compile_create_vars (cfg=0x7ff2b811a260) at mini.c:2085
#11 mini_method_compile (method=method@entry=0x7ff2b80db310, opts=opts@entry=374434303, domain=domain@entry=0x7ff2b8016a50, flags=flags@entry=JIT_FLAG_RUN_CCTORS, parts=parts@entry=0, aot_method_index=aot_method_index@entry=-1) at mini.c:3478
#12 0x0000560ad1a39df4 in mono_jit_compile_method_inner (method=method@entry=0x7ff2b80db310, target_domain=target_domain@entry=0x7ff2b8016a50, opt=opt@entry=374434303, error=error@entry=0x7ff2cc4a1c40) at mini.c:4071
#13 0x0000560ad1947c07 in mono_jit_compile_method_with_opt (method=0x7ff2b80db310, opt=374434303, jit_only=jit_only@entry=1, error=error@entry=0x7ff2cc4a1c40) at mini-runtime.c:2436
#14 0x0000560ad1948c2f in mono_jit_compile_method_jit_only (method=<optimized out>, error=error@entry=0x7ff2cc4a1c40) at mini-runtime.c:2499
#15 0x0000560ad1947f91 in compile_special (error=0x7ff2cc4a1c40, target_domain=0x7ff2b8016a50, method=0x560ad26b7d78) at mini-runtime.c:2182
#16 mono_jit_compile_method_with_opt (method=method@entry=0x560ad26b7d78, opt=374434303, jit_only=jit_only@entry=0, error=error@entry=0x7ff2cc4a1c40) at mini-runtime.c:2413
#17 0x0000560ad1948bfc in mono_jit_compile_method (method=method@entry=0x560ad26b7d78, error=error@entry=0x7ff2cc4a1c40) at mini-runtime.c:2485
#18 0x0000560ad19cc675 in common_call_trampoline (regs=regs@entry=0x7ff2cc4a1d18, code=code@entry=0x416360dd "H\213\330Hcs\030H\277\b\036
\270\362\177", m=0x560ad26b7d78, vt=vt@entry=0x7ff2b8088b18, vtable_slot=<optimized out>, error=error@entry=0x7ff2cc4a1c40) at mini-trampolines.c:751
#19 0x0000560ad19cd29f in mono_vcall_trampoline (regs=0x7ff2cc4a1d18, code=0x416360dd "H\213\330Hcs\030H\277\b\036
\270\362\177", slot=215, tramp=<optimized out>) at mini-trampolines.c:974
#20 0x0000000041066513 in ?? ()
#21 0x00007ff2cc4a247d in ?? ()
#22 0x00007ff2cc4a1d08 in ?? ()
#23 0x00007ff2cc4a1ee0 in ?? ()
#24 0x00007ff2cc4a1d18 in ?? ()
#25 0x00007ff2b8001a80 in ?? ()
#26 0x00007ff2b8088b18 in ?? ()
#27 0x00007ff2cf9cd060 in ?? ()
#28 0x00007ff2b8001040 in ?? ()
#29 0x00007ff2cc4a2138 in ?? ()
#30 0x00007ff2cc4a1ee0 in ?? ()
#31 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7ff2cc6a4700 (LWP 2409)):
#0  0x00007ff2d0019700 in futex_abstimed_wait_cancelable (private=0, abstime=0x7ff2cc6a3de0, expected=0, futex_word=0x560ad1f77a48 <worker+72>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x560ad1f77a48 <worker+72>, abstime=abstime@entry=0x7ff2cc6a3de0) at sem_waitcommon.c:111
#2  0x00007ff2d00197cf in __new_sem_wait_slow (sem=0x560ad1f77a48 <worker+72>, abstime=0x7ff2cc6a3de0) at sem_waitcommon.c:181
#3  0x00007ff2d0019882 in sem_timedwait (sem=sem@entry=0x560ad1f77a48 <worker+72>, abstime=abstime@entry=0x7ff2cc6a3de0) at sem_timedwait.c:36
#4  0x0000560ad1bb3ffa in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0x560ad1f77a48 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#5  mono_coop_sem_timedwait (sem=0x560ad1f77a48 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#6  worker_park () at threadpool-worker-default.c:391
#7  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:490
#8  0x0000560ad1b54cfe in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#9  start_wrapper (data=0x560ad26efe50) at threads.c:1234
#10 0x00007ff2d0011494 in start_thread (arg=0x7ff2cc6a4700) at pthread_create.c:333
#11 0x00007ff2cfb3cacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 4 (Thread 0x7ff2cc8a5700 (LWP 2408)):
#0  0x00007ff2d0019536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7ff2c0000920) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7ff2c0000920, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007ff2d00195e4 in __new_sem_wait_slow (sem=0x7ff2c0000920, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007ff2d0019679 in __new_sem_wait (sem=sem@entry=0x7ff2c0000920) at sem_wait.c:29
#4  0x0000560ad1c102a2 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_NONE, sem=0x7ff2c0000920) at ../../mono/utils/mono-os-semaphore.h:209
#5  mono_thread_info_wait_for_resume (info=info@entry=0x7ff2c00008c0) at mono-threads.c:225
#6  0x0000560ad1c15319 in mono_threads_exit_gc_safe_region_unbalanced_internal (cookie=0x7ff2c00008c0, stackdata=<optimized out>) at mono-threads-coop.c:324
#7  0x0000560ad1c12e5d in mono_coop_cond_timedwait (cond=0x560ad2000ac0 <sleep_cond>, mutex=0x560ad2000b00 <sleep_mutex>, timeout_ms=500) at ../../mono/utils/mono-coop-mutex.h:104
#8  sleep_interruptable (alerted=0x7ff2cc8a4dfc, ms=500) at mono-threads.c:1572
#9  mono_thread_info_sleep (ms=ms@entry=500, alerted=alerted@entry=0x7ff2cc8a4dfc) at mono-threads.c:1604
#10 0x0000560ad1bb4a92 in monitor_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:708
#11 0x0000560ad1b54cfe in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#12 start_wrapper (data=0x560ad26efe50) at threads.c:1234
#13 0x00007ff2d0011494 in start_thread (arg=0x7ff2cc8a5700) at pthread_create.c:333
#14 0x00007ff2cfb3cacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7ff2ccd18700 (LWP 2407)):
#0  0x00007ff2d0019536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7ff2c8000920) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x7ff2c8000920, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007ff2d00195e4 in __new_sem_wait_slow (sem=0x7ff2c8000920, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007ff2d0019679 in __new_sem_wait (sem=sem@entry=0x7ff2c8000920) at sem_wait.c:29
#4  0x0000560ad1c102a2 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_NONE, sem=0x7ff2c8000920) at ../../mono/utils/mono-os-semaphore.h:209
#5  mono_thread_info_wait_for_resume (info=info@entry=0x7ff2c80008c0) at mono-threads.c:225
#6  0x0000560ad1c15319 in mono_threads_exit_gc_safe_region_unbalanced_internal (cookie=0x7ff2c80008c0, stackdata=<optimized out>) at mono-threads-coop.c:324
#7  0x0000560ad1b526cf in threads_native_thread_join_lock (value=<optimized out>, tid=0x7ff2b7dff700) at threads.c:5560
#8  mono_threads_join_threads () at threads.c:5802
#9  0x0000560ad1b9f5d4 in mono_runtime_do_background_work () at gc.c:887
#10 finalizer_thread (unused=unused@entry=0x0) at gc.c:926
#11 0x0000560ad1b54cfe in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1174
#12 start_wrapper (data=0x560ad261e3c0) at threads.c:1234
#13 0x00007ff2d0011494 in start_thread (arg=0x7ff2ccd18700) at pthread_create.c:333
#14 0x00007ff2cfb3cacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7ff2cf3ff700 (LWP 2406)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ad1bfaedb in mono_os_cond_wait (mutex=0x560ad1f85f00 <lock>, cond=0x560ad1f85ec0 <work_cond>) at ../../mono/utils/mono-os-mutex.h:173
#2  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=0) at sgen-thread-pool.c:165
#3  thread_func (data=<optimized out>) at sgen-thread-pool.c:196
#4  0x00007ff2d0011494 in start_thread (arg=0x7ff2cf3ff700) at pthread_create.c:333
#5  0x00007ff2cfb3cacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7ff2d0b4b740 (LWP 2405)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ad1c08db5 in mono_os_cond_wait (mutex=0x560ad25c2608, cond=0x560ad25c2630) at mono-os-mutex.h:173
#2  mono_os_cond_timedwait (cond=cond@entry=0x560ad25c2630, mutex=mutex@entry=0x560ad25c2608, timeout_ms=timeout_ms@entry=4294967295) at mono-os-mutex.c:32
#3  0x0000560ad1b6d255 in mono_coop_cond_timedwait (timeout_ms=4294967295, mutex=0x560ad25c2608, cond=0x560ad25c2630) at ../../mono/utils/mono-coop-mutex.h:102
#4  mono_w32handle_timedwait_signal_naked (alerted=0x7ffdc675e900, poll=0, timeout=4294967295, mutex=0x560ad25c2608, cond=0x560ad25c2630) at w32handle.c:650
#5  mono_w32handle_timedwait_signal_handle (handle_data=0x560ad25c25f8, timeout=timeout@entry=4294967295, alerted=alerted@entry=0x7ffdc675e900, poll=0) at w32handle.c:765
#6  0x0000560ad1b6e55f in mono_w32handle_wait_one (handle=handle@entry=0x560ad25c25f8, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at w32handle.c:873
#7  0x0000560ad1ba8498 in ves_icall_System_Threading_Monitor_Monitor_wait (obj=0x7ff2cf407870, ms=4294967295) at monitor.c:1390
#8  0x0000000041f91316 in ?? ()
#9  0x00007ff2cf407870 in ?? ()
#10 0xffffffffffffffff in ?? ()
#11 0x0000000000000000 in ?? ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Oct 31, 2018

that's the interesting bit:

#8  0xf7571dd0 in raise () from /lib/i386-linux-gnu/libc.so.6
#9  0xf7573297 in abort () from /lib/i386-linux-gnu/libc.so.6
#10 0xf75ad38f in ?? () from /lib/i386-linux-gnu/libc.so.6
#11 0xf75b3fc7 in ?? () from /lib/i386-linux-gnu/libc.so.6
#12 0xf75b4806 in ?? () from /lib/i386-linux-gnu/libc.so.6
#13 0x568b4f70 in monoeg_g_free (ptr=0x583c6b28) at gmem.c:71
#14 0x5689f966 in mono_bitset_free (set=0x583c6b28) at monobitset.c:77
#15 0x566be367 in mono_compile_create_vars (cfg=<optimized out>) at mini.c:2085
#16 mini_method_compile (method=<optimized out>, opts=<optimized out>, domain=<optimized out>, flags=<optimized out>, parts=<optimized out>, aot_method_index=<optimized out>) at mini.c:3478
#17 0x566c045e in mono_jit_compile_method_inner (method=0x583ed558, target_domain=0xf4523120, opt=382822911, error=0xf47fe07c) at mini.c:4071
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Oct 31, 2018

the code that's crashing was introduced with e8b0376. @jaykrell any idea? I believe it's wrong to free (without synchronization) data from MonoMethodHeader, as this datastructure can be used by other threads; it can happen that a specific MonoMethod is compiled by two different threads at the same time. The compilation result should be the same, just one compilation thread wins (i.e. one result will be registered with the runtime, the other will be thrown away). Specifically here, I don't think we should free those bitsets:

mono/mono/mini/mini.c

Lines 2082 to 2086 in 7a23177

mono_bitset_free (header->volatile_locals);
header->volatile_locals = NULL;
mono_bitset_free (header->volatile_args);
header->volatile_args = NULL;

@vargaz

This comment has been minimized.

Copy link
Member

@vargaz vargaz commented Oct 31, 2018

That memory should be allocated from some kind of domain/image mempool, so it doesn't have to be freed.

@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Oct 31, 2018

Oh, understood. Or, move it to the cfg? Or, make it fixed size? Or, make all locals/parameters in icalls+uses_handles volatile?

@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Oct 31, 2018

mono_mempool_alloc0 (cfg->mempool?

@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Oct 31, 2018

Or, header itself is usually (non-dynamic) from mono_image_alloc0 (image.

lewurm added a commit to lewurm/mono that referenced this issue Oct 31, 2018
akoeplinger added a commit to akoeplinger/mono that referenced this issue Oct 31, 2018
jaykrell added a commit to jaykrell/mono that referenced this issue Oct 31, 2018
…s to end of MonoMethodHeader

in order to get its lifetime.

Alernative to mono#11490 though maybe
mono#11490 is simpler and faster.

Fixes mono#11488.
My regression from mono#11294.
jaykrell added a commit to jaykrell/mono that referenced this issue Oct 31, 2018
…s to end of MonoMethodHeader

in order to get its lifetime.

Alernative to mono#11490 though maybe
mono#11490 is simpler and faster.

Fixes mono#11488.
My regression from mono#11294.
jaykrell added a commit to jaykrell/mono that referenced this issue Oct 31, 2018
…s to MonoMethodHeader

in order to get its lifetime.

Alernative to mono#11490 though maybe
mono#11490 is simpler and faster.

Fixes mono#11488.
My regression from mono#11294.
akoeplinger added a commit to akoeplinger/mono that referenced this issue Oct 31, 2018
akoeplinger added a commit to akoeplinger/mono that referenced this issue Oct 31, 2018
jaykrell added a commit to jaykrell/mono that referenced this issue Oct 31, 2018
…s to MonoMethodHeader

in order to get its lifetime.

Alernative to mono#11490 though maybe
mono#11490 is simpler and faster.

Fixes mono#11488.
My regression from mono#11294.
akoeplinger added a commit that referenced this issue Oct 31, 2018
…1490)

Fixes #11488, an regression introduced by #11294
jonpryor added a commit to xamarin/xamarin-android that referenced this issue Apr 24, 2019
Bumps to mono/api-snapshot@ae01378
Bumps to mono/reference-assemblies@e5173a5
Bumps to mono/bockbuild@d30329d
Bumps to mono/boringssl@3d87996
Bumps to mono/corefx@72f7d76
Bumps to mono/corert@1b7d4a1
Bumps to mono/helix-binaries@7e893ea
Bumps to mono/illinker-test-assets@f21ff68
Bumps to mono/linker@13d864e
Bumps to mono/llvm@1aaaaa5 [mono]
Bumps to mono/llvm@2c2cffe [xamarin-android]
Bumps to mono/NUnitLite@0029561
Bumps to mono/roslyn-binaries@0bbc9b4
Bumps to mono/xunit-binaries@8f6e62e

	$ git diff --shortstat 886c4901..e66c7667      # mono
        3597 files changed, 350850 insertions(+), 91128 deletions(-)
	$ git diff --shortstat 349752c464c5fc93b32e7d45825f2890c85c8b7d..2c2cffedf01e0fe266b9aaad2c2563e05b750ff4
	 240 files changed, 18562 insertions(+), 6581 deletions(-)

Context: dotnet/coreclr#22046

Fixes: CVE 2018-8292 on macOS
Fixes: http://work.devdiv.io/737323
Fixes: dotnet/corefx#33965
Fixes: dotnet/standard#642
Fixes: mono/mono#6997
Fixes: mono/mono#7326
Fixes: mono/mono#7517
Fixes: mono/mono#7750
Fixes: mono/mono#7859
Fixes: mono/mono#8360
Fixes: mono/mono#8460
Fixes: mono/mono#8766
Fixes: mono/mono#8922
Fixes: mono/mono#9418
Fixes: mono/mono#9507
Fixes: mono/mono#9951
Fixes: mono/mono#10024
Fixes: mono/mono#10030
Fixes: mono/mono#10038
Fixes: mono/mono#10448
Fixes: mono/mono#10735
Fixes: mono/mono#10735
Fixes: mono/mono#10737
Fixes: mono/mono#10743
Fixes: mono/mono#10834
Fixes: mono/mono#10837
Fixes: mono/mono#10838
Fixes: mono/mono#10863
Fixes: mono/mono#10945
Fixes: mono/mono#11020
Fixes: mono/mono#11021
Fixes: mono/mono#11021
Fixes: mono/mono#11049
Fixes: mono/mono#11091
Fixes: mono/mono#11095
Fixes: mono/mono#11123
Fixes: mono/mono#11138
Fixes: mono/mono#11146
Fixes: mono/mono#11202
Fixes: mono/mono#11214
Fixes: mono/mono#11317
Fixes: mono/mono#11326
Fixes: mono/mono#11378
Fixes: mono/mono#11385
Fixes: mono/mono#11478
Fixes: mono/mono#11479
Fixes: mono/mono#11488
Fixes: mono/mono#11489
Fixes: mono/mono#11527
Fixes: mono/mono#11529
Fixes: mono/mono#11596
Fixes: mono/mono#11603
Fixes: mono/mono#11613
Fixes: mono/mono#11623
Fixes: mono/mono#11663
Fixes: mono/mono#11681
Fixes: mono/mono#11684
Fixes: mono/mono#11693
Fixes: mono/mono#11697
Fixes: mono/mono#11779
Fixes: mono/mono#11809
Fixes: mono/mono#11858
Fixes: mono/mono#11895
Fixes: mono/mono#11898
Fixes: mono/mono#11898
Fixes: mono/mono#11965
Fixes: mono/mono#12182
Fixes: mono/mono#12193
Fixes: mono/mono#12218
Fixes: mono/mono#12235
Fixes: mono/mono#12263
Fixes: mono/mono#12307
Fixes: mono/mono#12331
Fixes: mono/mono#12362
Fixes: mono/mono#12374
Fixes: mono/mono#12402
Fixes: mono/mono#12421
Fixes: mono/mono#12461
Fixes: mono/mono#12479
Fixes: mono/mono#12479
Fixes: mono/mono#12552
Fixes: mono/mono#12603
Fixes: mono/mono#12747
Fixes: mono/mono#12831
Fixes: mono/mono#12843
Fixes: mono/mono#12881
Fixes: mono/mono#13030
Fixes: mono/mono#13284
Fixes: mono/mono#13297
Fixes: mono/mono#13455
Fixes: mono/mono#13460
Fixes: mono/mono#13478
Fixes: mono/mono#13479
Fixes: mono/mono#13522
Fixes: mono/mono#13607
Fixes: mono/mono#13610
Fixes: mono/mono#13610
Fixes: mono/mono#13639
Fixes: mono/mono#13672
Fixes: mono/mono#13834
Fixes: mono/mono#13878
Fixes: mono/mono#6352
Fixes: mono/monodevelop#6898
Fixes: xamarin/maccore#1069
Fixes: xamarin/maccore#1407
Fixes: xamarin/maccore#604
Fixes: xamarin/xamarin-macios#4984
Fixes: xamarin/xamarin-macios#5289
Fixes: xamarin/xamarin-macios#5363
Fixes: xamarin/xamarin-macios#5381
Fixes: https://issuetracker.unity3d.com/issues/editor-crashes-with-g-logv-when-entering-play-mode-with-active-flowcanvas-script
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.