PR Title #1
Closed
PR Title #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
luhenry
pushed a commit
that referenced
this pull request
Dec 8, 2015
We've seen that error for quite some time: > `* Assertion at ../../mono/utils/mono-os-mutex.h:71, condition `res != EINVAL' not met` It looks like there's a race between `finalizer_thread()` and `mono_gc_cleanup()` around `reference_queue_mutex`. The source of the problem is, `finalizer_thread` could end up being alive, although `mono_gc_cleanup()` made some effort to kill it and then destroys `reference_queue_mutex`, while the finalizer thread is still using it in `reference_queue_proccess_all()`. > $ MONO_GC_DEBUG=bridge=Bridge MONO_GC_PARAMS=minor=split MONO_ENV_OPTIONS=--gc=sgen MONO_PATH=/home/lewurm/monoperf/mono/mcs/class/lib/net_4_x ../../mono/mini/mono sgen-bridge-major-fragmentation.exe --optimize=all --debug > [...] > [12/08/2015 01:49:31] done > Shutting down finalizer thread timed out. > * Assertion at gc.c:867, condition `finalizer_thread_exited' not met > > Stacktrace: > > > Native stacktrace: > > > Debug info from gdb: > > [New LWP 7623] > [New LWP 7622] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". > 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > Id Target Id Frame > 3 Thread 0x417ff430 (LWP 7622) "mono" 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > 2 Thread 0x42573430 (LWP 7623) "Finalizer" 0x0021482a in finalizers_with_predicate (predicate=0x1fd121 <object_in_domain_predicate>, user_data=0x37a348, out_array=0x42572c70, out_size=64, hash_table=0x338f68 <major_finalizable_hash>) at sgen-fin-weak-hash.c:581 > * 1 Thread 0x406c4250 (LWP 7620) "mono" 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > > Thread 3 (Thread 0x417ff430 (LWP 7622)): > #0 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #1 0x404371d8 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #2 0x00242266 in mono_os_cond_wait (cond=0x346e98 <work_cond>, mutex=0x346e80 <lock>) at ../../mono/utils/mono-os-mutex.h:105 > #3 0x00242c1a in thread_func (thread_data=0x0) at sgen-thread-pool.c:118 > #4 0x40433fbc in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0 > mono#5 0x405a0b3c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6 > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > > Thread 2 (Thread 0x42573430 (LWP 7623)): > #0 0x0021482a in finalizers_with_predicate (predicate=0x1fd121 <object_in_domain_predicate>, user_data=0x37a348, out_array=0x42572c70, out_size=64, hash_table=0x338f68 <major_finalizable_hash>) at sgen-fin-weak-hash.c:581 > #1 0x00214894 in sgen_gather_finalizers_if (predicate=0x1fd121 <object_in_domain_predicate>, user_data=0x37a348, out_array=0x42572c70, out_size=64) at sgen-fin-weak-hash.c:622 > #2 0x001fd16a in mono_gc_finalizers_for_domain (domain=0x37a348, out_array=0x42572c70, out_size=64) at sgen-mono.c:546 > #3 0x001ca2a4 in finalize_domain_objects (req=0x47ba68) at gc.c:678 > #4 0x001ca3b6 in finalizer_thread (unused=0x0) at gc.c:730 > mono#5 0x001a1392 in start_wrapper_internal (data=0x3b1100) at threads.c:713 > mono#6 0x001a1430 in start_wrapper (data=0x3b1100) at threads.c:760 > mono#7 0x0027097a in inner_start_thread (arg=0xbed58de0) at mono-threads-posix.c:92 > mono#8 0x40433fbc in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0 > mono#9 0x405a0b3c in ?? () from /lib/arm-linux-gnueabihf/libc.so.6 > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > > Thread 1 (Thread 0x406c4250 (LWP 7620)): > #0 0x4043b5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #1 0x4043a396 in waitpid () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #2 0x000d1b24 in mono_handle_native_sigsegv (signal=6, ctx=0xbed58a80, info=0xbed58a00) at mini-exceptions.c:2235 > #3 0x00112424 in sigabrt_signal_handler (_dummy=6, _info=0xbed58a00, context=0xbed58a80) at mini-posix.c:218 > #4 <signal handler called> > mono#5 0x405248e6 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6 > mono#6 0x405330fe in raise () from /lib/arm-linux-gnueabihf/libc.so.6 > mono#7 0x40535956 in abort () from /lib/arm-linux-gnueabihf/libc.so.6 > mono#8 0x00277eaa in monoeg_log_default_handler (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, message=0x47bb28 "* Assertion at gc.c:867, condition `finalizer_thread_exited' not met\n", unused_data=0x0) at goutput.c:233 > mono#9 0x00277dca in monoeg_g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, format=0x3089b8 "* Assertion at %s:%d, condition `%s' not met\n", args=...) at goutput.c:113 > mono#10 0x00277e32 in monoeg_assertion_message (format=0x3089b8 "* Assertion at %s:%d, condition `%s' not met\n") at goutput.c:133 > mono#11 0x001ca786 in mono_gc_cleanup () at gc.c:867 > mono#12 0x001c1d24 in mono_runtime_cleanup (domain=0x37a348) at appdomain.c:356 > mono#13 0x0001dec0 in mini_cleanup (domain=0x37a348) at mini-runtime.c:3560 > mono#14 0x000a672c in mono_main (argc=5, argv=0x358098) at driver.c:2065 > mono#15 0x00017c4e in mono_main_with_options (argc=5, argv=0x358098) at main.c:20 > mono#16 0x00017c7e in main (argc=4, argv=0xbed591b4) at main.c:53 ================================================================= Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. =================================================================
luhenry
pushed a commit
that referenced
this pull request
Feb 3, 2016
there's a short window where we release the lock and another thread could modify the state. That is what we saw on the ARM machines for a while, e.g. in `subthread-exit.cs` there's a race between `mono_thread_execute_interruption` and `mono_thread_suspend_all_other_threads`: > Thread 2 (Thread 0x429ff430 (LWP 5189)): // SUB THREAD > #0 0x40398e30 in nanosleep () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #1 0x0028c0ac in monoeg_g_usleep (microseconds=86650) at gdate-unix.c:53 > #2 0x0027b1d8 in suspend_sync_nolock (id=1079603792, interrupt_kernel=1) at mono-threads.c:913 > #3 0x0027b2d4 in mono_thread_info_safe_suspend_and_run (id=1079603792, interrupt_kernel=1, callback=0x1ab109 <suspend_thread_critical>, user_data=0x429fe534) > at mono-threads.c:935 > #4 0x001ab28a in suspend_thread_internal (thread=0x401c0120, interrupt=1) at threads.c:4807 > mono#5 0x001a91ec in mono_thread_suspend_all_other_threads () at threads.c:3297 > mono#6 0x00158d8c in ves_icall_System_Environment_Exit (result=0) at icall.c:6378 > mono#7 0x400e7d9c in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > > Thread 1 (Thread 0x40597250 (LWP 5186)): // MAIN THREAD > #0 0x4039a5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #1 0x4039796e in do_futex_wait () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #2 0x403979da in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #3 0x002788d8 in mono_os_sem_wait (sem=0x388248, flags=MONO_SEM_FLAGS_NONE) at ../../mono/utils/mono-os-semaphore.h:163 > #4 0x00279722 in mono_thread_info_wait_for_resume (info=0x388210) at mono-threads.c:144 > mono#5 0x0027ad5e in mono_thread_info_end_self_suspend () at mono-threads.c:700 > mono#6 0x001ab2d2 in self_suspend_internal (thread=0x401c0120) at threads.c:4822 > mono#7 0x001aab0a in mono_thread_execute_interruption () at threads.c:4337 > mono#8 0x001a6a82 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1200) at threads.c:1217 > mono#9 0x400da75c in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?)
luhenry
pushed a commit
that referenced
this pull request
Feb 3, 2016
[threads] avoid race between setting the state flag and suspension there's a short window where we release the lock and another thread could modify the state. That is what we saw on the ARM machines for a while, e.g. in `subthread-exit.cs` there's a race between `mono_thread_execute_interruption` and `mono_thread_suspend_all_other_threads`: > Thread 2 (Thread 0x429ff430 (LWP 5189)): // SUB THREAD > #0 0x40398e30 in nanosleep () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #1 0x0028c0ac in monoeg_g_usleep (microseconds=86650) at gdate-unix.c:53 > #2 0x0027b1d8 in suspend_sync_nolock (id=1079603792, interrupt_kernel=1) at mono-threads.c:913 > #3 0x0027b2d4 in mono_thread_info_safe_suspend_and_run (id=1079603792, interrupt_kernel=1, callback=0x1ab109 <suspend_thread_critical>, user_data=0x429fe534) > at mono-threads.c:935 > #4 0x001ab28a in suspend_thread_internal (thread=0x401c0120, interrupt=1) at threads.c:4807 > mono#5 0x001a91ec in mono_thread_suspend_all_other_threads () at threads.c:3297 > mono#6 0x00158d8c in ves_icall_System_Environment_Exit (result=0) at icall.c:6378 > mono#7 0x400e7d9c in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > > Thread 1 (Thread 0x40597250 (LWP 5186)): // MAIN THREAD > #0 0x4039a5f4 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #1 0x4039796e in do_futex_wait () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #2 0x403979da in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0 > #3 0x002788d8 in mono_os_sem_wait (sem=0x388248, flags=MONO_SEM_FLAGS_NONE) at ../../mono/utils/mono-os-semaphore.h:163 > #4 0x00279722 in mono_thread_info_wait_for_resume (info=0x388210) at mono-threads.c:144 > mono#5 0x0027ad5e in mono_thread_info_end_self_suspend () at mono-threads.c:700 > mono#6 0x001ab2d2 in self_suspend_internal (thread=0x401c0120) at threads.c:4822 > mono#7 0x001aab0a in mono_thread_execute_interruption () at threads.c:4337 > mono#8 0x001a6a82 in ves_icall_System_Threading_Thread_Sleep_internal (ms=1200) at threads.c:1217 > mono#9 0x400da75c in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?)
luhenry
pushed a commit
that referenced
this pull request
Jun 21, 2016
We were seeing intermittent failures on devices like: ``` [FAIL] TimerTest.EnabledInElapsed : #1 loss of events Expected: True But was: False at MonoTests.System.Timers.TimerTest.EnabledInElapsed () <0x33b168 + 0x0014f> in <filename unknown>:0 at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object , BindingFlags , System.Reflection.Binder , System.Object[] , System.Globalization.CultureInfo ) <0x66d400 + 0x000b7> in <filename unknown>:0 ``` Since OS scheduling and GCs can influence the exact timing of the timer intervals sleeping for 200ms is likely just too short. Instead of bumping the timeout, I decided to rewrite the test to not rely on Thread.Sleep (). I also added another test which verifies setting AutoResetEvent=false stops the timer event from firing after the first one, which was implicitly tested before by the other test. Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=41530 (cherry picked from commit bab393e)
luhenry
pushed a commit
that referenced
this pull request
Jun 22, 2016
As described in https://bugzilla.xamarin.com/show_bug.cgi?id=34883 at least Samsung SG3 doesn't allow access to the local network interface (in fact, even ifconfig doesn't work on the device). Therefore the test needs to be disabled on those devices. This is the first part of the fix, the second one will be implemented in Xamarin.Android test suite which will provide the AndroidShouldPingWork method which will check the device id against list of those that do not work for this test. Part #1 of fix for https://bugzilla.xamarin.com/show_bug.cgi?id=34883
luhenry
pushed a commit
that referenced
this pull request
Jun 23, 2016
We were seeing intermittent failures on devices like: ``` [FAIL] TimerTest.EnabledInElapsed : #1 loss of events Expected: True But was: False at MonoTests.System.Timers.TimerTest.EnabledInElapsed () <0x33b168 + 0x0014f> in <filename unknown>:0 at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object , BindingFlags , System.Reflection.Binder , System.Object[] , System.Globalization.CultureInfo ) <0x66d400 + 0x000b7> in <filename unknown>:0 ``` Since OS scheduling and GCs can influence the exact timing of the timer intervals sleeping for 200ms is likely just too short. Instead of bumping the timeout, I decided to rewrite the test to not rely on Thread.Sleep (). I also added another test which verifies setting AutoResetEvent=false stops the timer event from firing after the first one, which was implicitly tested before by the other test. Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=41530
luhenry
pushed a commit
that referenced
this pull request
Jun 23, 2016
As described in https://bugzilla.xamarin.com/show_bug.cgi?id=34883 at least Samsung SG3 doesn't allow access to the local network interface (in fact, even ifconfig doesn't work on the device). Therefore the test needs to be disabled on those devices. This is the first part of the fix, the second one will be implemented in Xamarin.Android test suite which will provide the AndroidShouldPingWork method which will check the device id against list of those that do not work for this test. Part #1 of fix for https://bugzilla.xamarin.com/show_bug.cgi?id=34883
luhenry
added a commit
that referenced
this pull request
Aug 30, 2016
We would previously observe the following crash: ``` ludovic-laptop :: ~/Xamarin/mono ‹16c07b5› » (cd tools/pedump && MONO_PATH=../../mcs/class/lib/net_4_x lldb -- ./pedump --verify metadata,code ../.././mono/tests/xdomain-threads.exe) (lldb) target create "./pedump" Current executable set to './pedump' (x86_64). (lldb) settings set -- target.run-args "--verify" "metadata,code" "../.././mono/tests/xdomain-threads.exe" (lldb) r Process 14407 launched: './pedump' (x86_64) mono_os_mutex_lock: pthread_mutex_lock failed with "Invalid argument" (22) Process 14407 stopped * thread #1: tid = 0x27c9715, 0x00007fff93cfcf06 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT frame #0: 0x00007fff93cfcf06 libsystem_kernel.dylib`__pthread_kill + 10 libsystem_kernel.dylib`__pthread_kill: -> 0x7fff93cfcf06 <+10>: jae 0x7fff93cfcf10 ; <+20> 0x7fff93cfcf08 <+12>: movq %rax, %rdi 0x7fff93cfcf0b <+15>: jmp 0x7fff93cf77cd ; cerror_nocancel 0x7fff93cfcf10 <+20>: retq (lldb) bt * thread #1: tid = 0x27c9715, 0x00007fff93cfcf06 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT * frame #0: 0x00007fff93cfcf06 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fff8b9b74ec libsystem_pthread.dylib`pthread_kill + 90 frame #2: 0x00007fff908d56e7 libsystem_c.dylib`abort + 129 frame #3: 0x000000010022da6a pedump`monoeg_log_default_handler(log_domain=0x0000000000000000, log_level=G_LOG_LEVEL_ERROR, message="mono_os_mutex_lock: pthread_mutex_lock failed with \"Invalid argument\" (22)", unused_data=0x0000000000000000) + 202 at goutput.c:231 frame #4: 0x000000010022d98d pedump`monoeg_g_logv(log_domain=0x0000000000000000, log_level=G_LOG_LEVEL_ERROR, format="%s: pthread_mutex_lock failed with \"%s\" (%d)", args=0x00007fff5fbff4a0) + 109 at goutput.c:111 frame mono#5: 0x000000010022dbd9 pedump`monoeg_g_log(log_domain=0x0000000000000000, log_level=G_LOG_LEVEL_ERROR, format="%s: pthread_mutex_lock failed with \"%s\" (%d)") + 361 at goutput.c:121 frame mono#6: 0x000000010022301e pedump`mono_os_mutex_lock(mutex=0x00000001002ec630) + 110 at mono-os-mutex.h:98 frame mono#7: 0x0000000100223a66 pedump`mono_w32handle_new(type=MONO_W32HANDLE_PROCESS, handle_specific=0x00007fff5fbff540) + 246 at w32handle.c:440 frame mono#8: 0x00000001001ef8bb pedump`_wapi_processes_init + 123 at processes.c:1149 frame mono#9: 0x00000001001fbcc3 pedump`wapi_init + 19 at wapi.c:23 frame mono#10: 0x0000000100116486 pedump`mono_init_internal(filename="pedump", exe_filename=0x0000000000000000, runtime_version="v4.0.30319") + 102 at domain.c:528 frame mono#11: 0x0000000100117124 pedump`mono_init_version(domain_name="pedump", version="v4.0.30319") + 36 at domain.c:868 frame mono#12: 0x0000000100001496 pedump`verify_image_file(fname="../.././mono/tests/xdomain-threads.exe") + 678 at pedump.c:466 frame mono#13: 0x0000000100000f87 pedump`main(argc=4, argv=0x00007fff5fbff9a0) + 1095 at pedump.c:708 frame mono#14: 0x00007fff93e655ad libdyld.dylib`start + 1 ``` That would be because the w32handle wouldn't be initialized before calling into `wapi_init` which allocates w32handle.
luhenry
added a commit
that referenced
this pull request
Feb 23, 2017
We would observe crashes like the following: ``` * Assertion at mono-threads.c:1009, condition `mono_thread_info_run_state (info) == STATE_ASYNC_SUSPENDED' not met Debug info from gdb: [New LWP 26793] [New LWP 26792] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Id Target Id Frame 3 Thread 0x2b0ff3e6c700 (LWP 26792) "SGen worker" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x2b0ff633e700 (LWP 26793) "Finalizer" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x2b0ff2b67c40 (LWP 26780) "mono" 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Thread 3 (Thread 0x2b0ff3e6c700 (LWP 26792)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000661bdf in mono_os_cond_wait (mutex=0x9dee40 <lock>, cond=0x9dee00 <work_cond>) at ../../mono/utils/mono-os-mutex.h:146 #2 thread_func (thread_data=0x2b0ff2bcb008) at sgen-thread-pool.c:129 #3 0x00002b0ff347a182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00002b0ff39a130d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 2 (Thread 0x2b0ff633e700 (LWP 26793)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005caff3 in mono_os_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-os-mutex.h:146 #2 mono_coop_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-coop-mutex.h:87 #3 mono_threadpool_io_remove_socket (fd=134239440) at threadpool-io.c:628 #4 0x00000000005ae4f4 in ves_icall_System_Net_Sockets_Socket_Close_internal (sock=42, werror=<optimized out>) at w32socket.c:708 mono#5 0x0000000041e8101f in ?? () mono#6 0x00002b1014abe070 in ?? () mono#7 0x0000000001615860 in ?? () mono#8 0x0000000001615860 in ?? () mono#9 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b0ff2b67c40 (LWP 26780)): #0 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004ac5e6 in mono_handle_native_crash (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2557 #2 <signal handler called> #3 0x00002b0ff38dcf79 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00002b0ff38e0388 in abort () from /lib/x86_64-linux-gnu/libc.so.6 mono#5 0x000000000066b459 in mono_log_write_logfile (log_domain=<optimized out>, level=<optimized out>, hdr=<optimized out>, message=<optimized out>) at mono-log-common.c:137 mono#6 0x00000000006805e0 in monoeg_g_logv (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7ffc7324ea78) at goutput.c:115 mono#7 0x0000000000680736 in monoeg_assertion_message (format=<optimized out>) at goutput.c:135 mono#8 0x00000000006759b8 in mono_thread_info_setup_async_call (info=info@entry=0x2b0ff80008c0, target_func=target_func@entry=0x5bac20 <suspend_for_shutdown_async_call>, user_data=user_data@entry=0x0) at mono-threads.c:1009 mono#9 0x00000000005bad50 in suspend_for_shutdown_critical (info=info@entry=0x2b0ff80008c0, unused=unused@entry=0x0) at threads.c:5019 mono#10 0x00000000006763ae in mono_thread_info_safe_suspend_and_run (id=47347555100416, interrupt_kernel=interrupt_kernel@entry=0, callback=callback@entry=0x5bad40 <suspend_for_shutdown_critical>, user_data=user_data@entry=0x0) at mono-threads.c:979 mono#11 0x00000000005c2e71 in mono_thread_internal_suspend_for_shutdown (thread=<optimized out>) at threads.c:5028 mono#12 0x00000000005e9100 in mono_gc_cleanup () at gc.c:1015 mono#13 0x00000000005e108e in mono_runtime_cleanup (domain=domain@entry=0x1615860) at appdomain.c:423 mono#14 0x00000000004228cb in mini_cleanup (domain=0x1615860) at mini-runtime.c:4111 mono#15 0x000000000047b99f in mono_main (argc=10, argv=<optimized out>) at driver.c:2167 mono#16 0x00000000004200db in mono_main_with_options (argv=0x7ffc7324ef68, argc=10) at main.c:45 mono#17 main (argc=10, argv=0x7ffc7324ef68) at main.c:338 ================================================================= Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= ```
luhenry
added a commit
that referenced
this pull request
Feb 24, 2017
We would observe crashes like the following: ``` * Assertion at mono-threads.c:1009, condition `mono_thread_info_run_state (info) == STATE_ASYNC_SUSPENDED' not met Debug info from gdb: [New LWP 26793] [New LWP 26792] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Id Target Id Frame 3 Thread 0x2b0ff3e6c700 (LWP 26792) "SGen worker" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x2b0ff633e700 (LWP 26793) "Finalizer" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x2b0ff2b67c40 (LWP 26780) "mono" 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Thread 3 (Thread 0x2b0ff3e6c700 (LWP 26792)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000661bdf in mono_os_cond_wait (mutex=0x9dee40 <lock>, cond=0x9dee00 <work_cond>) at ../../mono/utils/mono-os-mutex.h:146 #2 thread_func (thread_data=0x2b0ff2bcb008) at sgen-thread-pool.c:129 #3 0x00002b0ff347a182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00002b0ff39a130d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 2 (Thread 0x2b0ff633e700 (LWP 26793)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005caff3 in mono_os_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-os-mutex.h:146 #2 mono_coop_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-coop-mutex.h:87 #3 mono_threadpool_io_remove_socket (fd=134239440) at threadpool-io.c:628 #4 0x00000000005ae4f4 in ves_icall_System_Net_Sockets_Socket_Close_internal (sock=42, werror=<optimized out>) at w32socket.c:708 mono#5 0x0000000041e8101f in ?? () mono#6 0x00002b1014abe070 in ?? () mono#7 0x0000000001615860 in ?? () mono#8 0x0000000001615860 in ?? () mono#9 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b0ff2b67c40 (LWP 26780)): #0 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004ac5e6 in mono_handle_native_crash (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2557 #2 <signal handler called> #3 0x00002b0ff38dcf79 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00002b0ff38e0388 in abort () from /lib/x86_64-linux-gnu/libc.so.6 mono#5 0x000000000066b459 in mono_log_write_logfile (log_domain=<optimized out>, level=<optimized out>, hdr=<optimized out>, message=<optimized out>) at mono-log-common.c:137 mono#6 0x00000000006805e0 in monoeg_g_logv (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7ffc7324ea78) at goutput.c:115 mono#7 0x0000000000680736 in monoeg_assertion_message (format=<optimized out>) at goutput.c:135 mono#8 0x00000000006759b8 in mono_thread_info_setup_async_call (info=info@entry=0x2b0ff80008c0, target_func=target_func@entry=0x5bac20 <suspend_for_shutdown_async_call>, user_data=user_data@entry=0x0) at mono-threads.c:1009 mono#9 0x00000000005bad50 in suspend_for_shutdown_critical (info=info@entry=0x2b0ff80008c0, unused=unused@entry=0x0) at threads.c:5019 mono#10 0x00000000006763ae in mono_thread_info_safe_suspend_and_run (id=47347555100416, interrupt_kernel=interrupt_kernel@entry=0, callback=callback@entry=0x5bad40 <suspend_for_shutdown_critical>, user_data=user_data@entry=0x0) at mono-threads.c:979 mono#11 0x00000000005c2e71 in mono_thread_internal_suspend_for_shutdown (thread=<optimized out>) at threads.c:5028 mono#12 0x00000000005e9100 in mono_gc_cleanup () at gc.c:1015 mono#13 0x00000000005e108e in mono_runtime_cleanup (domain=domain@entry=0x1615860) at appdomain.c:423 mono#14 0x00000000004228cb in mini_cleanup (domain=0x1615860) at mini-runtime.c:4111 mono#15 0x000000000047b99f in mono_main (argc=10, argv=<optimized out>) at driver.c:2167 mono#16 0x00000000004200db in mono_main_with_options (argv=0x7ffc7324ef68, argc=10) at main.c:45 mono#17 main (argc=10, argv=0x7ffc7324ef68) at main.c:338 ================================================================= Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= ```
luhenry
added a commit
that referenced
this pull request
Mar 8, 2017
We would observe crashes like the following: ``` * Assertion at mono-threads.c:1009, condition `mono_thread_info_run_state (info) == STATE_ASYNC_SUSPENDED' not met Debug info from gdb: [New LWP 26793] [New LWP 26792] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Id Target Id Frame 3 Thread 0x2b0ff3e6c700 (LWP 26792) "SGen worker" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x2b0ff633e700 (LWP 26793) "Finalizer" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x2b0ff2b67c40 (LWP 26780) "mono" 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Thread 3 (Thread 0x2b0ff3e6c700 (LWP 26792)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000661bdf in mono_os_cond_wait (mutex=0x9dee40 <lock>, cond=0x9dee00 <work_cond>) at ../../mono/utils/mono-os-mutex.h:146 #2 thread_func (thread_data=0x2b0ff2bcb008) at sgen-thread-pool.c:129 #3 0x00002b0ff347a182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00002b0ff39a130d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 2 (Thread 0x2b0ff633e700 (LWP 26793)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005caff3 in mono_os_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-os-mutex.h:146 #2 mono_coop_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-coop-mutex.h:87 #3 mono_threadpool_io_remove_socket (fd=134239440) at threadpool-io.c:628 #4 0x00000000005ae4f4 in ves_icall_System_Net_Sockets_Socket_Close_internal (sock=42, werror=<optimized out>) at w32socket.c:708 mono#5 0x0000000041e8101f in ?? () mono#6 0x00002b1014abe070 in ?? () mono#7 0x0000000001615860 in ?? () mono#8 0x0000000001615860 in ?? () mono#9 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b0ff2b67c40 (LWP 26780)): #0 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004ac5e6 in mono_handle_native_crash (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2557 #2 <signal handler called> #3 0x00002b0ff38dcf79 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00002b0ff38e0388 in abort () from /lib/x86_64-linux-gnu/libc.so.6 mono#5 0x000000000066b459 in mono_log_write_logfile (log_domain=<optimized out>, level=<optimized out>, hdr=<optimized out>, message=<optimized out>) at mono-log-common.c:137 mono#6 0x00000000006805e0 in monoeg_g_logv (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7ffc7324ea78) at goutput.c:115 mono#7 0x0000000000680736 in monoeg_assertion_message (format=<optimized out>) at goutput.c:135 mono#8 0x00000000006759b8 in mono_thread_info_setup_async_call (info=info@entry=0x2b0ff80008c0, target_func=target_func@entry=0x5bac20 <suspend_for_shutdown_async_call>, user_data=user_data@entry=0x0) at mono-threads.c:1009 mono#9 0x00000000005bad50 in suspend_for_shutdown_critical (info=info@entry=0x2b0ff80008c0, unused=unused@entry=0x0) at threads.c:5019 mono#10 0x00000000006763ae in mono_thread_info_safe_suspend_and_run (id=47347555100416, interrupt_kernel=interrupt_kernel@entry=0, callback=callback@entry=0x5bad40 <suspend_for_shutdown_critical>, user_data=user_data@entry=0x0) at mono-threads.c:979 mono#11 0x00000000005c2e71 in mono_thread_internal_suspend_for_shutdown (thread=<optimized out>) at threads.c:5028 mono#12 0x00000000005e9100 in mono_gc_cleanup () at gc.c:1015 mono#13 0x00000000005e108e in mono_runtime_cleanup (domain=domain@entry=0x1615860) at appdomain.c:423 mono#14 0x00000000004228cb in mini_cleanup (domain=0x1615860) at mini-runtime.c:4111 mono#15 0x000000000047b99f in mono_main (argc=10, argv=<optimized out>) at driver.c:2167 mono#16 0x00000000004200db in mono_main_with_options (argv=0x7ffc7324ef68, argc=10) at main.c:45 mono#17 main (argc=10, argv=0x7ffc7324ef68) at main.c:338 ================================================================= Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= ```
luhenry
pushed a commit
that referenced
this pull request
Mar 22, 2017
``` (lldb) monobt * thread #1 * frame #0: 0x00000001001b236e mono-sgen`interp_transform_call(td=0x00007fff5fbfd080, method=0x0000000100915a90, target_method=0x0000000000000000, domain=0x000000010090b741 frame #1: 0x00000001001a1c2e mono-sgen`generate(method=0x0000000100915a90, rtm=0x000000010382ac70, is_bb_start="\x01", generic_context=0x0000000100915ad0) + 9454 at tran8 transforming TestMonoAsyncGenerics::AsyncWithAwait || frame #2: 0x000000010019f553 mono-sgen`mono_interp_transform_method(runtime_method=0x000000010382ac70, context=0x004 TestMonoAsyncGenerics::AsyncWithAwait @ 0 || frame #3: 0x000000010018a178 mono-sgen`ves_exec_method_with_context(frame=0x00007fff5fbfe290, context=0x00007fff5fbfe3a8) +9 TestMonoAsyncGenerics::Main @ 12 "pop" || frame #4: 0x000000010018b4b1 mono-sgen`ves_exec_method_with_context(frame=0x00007fff5fbfe420, context=0x00007fff5fbfe3a8) + 5081 frame mono#5: 0x0000000100189e43 mono-sgen`mono_interp_runtime_invoke(method=0x000000010090ce38, obj=0x0000000000000000, params=0x00007fff5fbfea40, exc=0x0000000000000000, e0 frame mono#6: 0x00000001000164a2 mono-sgen`mono_jit_runtime_invoke(method=0x000000010090ce38, obj=0x0000000000000000, params=0x00007fff5fbfea40, exc=0x0000000000000000, erro1 frame mono#7: 0x000000010038b2b5 mono-sgen`do_runtime_invoke(method=0x000000010090ce38, obj=0x0000000000000000, params=0x00007fff5fbfea40, exc=0x0000000000000000, error=0x002 frame mono#8: 0x0000000100384e97 mono-sgen`mono_runtime_invoke_checked(method=0x000000010090ce38, obj=0x0000000000000000, params=0x00007fff5fbfea40, error=0x00007fff5fbfeb000 frame mono#9: 0x000000010038f335 mono-sgen`do_exec_main_checked(method=0x000000010090ce38, args=0x00000001020003c8, error=0x00007fff5fbfeb00) + 197 at object.c:4672 frame mono#10: 0x000000010038dd5c mono-sgen`mono_runtime_exec_main_checked(method=0x000000010090ce38, args=0x00000001020003c8, error=0x00007fff5fbfeb00) + 76 at object.c:4773 frame mono#11: 0x000000010038ddbf mono-sgen`mono_runtime_run_main_checked(method=0x000000010090ce38, argc=1, argv=0x00007fff5fbfef68, error=0x00007fff5fbfeb00) + 79 at objec2 frame mono#12: 0x00000001000d9a33 mono-sgen`mono_jit_exec(domain=0x000000010090b740, assembly=0x0000000100913610, argc=1, argv=0x00007fff5fbfef68) + 403 at driver.c:1029 frame mono#13: 0x00000001000dd9da mono-sgen`main_thread_handler(user_data=0x00007fff5fbfeea0) + 538 at driver.c:1098 frame mono#14: 0x00000001000dc21c mono-sgen`mono_main(argc=3, argv=0x00007fff5fbfef58) + 8636 at driver.c:2163 frame mono#15: 0x0000000100001b9e mono-sgen`mono_main_with_options(argc=3, argv=0x00007fff5fbfef58) + 46 at main.c:45 frame mono#16: 0x00000001000012dd mono-sgen`main(argc=3, argv=0x00007fff5fbfef58) + 77 at main.c:338 frame mono#17: 0x00007fffc2e66255 libdyld.dylib`start + 1 frame mono#18: 0x00007fffc2e66255 libdyld.dylib`start + 1 ```
luhenry
added a commit
that referenced
this pull request
Mar 23, 2017
We would observe crashes like the following: ``` * Assertion at mono-threads.c:1009, condition `mono_thread_info_run_state (info) == STATE_ASYNC_SUSPENDED' not met Debug info from gdb: [New LWP 26793] [New LWP 26792] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Id Target Id Frame 3 Thread 0x2b0ff3e6c700 (LWP 26792) "SGen worker" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 2 Thread 0x2b0ff633e700 (LWP 26793) "Finalizer" 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 * 1 Thread 0x2b0ff2b67c40 (LWP 26780) "mono" 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 Thread 3 (Thread 0x2b0ff3e6c700 (LWP 26792)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000661bdf in mono_os_cond_wait (mutex=0x9dee40 <lock>, cond=0x9dee00 <work_cond>) at ../../mono/utils/mono-os-mutex.h:146 #2 thread_func (thread_data=0x2b0ff2bcb008) at sgen-thread-pool.c:129 #3 0x00002b0ff347a182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00002b0ff39a130d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 2 (Thread 0x2b0ff633e700 (LWP 26793)): #0 0x00002b0ff347e414 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000005caff3 in mono_os_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-os-mutex.h:146 #2 mono_coop_cond_wait (mutex=<optimized out>, cond=<optimized out>) at ../../mono/utils/mono-coop-mutex.h:87 #3 mono_threadpool_io_remove_socket (fd=134239440) at threadpool-io.c:628 #4 0x00000000005ae4f4 in ves_icall_System_Net_Sockets_Socket_Close_internal (sock=42, werror=<optimized out>) at w32socket.c:708 mono#5 0x0000000041e8101f in ?? () mono#6 0x00002b1014abe070 in ?? () mono#7 0x0000000001615860 in ?? () mono#8 0x0000000001615860 in ?? () mono#9 0x0000000000000000 in ?? () Thread 1 (Thread 0x2b0ff2b67c40 (LWP 26780)): #0 0x00002b0ff3481ee9 in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000004ac5e6 in mono_handle_native_crash (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2557 #2 <signal handler called> #3 0x00002b0ff38dcf79 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00002b0ff38e0388 in abort () from /lib/x86_64-linux-gnu/libc.so.6 mono#5 0x000000000066b459 in mono_log_write_logfile (log_domain=<optimized out>, level=<optimized out>, hdr=<optimized out>, message=<optimized out>) at mono-log-common.c:137 mono#6 0x00000000006805e0 in monoeg_g_logv (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7ffc7324ea78) at goutput.c:115 mono#7 0x0000000000680736 in monoeg_assertion_message (format=<optimized out>) at goutput.c:135 mono#8 0x00000000006759b8 in mono_thread_info_setup_async_call (info=info@entry=0x2b0ff80008c0, target_func=target_func@entry=0x5bac20 <suspend_for_shutdown_async_call>, user_data=user_data@entry=0x0) at mono-threads.c:1009 mono#9 0x00000000005bad50 in suspend_for_shutdown_critical (info=info@entry=0x2b0ff80008c0, unused=unused@entry=0x0) at threads.c:5019 mono#10 0x00000000006763ae in mono_thread_info_safe_suspend_and_run (id=47347555100416, interrupt_kernel=interrupt_kernel@entry=0, callback=callback@entry=0x5bad40 <suspend_for_shutdown_critical>, user_data=user_data@entry=0x0) at mono-threads.c:979 mono#11 0x00000000005c2e71 in mono_thread_internal_suspend_for_shutdown (thread=<optimized out>) at threads.c:5028 mono#12 0x00000000005e9100 in mono_gc_cleanup () at gc.c:1015 mono#13 0x00000000005e108e in mono_runtime_cleanup (domain=domain@entry=0x1615860) at appdomain.c:423 mono#14 0x00000000004228cb in mini_cleanup (domain=0x1615860) at mini-runtime.c:4111 mono#15 0x000000000047b99f in mono_main (argc=10, argv=<optimized out>) at driver.c:2167 mono#16 0x00000000004200db in mono_main_with_options (argv=0x7ffc7324ef68, argc=10) at main.c:45 mono#17 main (argc=10, argv=0x7ffc7324ef68) at main.c:338 ================================================================= Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= ```
luhenry
pushed a commit
that referenced
this pull request
Jun 5, 2017
Reverts 5e23a77 It seems to cause a crash during msbuild tests: ``` xunit -> Microsoft.Build.Engine.UnitTests... Stacktrace: Native stacktrace: 0 mono 0x0017cad4 mono_handle_native_crash + 308 1 mono 0x001e2b03 sigabrt_signal_handler + 147 2 libsystem_platform.dylib 0x920a8deb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 libsystem_c.dylib 0x940bb27c abort + 155 5 mono 0x0036886d mono_log_write_logfile + 381 6 mono 0x003630d2 structured_log_adapter + 50 7 mono 0x00380d1b monoeg_assertion_message + 107 8 mono 0x0033863d major_scan_object_with_evacuation + 3373 9 mono 0x0033b3d0 drain_gray_stack + 6608 10 mono 0x0032df57 finish_gray_stack + 151 11 mono 0x0032d3c6 major_finish_collection + 118 12 mono 0x00329909 major_do_collection + 169 13 mono 0x003288bd sgen_perform_collection + 605 14 mono 0x0032a90b sgen_gc_collect + 75 15 mono 0x002d5c4d unload_thread_main + 861 16 mono 0x002abb6b start_wrapper + 795 17 libsystem_pthread.dylib 0x9ae6d5fb _pthread_body + 144 18 libsystem_pthread.dylib 0x9ae6d485 _pthread_struct_init + 0 19 libsystem_pthread.dylib 0x9ae72cf2 thread_start + 34 Debug info from gdb: (lldb) command source -s 0 '/tmp/mono-gdb-commands.2tso9f' Executing commands in '/tmp/mono-gdb-commands.2tso9f'. (lldb) process attach --pid 2037 2017-06-02 13:07:23.000 lldb[2093:282f] Metadata.framework [Error]: couldn't get the client port Process 2037 stopped * thread #1, name = 'tid_507', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 libsystem_kernel.dylib`__psynch_cvwait: -> 0x923cc7ca <+10>: jae 0x923cc7da ; <+26> 0x923cc7cc <+12>: calll 0x923cc7d1 ; <+17> 0x923cc7d1 <+17>: popl %edx 0x923cc7d2 <+18>: movl 0xe0b084f(%edx), %edx Executable module set to "/Users/builder/data/lanes/2716/mono-mac-sdk/external/bockbuild/stage/bin/mono". Architecture set to: i386-apple-macosx. (lldb) thread list Process 2037 stopped * thread #1: tid = 0x530e96a, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'tid_507', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP thread #2: tid = 0x530e96b, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker' thread #3: tid = 0x530e96c, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker' thread #4: tid = 0x530e96d, 0x923c7fb6 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'Finalizer' thread mono#5: tid = 0x530e96e, 0x923cd992 libsystem_kernel.dylib`kevent64 + 10, queue = 'com.apple.libdispatch-manager' thread mono#6: tid = 0x530ea26, 0x923ccace libsystem_kernel.dylib`__select + 10, name = 'tid_5a03' thread mono#7: tid = 0x530edf1, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'tid_4007' thread mono#8: tid = 0x530edf2, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'Threadpool worker' thread mono#9: tid = 0x530edf3, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'Threadpool worker' thread mono#10: tid = 0x530edf4, 0x923ccff2 libsystem_kernel.dylib`__wait4 + 10, name = 'Domain unloader' (lldb) thread backtrace all * thread #1, name = 'tid_507', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP * frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728 frame #2: 0x9ae71bd9 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 71 frame #3: 0x00361ea9 mono`mono_os_event_wait_multiple + 505 frame #4: 0x00361ca5 mono`mono_os_event_wait_one + 53 frame mono#5: 0x00376b49 mono`mono_thread_info_wait_one_handle + 41 frame mono#6: 0x002d51b5 mono`mono_domain_try_unload + 485 frame mono#7: 0x002d4f6a mono`ves_icall_System_AppDomain_InternalUnload + 90 frame mono#8: 0x05bf9ee0 frame mono#9: 0x018ce01d mscorlib.dll.dylib`System_AppDomain_Unload_System_AppDomain + 45 frame mono#10: 0x05bf9d60 frame mono#11: 0x05bf9ccc frame mono#12: 0x05bf9d04 frame mono#13: 0x05bf9c90 frame mono#14: 0x05bf996d frame mono#15: 0x02e7d171 frame mono#16: 0x005c26d4 frame mono#17: 0x005b6878 frame mono#18: 0x005b6b7a frame mono#19: 0x000c38a8 mono`mono_jit_runtime_invoke + 1592 frame mono#20: 0x002e43fe mono`do_runtime_invoke + 94 frame mono#21: 0x002e7de3 mono`do_exec_main_checked + 147 frame mono#22: 0x002e69a5 mono`mono_runtime_run_main_checked + 69 frame mono#23: 0x0013b687 mono`mono_jit_exec + 311 frame mono#24: 0x0013e2b2 mono`mono_main + 10114 frame mono#25: 0x000b22db mono`main + 2011 frame mono#26: 0x000b1af5 mono`start + 53 thread #2, name = 'SGen worker' frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728 frame #2: 0x9ae71bd9 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 71 frame #3: 0x0035f0e9 mono`thread_func + 249 frame #4: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame mono#5: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#6: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread #3, name = 'SGen worker' frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728 frame #2: 0x9ae71bd9 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 71 frame #3: 0x0035f0e9 mono`thread_func + 249 frame #4: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame mono#5: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#6: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread #4, name = 'Finalizer' frame #0: 0x923c7fb6 libsystem_kernel.dylib`semaphore_wait_trap + 10 frame #1: 0x002dbeb6 mono`finalizer_thread + 278 frame #2: 0x002abb6b mono`start_wrapper + 795 frame #3: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame #4: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#5: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread mono#5, queue = 'com.apple.libdispatch-manager' frame #0: 0x923cd992 libsystem_kernel.dylib`kevent64 + 10 frame #1: 0x91b5c899 libdispatch.dylib`_dispatch_mgr_invoke + 238 frame #2: 0x91b5c532 libdispatch.dylib`_dispatch_mgr_thread + 52 thread mono#6, name = 'tid_5a03' frame #0: 0x923ccace libsystem_kernel.dylib`__select + 10 frame #1: 0x0036df29 mono`mono_poll + 409 frame #2: 0x002b482f mono`poll_event_wait + 111 frame #3: 0x002b345f mono`selector_thread + 1439 frame #4: 0x002abb6b mono`start_wrapper + 795 frame mono#5: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame mono#6: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#7: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread mono#7, name = 'tid_4007' frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728 frame #2: 0x9ae71c25 libsystem_pthread.dylib`pthread_cond_timedwait$UNIX2003 + 71 frame #3: 0x003760f3 mono`mono_thread_info_sleep + 979 frame #4: 0x002b1a86 mono`monitor_thread + 262 frame mono#5: 0x002abb6b mono`start_wrapper + 795 frame mono#6: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame mono#7: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#8: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread mono#8, name = 'Threadpool worker' frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728 frame #2: 0x9ae71c25 libsystem_pthread.dylib`pthread_cond_timedwait$UNIX2003 + 71 frame #3: 0x002b12e0 mono`worker_thread + 1024 frame #4: 0x002abb6b mono`start_wrapper + 795 frame mono#5: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame mono#6: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#7: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread mono#9, name = 'Threadpool worker' frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10 frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728 frame #2: 0x9ae71c25 libsystem_pthread.dylib`pthread_cond_timedwait$UNIX2003 + 71 frame #3: 0x002b12e0 mono`worker_thread + 1024 frame #4: 0x002abb6b mono`start_wrapper + 795 frame mono#5: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144 frame mono#6: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130 frame mono#7: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34 thread mono#10, name = 'Domain unloader' frame #0: 0x923ccff2 libsystem_kernel.dylib`__wait4 + 10 frame #1: 0x940d9ea5 libsystem_c.dylib`waitpid$UNIX2003 + 48 frame #2: 0x0017cba7 mono`mono_handle_native_crash + 519 frame #3: 0x001e2b03 mono`sigabrt_signal_handler + 147 frame #4: 0x920a8deb libsystem_platform.dylib`_sigtramp + 43 (lldb) detach Process 2037 detached (lldb) quit ================================================================= Got a SIGABRT while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application. ================================================================= ```
luhenry
pushed a commit
that referenced
this pull request
Sep 8, 2017
commit mono@9a634c1 introduces a regression for https://github.com/mono/coreclr/blob/mono/tests/src/JIT/Methodical/Invoke/25params/25param1c.il ``` Testing method of 25 parameters, all of int data type, tail.call mono-sgen(46435,0x7fff9d34b3c0) malloc: *** error for object 0x101045400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug Process 46435 stopped * thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 frame #0: 0x00007fff945c915f libsystem_malloc.dylib`malloc_error_break libsystem_malloc.dylib`malloc_error_break: -> 0x7fff945c915f <+0>: pushq %rbp 0x7fff945c9160 <+1>: movq %rsp, %rbp 0x7fff945c9163 <+4>: nop 0x7fff945c9164 <+5>: nopl (%rax) (lldb) mbt * thread #1 * frame #0: 0x00007fff945c915f libsystem_malloc.dylib`malloc_error_break frame #1: 0x00007fff945c5e81 libsystem_malloc.dylib`szone_error + 406 frame #2: 0x00007fff945c7925 libsystem_malloc.dylib`small_free_list_remove_ptr_no_clear + 766 frame #3: 0x00007fff945c7cb2 libsystem_malloc.dylib`free_small + 881 frame #4: 0x00000001004d2110 mono-sgen`monoeg_g_free(ptr=0x0000000101083c00) at gmem.c:66 frame mono#5: 0x0000000100007b64 mono-sgen`mono_codegen(cfg=0x000000010108c800) at mini.c:2300 frame mono#6: 0x000000010000aedf mono-sgen`mini_method_compile(method=0x0000000100910310, opts=370239999, domain=0x000000010090ebd0, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at mini.c:3829 frame mono#7: 0x000000010000edd0 mono-sgen`mono_jit_compile_method_inner(method=0x0000000100910310, target_domain=0x000000010090ebd0, opt=370239999, error=0x00007fff5fbfe048) at mini.c:4156 frame mono#8: 0x000000010001459e mono-sgen`mono_jit_compile_method_with_opt(method=0x0000000100910310, opt=370239999, jit_only=0, error=0x00007fff5fbfe048) at mini-runtime.c:2127 frame mono#9: 0x0000000100013e6d mono-sgen`mono_jit_compile_method(method=0x0000000100910310, error=0x00007fff5fbfe048) at mini-runtime.c:2173 frame mono#10: 0x0000000100131eea mono-sgen`common_call_trampoline(regs=0x00007fff5fbfe128, code="H\x8bй\x01", m=0x0000000100910310, vt=0x0000000000000000, vtable_slot=0x0000000000000000, error=0x00007fff5fbfe048) at mini-trampolines.c:704 frame mono#11: 0x00000001001313b7 mono-sgen`mono_magic_trampoline(regs=0x00007fff5fbfe128, code="H\x8bй\x01", arg=0x0000000100910310, tramp="����\b\x10\x03\x91") at mini-trampolines.c:835 ``` turns out we smashed our code buffer. Also, transform this comparison: ``` if (G_UNLIKELY (offset > (cfg->code_size - max_len - EXTRA_CODE_SPACE))) { if (G_UNLIKELY ((offset + max_len + EXTRA_CODE_SPACE) > cfg->code_size)) { ``` we deal with unsigned values here, and if `max_len` is bigger then `cfg->code_size`, we won't resize the buffer.
luhenry
pushed a commit
that referenced
this pull request
Mar 30, 2018
clang fails to properly align 64bit members of the `MonoPerfCounters` struct on 32bit builds: ``` * thread #1, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_ARM_DA_ALIGN, address=0x16933124) frame #0: 0x0066f4b2 mini`mono_atomic_cas_i64(dest=0x16933124, exch=<unavailable>, comp=<unavailable>) at atomic.c:519 [opt] 516 gint64 517 mono_atomic_cas_i64(volatile gint64 *dest, gint64 exch, gint64 comp) 518 { -> 519 return __sync_val_compare_and_swap (dest, comp, exch); 520 } 521 522 #elif defined (TARGET_ANDROID) (lldb) up frame #1: 0x006292ea mini`ves_icall_System_Threading_ThreadPool_NotifyWorkItemQueued [inlined] mono_atomic_inc_i64 at atomic.h:397 [opt] 394 do { 395 get = *val; 396 set = get + 1; -> 397 } while (mono_atomic_cas_i64 (val, set, get) != get); 398 return set; 399 } 400 (lldb) up frame #2: 0x006292d6 mini`ves_icall_System_Threading_ThreadPool_NotifyWorkItemQueued at threadpool.c:743 [opt] 740 ves_icall_System_Threading_ThreadPool_NotifyWorkItemQueued (void) 741 { 742 #ifndef DISABLE_PERFCOUNTERS -> 743 mono_atomic_inc_i64 (&mono_perfcounters->threadpool_workitems); 744 #endif 745 } 746 ``` Note that the member `threadpool_workitems` is on an unnatural alignment (`0x16933124`) regarding to its type `gint64`. related xamarin/xamarin-macios#3826
luhenry
pushed a commit
that referenced
this pull request
Apr 3, 2018
clang fails to properly align 64bit members of the `MonoPerfCounters` struct on 32bit builds: ``` * thread #1, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_ARM_DA_ALIGN, address=0x16933124) frame #0: 0x0066f4b2 mini`mono_atomic_cas_i64(dest=0x16933124, exch=<unavailable>, comp=<unavailable>) at atomic.c:519 [opt] 516 gint64 517 mono_atomic_cas_i64(volatile gint64 *dest, gint64 exch, gint64 comp) 518 { -> 519 return __sync_val_compare_and_swap (dest, comp, exch); 520 } 521 522 #elif defined (TARGET_ANDROID) (lldb) up frame #1: 0x006292ea mini`ves_icall_System_Threading_ThreadPool_NotifyWorkItemQueued [inlined] mono_atomic_inc_i64 at atomic.h:397 [opt] 394 do { 395 get = *val; 396 set = get + 1; -> 397 } while (mono_atomic_cas_i64 (val, set, get) != get); 398 return set; 399 } 400 (lldb) up frame #2: 0x006292d6 mini`ves_icall_System_Threading_ThreadPool_NotifyWorkItemQueued at threadpool.c:743 [opt] 740 ves_icall_System_Threading_ThreadPool_NotifyWorkItemQueued (void) 741 { 742 #ifndef DISABLE_PERFCOUNTERS -> 743 mono_atomic_inc_i64 (&mono_perfcounters->threadpool_workitems); 744 #endif 745 } 746 ``` Note that the member `threadpool_workitems` is on an unnatural alignment (`0x16933124`) regarding to its type `gint64`. related xamarin/xamarin-macios#3826
luhenry
pushed a commit
that referenced
this pull request
May 10, 2018
* [btls] Do not give gcc flags to msvc. * [btls] #include "btls-foo.h" instead of #include <btls-foo.h> to work with existing build system, on Windows, with out-of-tree build. Not sure which factor does not like <> but "" should be ok. * [blts] MONO_API is not used consistently and therefore errors on Windows. 1. Make it be nothing. 1b. Use a .def file if there are actual exports. Really, .def files are a good idea. or 2. Use it consistently. This does #1.
luhenry
pushed a commit
that referenced
this pull request
Nov 5, 2018
We were previously only setting it from the icall. The icall was therefore fine, while invocations associated with actual dumps caused crashes like this: ``` thread mono#10, name = 'Thread Pool Worker' frame #0: 0x00007fff978dac22 libsystem_kernel.dylib`__psynch_mutexwait + 10 frame #1: 0x00007fff979c5dfa libsystem_pthread.dylib`_pthread_mutex_lock_wait + 100 frame #2: 0x00007fff979c3519 libsystem_pthread.dylib`_pthread_mutex_lock_slow + 285 frame #3: 0x0000000108e0d4a9 mono`mono_loader_lock + 73 frame #4: 0x0000000108dc9106 mono`mono_class_create_from_typedef + 182 frame mono#5: 0x0000000108dc1f2c mono`mono_class_get_checked + 92 frame mono#6: 0x0000000108e21da2 mono`mono_metadata_parse_type_internal + 1378 frame mono#7: 0x0000000108e25e11 mono`mono_metadata_parse_mh_full + 1233 frame mono#8: 0x0000000108ca4f79 mono`mono_debug_add_aot_method + 121 frame mono#9: 0x0000000108cc6b1e mono`decode_exception_debug_info + 5918 frame mono#10: 0x0000000108cc5047 mono`mono_aot_find_jit_info + 1367 frame mono#11: 0x0000000108e082cc mono`mono_jit_info_table_find_internal + 204 frame mono#12: 0x0000000108cdb035 mono`mini_jit_info_table_find_ext + 69 frame mono#13: 0x0000000108cdad5c mono`mono_find_jit_info_ext + 124 frame mono#14: 0x0000000108cdc3a5 mono`unwinder_unwind_frame + 229 frame mono#15: 0x0000000108cdbade mono`mono_walk_stack_full + 798 frame mono#16: 0x0000000108cda1a4 mono`mono_summarize_managed_stack + 100 frame mono#17: 0x0000000108e6dc03 mono`mono_threads_summarize_execute + 1347 frame mono#18: 0x0000000108e6df88 mono`mono_threads_summarize + 360 ```
luhenry
pushed a commit
that referenced
this pull request
Nov 5, 2018
We were previously only setting it from the icall. The icall was therefore fine, while invocations associated with actual dumps caused crashes like this: ``` thread mono#10, name = 'Thread Pool Worker' frame #0: 0x00007fff978dac22 libsystem_kernel.dylib`__psynch_mutexwait + 10 frame #1: 0x00007fff979c5dfa libsystem_pthread.dylib`_pthread_mutex_lock_wait + 100 frame #2: 0x00007fff979c3519 libsystem_pthread.dylib`_pthread_mutex_lock_slow + 285 frame #3: 0x0000000108e0d4a9 mono`mono_loader_lock + 73 frame #4: 0x0000000108dc9106 mono`mono_class_create_from_typedef + 182 frame mono#5: 0x0000000108dc1f2c mono`mono_class_get_checked + 92 frame mono#6: 0x0000000108e21da2 mono`mono_metadata_parse_type_internal + 1378 frame mono#7: 0x0000000108e25e11 mono`mono_metadata_parse_mh_full + 1233 frame mono#8: 0x0000000108ca4f79 mono`mono_debug_add_aot_method + 121 frame mono#9: 0x0000000108cc6b1e mono`decode_exception_debug_info + 5918 frame mono#10: 0x0000000108cc5047 mono`mono_aot_find_jit_info + 1367 frame mono#11: 0x0000000108e082cc mono`mono_jit_info_table_find_internal + 204 frame mono#12: 0x0000000108cdb035 mono`mini_jit_info_table_find_ext + 69 frame mono#13: 0x0000000108cdad5c mono`mono_find_jit_info_ext + 124 frame mono#14: 0x0000000108cdc3a5 mono`unwinder_unwind_frame + 229 frame mono#15: 0x0000000108cdbade mono`mono_walk_stack_full + 798 frame mono#16: 0x0000000108cda1a4 mono`mono_summarize_managed_stack + 100 frame mono#17: 0x0000000108e6dc03 mono`mono_threads_summarize_execute + 1347 frame mono#18: 0x0000000108e6df88 mono`mono_threads_summarize + 360 ```
luhenry
pushed a commit
that referenced
this pull request
Feb 12, 2019
* [runtime] Make infrastructure for merp tests * [runtime] Fix dumping when crash happens without sigctx * [runtime] Disable crashy (886/1000 runs) stacktrace walker * [crash] Remove usage of allocating build/os info functions * [crash] Remove often-crashing g_free on native crash path * [crash] Add crash_reporter checked build We add a checked build mode that asserts when mono mallocs inside of the crash reporter. It makes risky allocations into assertions. It's useful for automated testing because the double-abort often represents itself as an indefinite hang. If it happens before the thread dumping supervisor process is started, or after it ends, the crash reporter hangs. * [crash] Remove reliance on nested SIGABRT/double-fault (broken on OSX) * [crash] Fix top-level handling of double faults/assertions * [runtime] Make fatal unwinding errors return into handled error paths * [crash] Change dumper logging for better info * [runtime] Fix handling of segfault on sgen thread Threads without domains that get segfaults will end up in this handler. It's not safe to call this function with a NULL domain. See crash below: ``` * thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8) * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414 frame #1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508 frame #2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026 frame #3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147 frame #4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227 frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574 frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612 frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26 frame mono#8: 0x0000000110bb81c1 frame mono#9: 0x000000011085ffe1 frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215 frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977 frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145 frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042 frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138 frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599 frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298 frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257 frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375 frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551 frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50 frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406 frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1 frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1 thread #2, name = 'SGen worker' frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208 frame #1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304 frame #2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540 frame #3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612 frame #4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26 frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11 frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732 frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168 frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165 frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196 frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340 frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377 frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13 thread #3, name = 'Finalizer' frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10 frame #1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84 frame #2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41 frame #3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920 frame #4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178 frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238 frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340 frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377 frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13 thread #4 frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10 frame #1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035 frame #2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13 (lldb) ``` * [crash] Add signal-safe mmap/file "allocator" * [crash] Remove use of static memory from dumper * [runtime] Reduce print buffer size for lockless printer. Each frame that prints ends up increased by the size of buff. In practice, clang often fails to deduplicate some of these buffers, leading to 30k-big stackframes. It was noticed by a series of hard-to-diagnose segfaults on stacks that looked otherwise fine during the crash reporting stress test. This change fixes this, making stacks a 1/10th of the size. It doesn't seem to break the crash reporter messages anywhere (may need to shrink other "max name length" fields), and it's not mission-critical anywhere else. * [crash] Use async-safe file memory for dumper internals * [crash] Add memory barriers around merp configuration * [crash] Use signal-safe printers on all native crash paths * [crash] Move gdb/lldb lookup to startup * [runtime] Move MOSTLY_ASYNC_SAFE_FPRINTF to eglib * [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_PRINTF * [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_FPRINTF * [crash] Switch to signal-safe exit function * [crash] Make dumper enum in managed * [runtime] Add more information to managed frame * [crash] Make async_safe printers inlined * [runtime] Move basic pe_file functionality into proclib * [crash] Fix handling of thread attributes
luhenry
pushed a commit
that referenced
this pull request
Mar 6, 2019
We were previously only setting it from the icall. The icall was therefore fine, while invocations associated with actual dumps caused crashes like this: ``` thread mono#10, name = 'Thread Pool Worker' frame #0: 0x00007fff978dac22 libsystem_kernel.dylib`__psynch_mutexwait + 10 frame #1: 0x00007fff979c5dfa libsystem_pthread.dylib`_pthread_mutex_lock_wait + 100 frame #2: 0x00007fff979c3519 libsystem_pthread.dylib`_pthread_mutex_lock_slow + 285 frame #3: 0x0000000108e0d4a9 mono`mono_loader_lock + 73 frame #4: 0x0000000108dc9106 mono`mono_class_create_from_typedef + 182 frame mono#5: 0x0000000108dc1f2c mono`mono_class_get_checked + 92 frame mono#6: 0x0000000108e21da2 mono`mono_metadata_parse_type_internal + 1378 frame mono#7: 0x0000000108e25e11 mono`mono_metadata_parse_mh_full + 1233 frame mono#8: 0x0000000108ca4f79 mono`mono_debug_add_aot_method + 121 frame mono#9: 0x0000000108cc6b1e mono`decode_exception_debug_info + 5918 frame mono#10: 0x0000000108cc5047 mono`mono_aot_find_jit_info + 1367 frame mono#11: 0x0000000108e082cc mono`mono_jit_info_table_find_internal + 204 frame mono#12: 0x0000000108cdb035 mono`mini_jit_info_table_find_ext + 69 frame mono#13: 0x0000000108cdad5c mono`mono_find_jit_info_ext + 124 frame mono#14: 0x0000000108cdc3a5 mono`unwinder_unwind_frame + 229 frame mono#15: 0x0000000108cdbade mono`mono_walk_stack_full + 798 frame mono#16: 0x0000000108cda1a4 mono`mono_summarize_managed_stack + 100 frame mono#17: 0x0000000108e6dc03 mono`mono_threads_summarize_execute + 1347 frame mono#18: 0x0000000108e6df88 mono`mono_threads_summarize + 360 ```
luhenry
pushed a commit
that referenced
this pull request
Mar 6, 2019
We were previously only setting it from the icall. The icall was therefore fine, while invocations associated with actual dumps caused crashes like this: ``` thread mono#10, name = 'Thread Pool Worker' frame #0: 0x00007fff978dac22 libsystem_kernel.dylib`__psynch_mutexwait + 10 frame #1: 0x00007fff979c5dfa libsystem_pthread.dylib`_pthread_mutex_lock_wait + 100 frame #2: 0x00007fff979c3519 libsystem_pthread.dylib`_pthread_mutex_lock_slow + 285 frame #3: 0x0000000108e0d4a9 mono`mono_loader_lock + 73 frame #4: 0x0000000108dc9106 mono`mono_class_create_from_typedef + 182 frame mono#5: 0x0000000108dc1f2c mono`mono_class_get_checked + 92 frame mono#6: 0x0000000108e21da2 mono`mono_metadata_parse_type_internal + 1378 frame mono#7: 0x0000000108e25e11 mono`mono_metadata_parse_mh_full + 1233 frame mono#8: 0x0000000108ca4f79 mono`mono_debug_add_aot_method + 121 frame mono#9: 0x0000000108cc6b1e mono`decode_exception_debug_info + 5918 frame mono#10: 0x0000000108cc5047 mono`mono_aot_find_jit_info + 1367 frame mono#11: 0x0000000108e082cc mono`mono_jit_info_table_find_internal + 204 frame mono#12: 0x0000000108cdb035 mono`mini_jit_info_table_find_ext + 69 frame mono#13: 0x0000000108cdad5c mono`mono_find_jit_info_ext + 124 frame mono#14: 0x0000000108cdc3a5 mono`unwinder_unwind_frame + 229 frame mono#15: 0x0000000108cdbade mono`mono_walk_stack_full + 798 frame mono#16: 0x0000000108cda1a4 mono`mono_summarize_managed_stack + 100 frame mono#17: 0x0000000108e6dc03 mono`mono_threads_summarize_execute + 1347 frame mono#18: 0x0000000108e6df88 mono`mono_threads_summarize + 360 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR message
THIS IS A TEST! DO NOT MERGE!