Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed System.Resources.ResXResourceWriter losing comments of ResXNodes #21

Closed
wants to merge 2 commits into from

Conversation

@ngeor
Copy link
Contributor

ngeor commented Oct 31, 2010

The ResXResourceWriter.AddResource (ResXNode) method was not writing the comment of the ResXNode to the resx file. This has been fixed with this fork. Also a unit test has been added in mcs/class/Managed.Windows.Forms/Test/System.Resources/WriterTest.cs.

This bug has been reported here by me:
https://bugzilla.novell.com/show_bug.cgi?id=650365

@gonzalop

This comment has been minimized.

Copy link
Contributor

gonzalop commented Oct 31, 2010

Pushed to master 023a11b

luliyi1024 referenced this pull request in luliyi1024/mono Mar 19, 2013
akoeplinger added a commit that referenced this pull request Jun 5, 2017
Reverts 5e23a77

It seems to cause a crash during msbuild tests:

```
  xunit -> Microsoft.Build.Engine.UnitTests...
  Stacktrace:

  Native stacktrace:

  	0   mono                                0x0017cad4 mono_handle_native_crash + 308
  	1   mono                                0x001e2b03 sigabrt_signal_handler + 147
  	2   libsystem_platform.dylib            0x920a8deb _sigtramp + 43
  	3   ???                                 0xffffffff 0x0 + 4294967295
  	4   libsystem_c.dylib                   0x940bb27c abort + 155
  	5   mono                                0x0036886d mono_log_write_logfile + 381
  	6   mono                                0x003630d2 structured_log_adapter + 50
  	7   mono                                0x00380d1b monoeg_assertion_message + 107
  	8   mono                                0x0033863d major_scan_object_with_evacuation + 3373
  	9   mono                                0x0033b3d0 drain_gray_stack + 6608
  	10  mono                                0x0032df57 finish_gray_stack + 151
  	11  mono                                0x0032d3c6 major_finish_collection + 118
  	12  mono                                0x00329909 major_do_collection + 169
  	13  mono                                0x003288bd sgen_perform_collection + 605
  	14  mono                                0x0032a90b sgen_gc_collect + 75
  	15  mono                                0x002d5c4d unload_thread_main + 861
  	16  mono                                0x002abb6b start_wrapper + 795
  	17  libsystem_pthread.dylib             0x9ae6d5fb _pthread_body + 144
  	18  libsystem_pthread.dylib             0x9ae6d485 _pthread_struct_init + 0
  	19  libsystem_pthread.dylib             0x9ae72cf2 thread_start + 34

  Debug info from gdb:

  (lldb) command source -s 0 '/tmp/mono-gdb-commands.2tso9f'
  Executing commands in '/tmp/mono-gdb-commands.2tso9f'.
  (lldb) process attach --pid 2037
  2017-06-02 13:07:23.000 lldb[2093:282f] Metadata.framework [Error]: couldn't get the client port
  Process 2037 stopped
  * thread #1, name = 'tid_507', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
      frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
  libsystem_kernel.dylib`__psynch_cvwait:
  ->  0x923cc7ca <+10>: jae    0x923cc7da                ; <+26>
      0x923cc7cc <+12>: calll  0x923cc7d1                ; <+17>
      0x923cc7d1 <+17>: popl   %edx
      0x923cc7d2 <+18>: movl   0xe0b084f(%edx), %edx

  Executable module set to "/Users/builder/data/lanes/2716/mono-mac-sdk/external/bockbuild/stage/bin/mono".
  Architecture set to: i386-apple-macosx.
  (lldb) thread list
  Process 2037 stopped
  * thread #1: tid = 0x530e96a, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'tid_507', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    thread #2: tid = 0x530e96b, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker'
    thread #3: tid = 0x530e96c, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'SGen worker'
    thread #4: tid = 0x530e96d, 0x923c7fb6 libsystem_kernel.dylib`semaphore_wait_trap + 10, name = 'Finalizer'
    thread #5: tid = 0x530e96e, 0x923cd992 libsystem_kernel.dylib`kevent64 + 10, queue = 'com.apple.libdispatch-manager'
    thread #6: tid = 0x530ea26, 0x923ccace libsystem_kernel.dylib`__select + 10, name = 'tid_5a03'
    thread #7: tid = 0x530edf1, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'tid_4007'
    thread #8: tid = 0x530edf2, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'Threadpool worker'
    thread #9: tid = 0x530edf3, 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'Threadpool worker'
    thread #10: tid = 0x530edf4, 0x923ccff2 libsystem_kernel.dylib`__wait4 + 10, name = 'Domain unloader'
  (lldb) thread backtrace all
  * thread #1, name = 'tid_507', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    * frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728
      frame #2: 0x9ae71bd9 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 71
      frame #3: 0x00361ea9 mono`mono_os_event_wait_multiple + 505
      frame #4: 0x00361ca5 mono`mono_os_event_wait_one + 53
      frame #5: 0x00376b49 mono`mono_thread_info_wait_one_handle + 41
      frame #6: 0x002d51b5 mono`mono_domain_try_unload + 485
      frame #7: 0x002d4f6a mono`ves_icall_System_AppDomain_InternalUnload + 90
      frame #8: 0x05bf9ee0
      frame #9: 0x018ce01d mscorlib.dll.dylib`System_AppDomain_Unload_System_AppDomain + 45
      frame #10: 0x05bf9d60
      frame #11: 0x05bf9ccc
      frame #12: 0x05bf9d04
      frame #13: 0x05bf9c90
      frame #14: 0x05bf996d
      frame #15: 0x02e7d171
      frame #16: 0x005c26d4
      frame #17: 0x005b6878
      frame #18: 0x005b6b7a
      frame #19: 0x000c38a8 mono`mono_jit_runtime_invoke + 1592
      frame #20: 0x002e43fe mono`do_runtime_invoke + 94
      frame #21: 0x002e7de3 mono`do_exec_main_checked + 147
      frame #22: 0x002e69a5 mono`mono_runtime_run_main_checked + 69
      frame #23: 0x0013b687 mono`mono_jit_exec + 311
      frame #24: 0x0013e2b2 mono`mono_main + 10114
      frame #25: 0x000b22db mono`main + 2011
      frame #26: 0x000b1af5 mono`start + 53
    thread #2, name = 'SGen worker'
      frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728
      frame #2: 0x9ae71bd9 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 71
      frame #3: 0x0035f0e9 mono`thread_func + 249
      frame #4: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #5: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #6: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #3, name = 'SGen worker'
      frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728
      frame #2: 0x9ae71bd9 libsystem_pthread.dylib`pthread_cond_wait$UNIX2003 + 71
      frame #3: 0x0035f0e9 mono`thread_func + 249
      frame #4: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #5: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #6: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #4, name = 'Finalizer'
      frame #0: 0x923c7fb6 libsystem_kernel.dylib`semaphore_wait_trap + 10
      frame #1: 0x002dbeb6 mono`finalizer_thread + 278
      frame #2: 0x002abb6b mono`start_wrapper + 795
      frame #3: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #4: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #5: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #5, queue = 'com.apple.libdispatch-manager'
      frame #0: 0x923cd992 libsystem_kernel.dylib`kevent64 + 10
      frame #1: 0x91b5c899 libdispatch.dylib`_dispatch_mgr_invoke + 238
      frame #2: 0x91b5c532 libdispatch.dylib`_dispatch_mgr_thread + 52
    thread #6, name = 'tid_5a03'
      frame #0: 0x923ccace libsystem_kernel.dylib`__select + 10
      frame #1: 0x0036df29 mono`mono_poll + 409
      frame #2: 0x002b482f mono`poll_event_wait + 111
      frame #3: 0x002b345f mono`selector_thread + 1439
      frame #4: 0x002abb6b mono`start_wrapper + 795
      frame #5: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #6: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #7: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #7, name = 'tid_4007'
      frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728
      frame #2: 0x9ae71c25 libsystem_pthread.dylib`pthread_cond_timedwait$UNIX2003 + 71
      frame #3: 0x003760f3 mono`mono_thread_info_sleep + 979
      frame #4: 0x002b1a86 mono`monitor_thread + 262
      frame #5: 0x002abb6b mono`start_wrapper + 795
      frame #6: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #7: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #8: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #8, name = 'Threadpool worker'
      frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728
      frame #2: 0x9ae71c25 libsystem_pthread.dylib`pthread_cond_timedwait$UNIX2003 + 71
      frame #3: 0x002b12e0 mono`worker_thread + 1024
      frame #4: 0x002abb6b mono`start_wrapper + 795
      frame #5: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #6: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #7: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #9, name = 'Threadpool worker'
      frame #0: 0x923cc7ca libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x9ae6fd1d libsystem_pthread.dylib`_pthread_cond_wait + 728
      frame #2: 0x9ae71c25 libsystem_pthread.dylib`pthread_cond_timedwait$UNIX2003 + 71
      frame #3: 0x002b12e0 mono`worker_thread + 1024
      frame #4: 0x002abb6b mono`start_wrapper + 795
      frame #5: 0x9ae6d5fb libsystem_pthread.dylib`_pthread_body + 144
      frame #6: 0x9ae6d485 libsystem_pthread.dylib`_pthread_start + 130
      frame #7: 0x9ae72cf2 libsystem_pthread.dylib`thread_start + 34
    thread #10, name = 'Domain unloader'
      frame #0: 0x923ccff2 libsystem_kernel.dylib`__wait4 + 10
      frame #1: 0x940d9ea5 libsystem_c.dylib`waitpid$UNIX2003 + 48
      frame #2: 0x0017cba7 mono`mono_handle_native_crash + 519
      frame #3: 0x001e2b03 mono`sigabrt_signal_handler + 147
      frame #4: 0x920a8deb libsystem_platform.dylib`_sigtramp + 43
  (lldb) detach
  Process 2037 detached
  (lldb) quit

  =================================================================
  Got a SIGABRT while executing native code. This usually indicates
  a fatal error in the mono runtime or one of the native libraries
  used by your application.
  =================================================================
```
@Rombersoft Rombersoft mentioned this pull request May 8, 2018
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 4, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 4, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 7, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 8, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 8, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 8, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 8, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 10, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 10, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 10, 2019
The MERP stress test often catches a crash like so:

```
 * frame #0: 0x0000000103409827 mono-sgen`mono_get_hazardous_pointer(pp=0x00007fc559a1a1f8, hp=0x00000001037e9600, hazard_index=1) at hazard-pointer.c:208
    frame mono#1: 0x000000010320eb8e mono-sgen`jit_info_table_chunk_index(chunk=0x00007fc4186060a0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:200
    frame mono#2: 0x000000010320d7fa mono-sgen`jit_info_table_find(table=0x00007fc4186057c0, hp=0x00000001037e9600, addr=0x00007fff73bffb66) at jit-info.c:222
    frame mono#3: 0x000000010320d5e6 mono-sgen`mono_jit_info_table_find_internal(domain=0x00007fc418512bc0, addr=0x00007fff73bffb66, try_aot=1, allow_trampolines=1) at jit-info.c:293
    frame mono#4: 0x00000001030d67af mono-sgen`sigabrt_signal_handler(_dummy=6, _info=0x00007ffeecd4ea28, context=0x00007ffeecd4ea90) at mini-posix.c:219
    frame mono#5: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#6: 0x00007fff73bffb67 libsystem_kernel.dylib`__pthread_kill + 11
    frame mono#7: 0x00007fff73dca080 libsystem_pthread.dylib`pthread_kill + 333
    frame mono#8: 0x00007fff73b5b1ae libsystem_c.dylib`abort + 127
    frame mono#9: 0x00007fff73c59822 libsystem_malloc.dylib`free + 521
    frame mono#10: 0x0000000103c7d9d4 libtest.0.dylib`monoeg_g_free(ptr=0x00007ffeecd4ec34) at gmem.c:86
    frame mono#11: 0x0000000103c7733b libtest.0.dylib`mono_test_MerpCrashMalloc at libtest.c:7710
    frame mono#12: 0x0000000103c5c288
    frame mono#13: 0x000000010385dfd1
    frame mono#14: 0x0000000102ec85c3 mono-sgen`mono_jit_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x00007ffeecd4ef08, error=0x00007ffeecd4f250) at mini-runtime.c:3215
    frame mono#15: 0x000000010326f39d mono-sgen`do_runtime_invoke(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, exc=0x0000000000000000, error=0x00007ffeecd4f250) at object.c:2977
    frame mono#16: 0x0000000103267c61 mono-sgen`mono_runtime_invoke_checked(method=0x00007fc418514288, obj=0x0000000000000000, params=0x00007ffeecd4f180, error=0x00007ffeecd4f250) at object.c:3145
    frame mono#17: 0x0000000103274d58 mono-sgen`do_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5042
    frame mono#18: 0x0000000103272b03 mono-sgen`mono_runtime_exec_main_checked(method=0x00007fc418514288, args=0x00000001040003e8, error=0x00007ffeecd4f250) at object.c:5138
    frame mono#19: 0x0000000103272b56 mono-sgen`mono_runtime_run_main_checked(method=0x00007fc418514288, argc=2, argv=0x00007ffeecd4f760, error=0x00007ffeecd4f250) at object.c:4599
    frame mono#20: 0x0000000102f78bff mono-sgen`mono_jit_exec_internal(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1298
    frame mono#21: 0x0000000102f78a2d mono-sgen`mono_jit_exec(domain=0x00007fc418512bc0, assembly=0x00007fc4185445f0, argc=2, argv=0x00007ffeecd4f760) at driver.c:1257
    frame mono#22: 0x0000000102f7d60f mono-sgen`main_thread_handler(user_data=0x00007ffeecd4f6a0) at driver.c:1375
    frame mono#23: 0x0000000102f7b8dd mono-sgen`mono_main(argc=3, argv=0x00007ffeecd4f758) at driver.c:2551
    frame mono#24: 0x0000000102eb1e4e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffeecd4f758) at main.c:50
    frame mono#25: 0x0000000102eb145d mono-sgen`main(argc=3, argv=0x00007ffeecd4f758) at main.c:406
    frame mono#26: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#27: 0x00007fff73aaf015 libdyld.dylib`start + 1
```

because of

```
(lldb) p table->num_chunks
(int) $24 = 6
(lldb) p table->chunks [0]->num_elements
(volatile int) $18 = 50
(lldb) p table->chunks [1]->num_elements
(volatile int) $19 = 48
(lldb) p table->chunks [2]->num_elements
(volatile int) $20 = 48
(lldb) p table->chunks [3]->num_elements
(volatile int) $21 = 32
(lldb) p table->chunks [4]->num_elements
(volatile int) $23 = 32
(lldb) p table->chunks [5]->num_elements
(volatile int) $22 = 1347440720
```
.

It reproduces very often. Adding assertions / memory fences at the end
of the add/remove functions seems to remove this crash.

Looking at the code in the add function, I can picture how a missing
fence right at the end could lead to what we were seeing.

Not 100% happy with this fix, as it's going to have a perf cost. We have
an earlier fence that might be able to be coalesced into this one, but
the logic as-is makes that look unsafe.
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 17, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 17, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 17, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
luhenry added a commit that referenced this pull request Jan 22, 2019
* [runtime] Make infrastructure for merp tests

* [runtime] Fix dumping when crash happens without sigctx

* [runtime] Disable crashy (886/1000 runs) stacktrace walker

* [crash] Remove usage of allocating build/os info functions

* [crash] Remove often-crashing g_free on native crash path

* [crash] Add crash_reporter checked build

We add a checked build mode that asserts when mono mallocs inside of
the crash reporter. It makes risky allocations into assertions. It's
useful for automated testing because the double-abort often represents
itself as an indefinite hang. If it happens before the thread dumping
supervisor process is started, or after it ends, the crash reporter
hangs.

* [crash] Remove reliance on nested SIGABRT/double-fault (broken on OSX)

* [crash] Fix top-level handling of double faults/assertions

* [runtime] Make fatal unwinding errors return into handled error paths

* [crash] Change dumper logging for better info

* [runtime] Fix handling of segfault on sgen thread

Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame #1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame #2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame #3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame #4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame #5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame #6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame #7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame #8: 0x0000000110bb81c1
    frame #9: 0x000000011085ffe1
    frame #10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame #11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame #12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame #13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame #14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame #15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame #16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame #17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame #18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame #19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame #20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame #21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame #22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame #23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread #2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame #1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame #2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame #3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame #4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame #5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame #6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame #8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame #9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame #10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame #11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame #12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread #3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame #2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame #3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame #4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame #5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame #6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread #4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame #2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```

* [crash] Add signal-safe mmap/file "allocator"

* [crash] Remove use of static memory from dumper

* [runtime] Reduce print buffer size for lockless printer.

Each frame that prints ends up increased by the size of buff.
In practice, clang often fails to deduplicate some of these buffers,
leading to 30k-big stackframes.

It was noticed by a series of hard-to-diagnose segfaults on stacks that
looked otherwise fine during the crash reporting stress test.

This change fixes this, making stacks a 1/10th of the size. It doesn't
seem to break the crash reporter messages anywhere (may need to shrink
other "max name length" fields), and it's not mission-critical anywhere
else.

* [crash] Use async-safe file memory for dumper internals

* [crash] Add memory barriers around merp configuration

* [crash] Use signal-safe printers on all native crash paths

* [crash] Move gdb/lldb lookup to startup

* [runtime] Move MOSTLY_ASYNC_SAFE_FPRINTF to eglib

* [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_PRINTF

* [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_FPRINTF

* [crash] Switch to signal-safe exit function

* [crash] Make dumper enum in managed

* [runtime] Add more information to managed frame

* [crash] Make async_safe printers inlined

* [runtime] Move basic pe_file functionality into proclib

* [crash] Fix handling of thread attributes
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 22, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 22, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
alexanderkyte added a commit to alexanderkyte/mono that referenced this pull request Jan 22, 2019
Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread mono#1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame mono#1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame mono#2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame mono#3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame mono#4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame mono#5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame mono#6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame mono#7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#8: 0x0000000110bb81c1
    frame mono#9: 0x000000011085ffe1
    frame mono#10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame mono#11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame mono#12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame mono#13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame mono#14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame mono#15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame mono#16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame mono#17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame mono#18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame mono#19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame mono#20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame mono#21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame mono#22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame mono#23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread mono#2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame mono#1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame mono#2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame mono#3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame mono#4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame mono#5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame mono#6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame mono#7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame mono#8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame mono#9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame mono#10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame mono#1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame mono#2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame mono#3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame mono#4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame mono#5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame mono#6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame mono#7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame mono#8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread mono#4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame mono#1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame mono#2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```
marek-safar added a commit that referenced this pull request Jan 25, 2019
* [crash] Support extra merp params

* [runtime] Make infrastructure for merp tests

* [runtime] Fix dumping when crash happens without sigctx

* [runtime] Disable crashy (886/1000 runs) stacktrace walker

* [crash] Remove usage of allocating build/os info functions

* [crash] Remove often-crashing g_free on native crash path

* [crash] Add crash_reporter checked build

We add a checked build mode that asserts when mono mallocs inside of
the crash reporter. It makes risky allocations into assertions. It's
useful for automated testing because the double-abort often represents
itself as an indefinite hang. If it happens before the thread dumping
supervisor process is started, or after it ends, the crash reporter
hangs.

* [crash] Remove reliance on nested SIGABRT/double-fault (broken on OSX)

* [crash] Fix top-level handling of double faults/assertions

* [runtime] Make fatal unwinding errors return into handled error paths

* [crash] Change dumper logging for better info

* [runtime] Fix handling of segfault on sgen thread

Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame #1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame #2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame #3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame #4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame #5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame #6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame #7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame #8: 0x0000000110bb81c1
    frame #9: 0x000000011085ffe1
    frame #10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame #11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame #12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame #13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame #14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame #15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame #16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame #17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame #18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame #19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame #20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame #21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame #22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame #23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread #2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame #1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame #2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame #3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame #4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame #5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame #6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame #8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame #9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame #10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame #11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame #12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread #3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame #2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame #3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame #4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame #5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame #6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread #4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame #2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```

* [crash] Add signal-safe mmap/file "allocator"

* [crash] Remove use of static memory from dumper

* [runtime] Reduce print buffer size for lockless printer.

Each frame that prints ends up increased by the size of buff.
In practice, clang often fails to deduplicate some of these buffers,
leading to 30k-big stackframes.

It was noticed by a series of hard-to-diagnose segfaults on stacks that
looked otherwise fine during the crash reporting stress test.

This change fixes this, making stacks a 1/10th of the size. It doesn't
seem to break the crash reporter messages anywhere (may need to shrink
other "max name length" fields), and it's not mission-critical anywhere
else.

* [crash] Use async-safe file memory for dumper internals

* [crash] Add memory barriers around merp configuration

* [crash] Use signal-safe printers on all native crash paths

* [crash] Move gdb/lldb lookup to startup

* [runtime] Move MOSTLY_ASYNC_SAFE_FPRINTF to eglib

* [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_PRINTF

* [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_FPRINTF

* [crash] Switch to signal-safe exit function

* [crash] Make dumper enum in managed

* [runtime] Add more information to managed frame

* [crash] Make async_safe printers inlined

* [runtime] Move basic pe_file functionality into proclib

* [crash] Fix handling of thread attributes

* [crash] Place hashes into the json file for all threads

* [crash] Fix 2018-08 CI, disable test
luhenry added a commit that referenced this pull request Jan 28, 2019
* [crash] Support extra merp params

* [runtime] Make infrastructure for merp tests

* [runtime] Fix dumping when crash happens without sigctx

* [runtime] Disable crashy (886/1000 runs) stacktrace walker

* [crash] Remove usage of allocating build/os info functions

* [crash] Remove often-crashing g_free on native crash path

* [crash] Add crash_reporter checked build

We add a checked build mode that asserts when mono mallocs inside of
the crash reporter. It makes risky allocations into assertions. It's
useful for automated testing because the double-abort often represents
itself as an indefinite hang. If it happens before the thread dumping
supervisor process is started, or after it ends, the crash reporter
hangs.

* [crash] Remove reliance on nested SIGABRT/double-fault (broken on OSX)

* [crash] Fix top-level handling of double faults/assertions

* [runtime] Make fatal unwinding errors return into handled error paths

* [crash] Change dumper logging for better info

* [runtime] Fix handling of segfault on sgen thread

Threads without domains that get segfaults will end up in
this handler. It's not safe to call this function with a NULL domain.

See crash below:

```
* thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10eff40f8)
  * frame #0: 0x000000010e1510d9 mono-sgen`mono_threads_summarize_execute(ctx=0x0000000000000000, out=0x0000001000000000, hashes=0x0000100000100000, silent=4096, mem="", provided_size=2199023296512) at threads.c:6414
    frame #1: 0x000000010e152092 mono-sgen`mono_threads_summarize(ctx=0x000000010effda00, out=0x000000010effdba0, hashes=0x000000010effdb90, silent=0, signal_handler_controller=1, mem=0x0000000000000000, provided_size=0) at threads.c:6508
    frame #2: 0x000000010df7c69f mono-sgen`dump_native_stacktrace(signal="SIGSEGV", ctx=0x000000010effef48) at mini-posix.c:1026
    frame #3: 0x000000010df7c37f mono-sgen`mono_dump_native_crash_info(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-posix.c:1147
    frame #4: 0x000000010de720a9 mono-sgen`mono_handle_native_crash(signal="SIGSEGV", ctx=0x000000010effef48, info=0x000000010effeee0) at mini-exceptions.c:3227
    frame #5: 0x000000010dd6ac0d mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48, debug_fault_addr=0xffffffffffffffff) at mini-runtime.c:3574
    frame #6: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000000010effeee0, context=0x000000010effef48) at mini-runtime.c:3612
    frame #7: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame #8: 0x0000000110bb81c1
    frame #9: 0x000000011085ffe1
    frame #10: 0x000000010dd6d4f3 mono-sgen`mono_jit_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x00007ffee1ea9f08, error=0x00007ffee1eaa250) at mini-runtime.c:3215
    frame #11: 0x000000010e11509d mono-sgen`do_runtime_invoke(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, exc=0x0000000000000000, error=0x00007ffee1eaa250) at object.c:2977
    frame #12: 0x000000010e10d961 mono-sgen`mono_runtime_invoke_checked(method=0x00007faae4f01fe8, obj=0x0000000000000000, params=0x00007ffee1eaa180, error=0x00007ffee1eaa250) at object.c:3145
    frame #13: 0x000000010e11aa58 mono-sgen`do_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5042
    frame #14: 0x000000010e118803 mono-sgen`mono_runtime_exec_main_checked(method=0x00007faae4f01fe8, args=0x000000010f0003e8, error=0x00007ffee1eaa250) at object.c:5138
    frame #15: 0x000000010e118856 mono-sgen`mono_runtime_run_main_checked(method=0x00007faae4f01fe8, argc=2, argv=0x00007ffee1eaa760, error=0x00007ffee1eaa250) at object.c:4599
    frame #16: 0x000000010de1db2f mono-sgen`mono_jit_exec_internal(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1298
    frame #17: 0x000000010de1d95d mono-sgen`mono_jit_exec(domain=0x00007faae4f00860, assembly=0x00007faae4c02ab0, argc=2, argv=0x00007ffee1eaa760) at driver.c:1257
    frame #18: 0x000000010de2257f mono-sgen`main_thread_handler(user_data=0x00007ffee1eaa6a0) at driver.c:1375
    frame #19: 0x000000010de20852 mono-sgen`mono_main(argc=3, argv=0x00007ffee1eaa758) at driver.c:2551
    frame #20: 0x000000010dd56d7e mono-sgen`mono_main_with_options(argc=3, argv=0x00007ffee1eaa758) at main.c:50
    frame #21: 0x000000010dd5638d mono-sgen`main(argc=3, argv=0x00007ffee1eaa758) at main.c:406
    frame #22: 0x00007fff73aaf015 libdyld.dylib`start + 1
    frame #23: 0x00007fff73aaf015 libdyld.dylib`start + 1
  thread #2, name = 'SGen worker'
    frame #0: 0x000000010e2afd77 mono-sgen`mono_get_hazardous_pointer(pp=0x0000000000000178, hp=0x000000010ef87618, hazard_index=0) at hazard-pointer.c:208
    frame #1: 0x000000010e0b28e1 mono-sgen`mono_jit_info_table_find_internal(domain=0x0000000000000000, addr=0x00007fff73bffa16, try_aot=1, allow_trampolines=1) at jit-info.c:304
    frame #2: 0x000000010dd6aa5f mono-sgen`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0, debug_fault_addr=0x000000010e28fb20) at mini-runtime.c:3540
    frame #3: 0x000000010dd6a8d3 mono-sgen`mono_sigsegv_signal_handler(_dummy=11, _info=0x000070000fb81c58, context=0x000070000fb81cc0) at mini-runtime.c:3612
    frame #4: 0x00007fff73dbdf5a libsystem_platform.dylib`_sigtramp + 26
    frame #5: 0x00007fff73bffa17 libsystem_kernel.dylib`__psynch_cvwait + 11
    frame #6: 0x00007fff73dc8589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #7: 0x000000010e28d76d mono-sgen`mono_os_cond_wait(cond=0x000000010e44c9d8, mutex=0x000000010e44c998) at mono-os-mutex.h:168
    frame #8: 0x000000010e28df4f mono-sgen`get_work(worker_index=0, work_context=0x000070000fb81ee0, do_idle=0x000070000fb81ed4, job=0x000070000fb81ec8) at sgen-thread-pool.c:165
    frame #9: 0x000000010e28d2cb mono-sgen`thread_func(data=0x0000000000000000) at sgen-thread-pool.c:196
    frame #10: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame #11: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame #12: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread #3, name = 'Finalizer'
    frame #0: 0x00007fff73bf6246 libsystem_kernel.dylib`semaphore_wait_trap + 10
    frame #1: 0x000000010e1d9c0a mono-sgen`mono_os_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-os-semaphore.h:84
    frame #2: 0x000000010e1d832d mono-sgen`mono_coop_sem_wait(sem=0x000000010e43e400, flags=MONO_SEM_FLAGS_ALERTABLE) at mono-coop-semaphore.h:41
    frame #3: 0x000000010e1da787 mono-sgen`finalizer_thread(unused=0x0000000000000000) at gc.c:920
    frame #4: 0x000000010e152919 mono-sgen`start_wrapper_internal(start_info=0x0000000000000000, stack_ptr=0x000070000fd85000) at threads.c:1178
    frame #5: 0x000000010e1525b6 mono-sgen`start_wrapper(data=0x00007faae4f31bd0) at threads.c:1238
    frame #6: 0x00007fff73dc7661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff73dc750d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff73dc6bf9 libsystem_pthread.dylib`thread_start + 13
  thread #4
    frame #0: 0x00007fff73c0028a libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff73dc7009 libsystem_pthread.dylib`_pthread_wqthread + 1035
    frame #2: 0x00007fff73dc6be9 libsystem_pthread.dylib`start_wqthread + 13
(lldb)
```

* [crash] Add signal-safe mmap/file "allocator"

* [crash] Remove use of static memory from dumper

* [runtime] Reduce print buffer size for lockless printer.

Each frame that prints ends up increased by the size of buff.
In practice, clang often fails to deduplicate some of these buffers,
leading to 30k-big stackframes.

It was noticed by a series of hard-to-diagnose segfaults on stacks that
looked otherwise fine during the crash reporting stress test.

This change fixes this, making stacks a 1/10th of the size. It doesn't
seem to break the crash reporter messages anywhere (may need to shrink
other "max name length" fields), and it's not mission-critical anywhere
else.

* [crash] Use async-safe file memory for dumper internals

* [crash] Add memory barriers around merp configuration

* [crash] Use signal-safe printers on all native crash paths

* [crash] Move gdb/lldb lookup to startup

* [runtime] Move MOSTLY_ASYNC_SAFE_FPRINTF to eglib

* [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_PRINTF

* [runtime] Fix all callsites of MOSTLY_ASYNC_SAFE_FPRINTF

* [crash] Switch to signal-safe exit function

* [crash] Make dumper enum in managed

* [runtime] Add more information to managed frame

* [crash] Make async_safe printers inlined

* [runtime] Move basic pe_file functionality into proclib

* [crash] Fix handling of thread attributes

* [crash] Place hashes into the json file for all threads

* [crash] Improve reporting on merp test

* [crash] Log the locations of written-to files, for better post-portems
lewurm added a commit to lewurm/mono that referenced this pull request Sep 20, 2019
Fixes
```
* thread mono#12, name = 'tid_a507', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1be66144)
  * frame #0: 0x1be66144 libsystem_c.dylib`__abort + 184
    frame #1: 0x1be6608c libsystem_c.dylib`abort + 152
    frame #2: 0x003e1fa0 monotouchtest`log_callback(log_domain=0x00000000, log_level="error", message="* Assertion at ../../../../../mono/utils/hazard-pointer.c:158, condition `mono_bitset_test_fast (small_id_table, id)' not met\n", fatal=4, user_data=0x00000000) at runtime.m:1251:3
    frame mono#3: 0x003abf44 monotouchtest`monoeg_g_logv_nofree(log_domain=0x00000000, log_level=G_LOG_LEVEL_ERROR, format=<unavailable>, args=<unavailable>) at goutput.c:149:2 [opt]
    frame mono#4: 0x003abfb4 monotouchtest`monoeg_assertion_message(format=<unavailable>) at goutput.c:184:22 [opt]
    frame mono#5: 0x003904dc monotouchtest`mono_thread_small_id_free(id=<unavailable>) at hazard-pointer.c:0:2 [opt]
    frame mono#6: 0x003a0a74 monotouchtest`unregister_thread(arg=0x15c88400) at mono-threads.c:588:2 [opt]
    frame mono#7: 0x00336110 monotouchtest`mono_thread_detach_if_exiting at threads.c:1571:4 [opt]
    frame mono#8: 0x003e7a14 monotouchtest`::xamarin_release_trampoline(self=0x166452f0, sel="release") at trampolines.m:644:3
    frame mono#9: 0x001cdc40 monotouchtest`::-[__Xamarin_NSTimerActionDispatcher release](self=0x166452f0, _cmd="release") at registrar.m:83445:3
    frame mono#10: 0x1ce2ae68 Foundation`_timerRelease + 80
    frame mono#11: 0x1c31b56c CoreFoundation`CFRunLoopTimerInvalidate + 612
    frame mono#12: 0x1c31a554 CoreFoundation`__CFRunLoopTimerDeallocate + 32
    frame mono#13: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame mono#14: 0x1c2be6e8 CoreFoundation`__CFArrayReleaseValues + 500
    frame mono#15: 0x1c2be4c4 CoreFoundation`CFArrayRemoveAllValues + 104
    frame mono#16: 0x1c31ff64 CoreFoundation`__CFSetApplyFunction_block_invoke + 24
    frame mono#17: 0x1c3b2e18 CoreFoundation`CFBasicHashApply + 116
    frame mono#18: 0x1c31ff10 CoreFoundation`CFSetApplyFunction + 160
    frame mono#19: 0x1c3152cc CoreFoundation`__CFRunLoopDeallocate + 204
    frame mono#20: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame mono#21: 0x1c304a80 CoreFoundation`__CFTSDFinalize + 144
    frame mono#22: 0x1bfa324c libsystem_pthread.dylib`_pthread_tsd_cleanup + 644
    frame mono#23: 0x1bf9cc08 libsystem_pthread.dylib`_pthread_exit + 80
    frame mono#24: 0x1bf9af24 libsystem_pthread.dylib`pthread_exit + 36
    frame mono#25: 0x0039df84 monotouchtest`mono_threads_platform_exit(exit_code=<unavailable>) at mono-threads-posix.c:145:2 [opt]
    frame mono#26: 0x0033bb84 monotouchtest`start_wrapper(data=0x1609e1c0) at threads.c:1296:2 [opt]
    frame mono#27: 0x1bf9b914 libsystem_pthread.dylib`_pthread_body + 128
    frame mono#28: 0x1bf9b874 libsystem_pthread.dylib`_pthread_start + 44
    frame mono#29: 0x1bfa3b94 libsystem_pthread.dylib`thread_start + 4
```
lewurm added a commit to lewurm/mono that referenced this pull request Sep 20, 2019
Fixes
```
* thread mono#12, name = 'tid_a507', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1be66144)
  * frame #0: 0x1be66144 libsystem_c.dylib`__abort + 184
    frame #1: 0x1be6608c libsystem_c.dylib`abort + 152
    frame #2: 0x003e1fa0 monotouchtest`log_callback(log_domain=0x00000000, log_level="error", message="* Assertion at ../../../../../mono/utils/hazard-pointer.c:158, condition `mono_bitset_test_fast (small_id_table, id)' not met\n", fatal=4, user_data=0x00000000) at runtime.m:1251:3
    frame mono#3: 0x003abf44 monotouchtest`monoeg_g_logv_nofree(log_domain=0x00000000, log_level=G_LOG_LEVEL_ERROR, format=<unavailable>, args=<unavailable>) at goutput.c:149:2 [opt]
    frame mono#4: 0x003abfb4 monotouchtest`monoeg_assertion_message(format=<unavailable>) at goutput.c:184:22 [opt]
    frame mono#5: 0x003904dc monotouchtest`mono_thread_small_id_free(id=<unavailable>) at hazard-pointer.c:0:2 [opt]
    frame mono#6: 0x003a0a74 monotouchtest`unregister_thread(arg=0x15c88400) at mono-threads.c:588:2 [opt]
    frame mono#7: 0x00336110 monotouchtest`mono_thread_detach_if_exiting at threads.c:1571:4 [opt]
    frame mono#8: 0x003e7a14 monotouchtest`::xamarin_release_trampoline(self=0x166452f0, sel="release") at trampolines.m:644:3
    frame mono#9: 0x001cdc40 monotouchtest`::-[__Xamarin_NSTimerActionDispatcher release](self=0x166452f0, _cmd="release") at registrar.m:83445:3
    frame mono#10: 0x1ce2ae68 Foundation`_timerRelease + 80
    frame mono#11: 0x1c31b56c CoreFoundation`CFRunLoopTimerInvalidate + 612
    frame mono#12: 0x1c31a554 CoreFoundation`__CFRunLoopTimerDeallocate + 32
    frame mono#13: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame mono#14: 0x1c2be6e8 CoreFoundation`__CFArrayReleaseValues + 500
    frame mono#15: 0x1c2be4c4 CoreFoundation`CFArrayRemoveAllValues + 104
    frame mono#16: 0x1c31ff64 CoreFoundation`__CFSetApplyFunction_block_invoke + 24
    frame mono#17: 0x1c3b2e18 CoreFoundation`CFBasicHashApply + 116
    frame mono#18: 0x1c31ff10 CoreFoundation`CFSetApplyFunction + 160
    frame mono#19: 0x1c3152cc CoreFoundation`__CFRunLoopDeallocate + 204
    frame mono#20: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame mono#21: 0x1c304a80 CoreFoundation`__CFTSDFinalize + 144
    frame mono#22: 0x1bfa324c libsystem_pthread.dylib`_pthread_tsd_cleanup + 644
    frame mono#23: 0x1bf9cc08 libsystem_pthread.dylib`_pthread_exit + 80
    frame mono#24: 0x1bf9af24 libsystem_pthread.dylib`pthread_exit + 36
    frame mono#25: 0x0039df84 monotouchtest`mono_threads_platform_exit(exit_code=<unavailable>) at mono-threads-posix.c:145:2 [opt]
    frame mono#26: 0x0033bb84 monotouchtest`start_wrapper(data=0x1609e1c0) at threads.c:1296:2 [opt]
    frame mono#27: 0x1bf9b914 libsystem_pthread.dylib`_pthread_body + 128
    frame mono#28: 0x1bf9b874 libsystem_pthread.dylib`_pthread_start + 44
    frame mono#29: 0x1bfa3b94 libsystem_pthread.dylib`thread_start + 4
```
lewurm added a commit that referenced this pull request Sep 24, 2019
* [threads] clear small_id_key TLS when unregistering a thread

Fixes
```
* thread #12, name = 'tid_a507', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1be66144)
  * frame #0: 0x1be66144 libsystem_c.dylib`__abort + 184
    frame #1: 0x1be6608c libsystem_c.dylib`abort + 152
    frame #2: 0x003e1fa0 monotouchtest`log_callback(log_domain=0x00000000, log_level="error", message="* Assertion at ../../../../../mono/utils/hazard-pointer.c:158, condition `mono_bitset_test_fast (small_id_table, id)' not met\n", fatal=4, user_data=0x00000000) at runtime.m:1251:3
    frame #3: 0x003abf44 monotouchtest`monoeg_g_logv_nofree(log_domain=0x00000000, log_level=G_LOG_LEVEL_ERROR, format=<unavailable>, args=<unavailable>) at goutput.c:149:2 [opt]
    frame #4: 0x003abfb4 monotouchtest`monoeg_assertion_message(format=<unavailable>) at goutput.c:184:22 [opt]
    frame #5: 0x003904dc monotouchtest`mono_thread_small_id_free(id=<unavailable>) at hazard-pointer.c:0:2 [opt]
    frame #6: 0x003a0a74 monotouchtest`unregister_thread(arg=0x15c88400) at mono-threads.c:588:2 [opt]
    frame #7: 0x00336110 monotouchtest`mono_thread_detach_if_exiting at threads.c:1571:4 [opt]
    frame #8: 0x003e7a14 monotouchtest`::xamarin_release_trampoline(self=0x166452f0, sel="release") at trampolines.m:644:3
    frame #9: 0x001cdc40 monotouchtest`::-[__Xamarin_NSTimerActionDispatcher release](self=0x166452f0, _cmd="release") at registrar.m:83445:3
    frame #10: 0x1ce2ae68 Foundation`_timerRelease + 80
    frame #11: 0x1c31b56c CoreFoundation`CFRunLoopTimerInvalidate + 612
    frame #12: 0x1c31a554 CoreFoundation`__CFRunLoopTimerDeallocate + 32
    frame #13: 0x1c31dde4 CoreFoundation`_CFRelease + 220
    frame #14: 0x1c2be6e8 CoreFoundation`__CFArrayReleaseValues + 500
    frame #15: 0x1c2be4c4 CoreFoundation`CFArrayRemoveAllValues + 104
    frame