AppDomain Unload GC Crash

[jbatt33]

I'm sure the issue is something in my program, but I'm just trying to get some further information from mono's runtime to how I can determine what's going on.

When attempting to unload an AppDomain I getting the following crash from GC:

Thread 16 (Thread 0xf00feb40 (LWP 2503)):
#0 0xf7fc3049 in __kernel_vsyscall ()
#1 0xf7ece697 in waitpid () from /lib/libpthread.so.0
#2 0x566e765b in dump_native_stacktrace (signal=signal@entry=0x56975ae5 "SIGABRT", mctx=mctx@entry=0xf00fd5f8) at mini-posix.c:1112
#3 0x566e77af in mono_dump_native_crash_info (signal=0x56975ae5 "SIGABRT", mctx=0xf00fd5f8, info=0xf00fd64c) at mini-posix.c:1156
#4 0x566a2f72 in mono_handle_native_crash (signal=0x56975ae5 "SIGABRT", mctx=0xf00fd5f8, info=0xf00fd64c) at mini-exceptions.c:3423
#5 0x566e68c4 in sigabrt_signal_handler (_dummy=6, _info=0xf00fd64c, context=0xf00fd6cc) at mini-posix.c:234
#6
#7 0xf7fc3049 in __kernel_vsyscall ()
#8 0xf7d430f6 in raise () from /lib/libc.so.6
#9 0xf7d2cdd4 in abort () from /lib/libc.so.6
#10 0x56957474 in monoeg_assert_abort () at goutput.c:57
#11 0x56939067 in mono_log_write_logfile (log_domain=0x0, level=G_LOG_LEVEL_ERROR, hdr=0, message=0xf06994d0 "* Assertion: should not be reached at sgen-scan-object.h:91\n") at mono-log-common.c:136
#12 0x569573de in monoeg_g_logstr (msg=, log_level=G_LOG_LEVEL_ERROR, log_domain=0x0) at goutput.c:151
#13 monoeg_g_logv_nofree (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x56a26b54 "* Assertion: should not be reached at %s:%d\n", args=0xf00fdea4 "\255ˡV[") at goutput.c:166
#14 0x56957777 in monoeg_assertion_message (format=0x56a26b54 "* Assertion: should not be reached at %s:%d\n") at goutput.c:201
#15 0x56957813 in mono_assertion_message_unreachable (file=0x56a1cbad "sgen-scan-object.h", line=91) at goutput.c:228
#16 0x569188f9 in major_scan_object_no_evacuation (queue=0xf00fe0d0, desc=, full_object=0xf79a4150) at sgen-scan-object.h:91
#17 drain_gray_stack_no_evacuation (queue=) at sgen-marksweep-drain-gray-stack.h:347
#18 drain_gray_stack (queue=0xf00fe0d0) at sgen-marksweep.c:1330
#19 0x568fdaf9 in sgen_drain_gray_stack (ctx=...) at sgen-gc.c:565
#20 finish_gray_stack (generation=generation@entry=1, ctx=...) at sgen-gc.c:1127
#21 0x568ffe62 in major_finish_collection (gc_thread_gray_queue=gc_thread_gray_queue@entry=0xf00fe0d0, reason=reason@entry=0x56a1f556 "user request", is_overflow=is_overflow@entry=0, old_next_pin_slot=497, forced=1) at sgen-gc.c:2252
#22 0x569002f5 in major_do_collection (reason=0x56a1f556 "user request", is_overflow=0, forced=1) at sgen-gc.c:2394
#23 0x569040ae in sgen_perform_collection_inner (requested_size=requested_size@entry=0, generation_to_collect=generation_to_collect@entry=1, reason=reason@entry=0x56a1f556 "user request", forced_serial=1, stw=1) at sgen-gc.c:2594
#24 0x5690474a in sgen_perform_collection (stw=1, forced_serial=1, reason=0x56a1f556 "user request", generation_to_collect=1, requested_size=0) at sgen-gc.c:2679
#25 sgen_gc_collect (generation=1) at sgen-gc.c:3132
#26 0x568db244 in mono_gc_collect (generation=1) at sgen-mono.c:2565
#27 0x567b7a26 in unload_thread_main (arg=0xf4e32910) at appdomain.c:3324
#28 0x5688e5ef in start_wrapper_internal (stack_ptr=0xf00ff000, start_info=0x0) at threads.c:1237
#29 start_wrapper (data=0xf4e329e0) at threads.c:1312
#30 0xf7ec2e99 in start_thread () from /lib/libpthread.so.0
#31 0xf7e00b1a in clone () from /lib/libc.so.6

Checking the source code, this appears to happen when the object's description field is not valid. So my question is really, how can I debug this further?

I have tried using trace and logging statements, but there is no logging at this deep a level (which is understandable).

My though is I have a pointer to an object that was already collected at an earlier time, but the reference pointer is still alive, but not sure. I'm far from an GC expert.

This is with mono 6.12.0.107

[jbatt33]

After a bunch of debugging and head smashing. I got a build to generate the following GC log information.

    Line  25791: 2023-09-15 13:28:53 Allocated pinned object 0xf7944150, vtable: 0xf139f188 (Context), size: 48d
Line  80051: 2023-09-15 13:28:54 Added finalizer for object: 0xf7944150 (Context) (18) to old table
Line  98659: 2023-09-15 13:28:54 Overwrote root at 0xf13655b8 with 0xf7944150
Line 185330: 2023-09-15 13:35:55 Overwrote root at 0xf13655b8 with 0xf7944150
Line 191570: 2023-09-15 13:35:55 Unregistering finalizer for object: 0xf7944150 (Context)
Line 191571: 2023-09-15 13:35:55 Enqueuing object for finalization: 0xf7944150 (Context) (64)
Line 191647: 2023-09-15 13:35:55 Finalizing object 0xf7944150 (Context)
Line 193273: 2023-09-15 13:35:55 Overwrote root at 0xf13655b8 with 0xf7944150
Line 194193: 2023-09-15 13:35:55 Need to cleanup object 0xf7944150
Line 226036: 2023-09-15 13:35:55 Need to cleanup object 0xf7944150
Line 232669: 2023-09-15 13:35:55 Error: gray object 0xf7944150  (desc: 0)

The Error message at the end is inplace of the assert that the object scan normally has:

[AmrAlSayed0]

I'm also facing this issue on Android. I got nothing except this line in the logs

* Assertion: should not be reached at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mono/sgen/sgen-scan-object.h:91