Crash in GC on arm64

[peters]

Description

.NET Runtime crash in GC with stack.

libcoreclr.so!sigsegv_handler(int code, siginfo_t * siginfo, void * context) Line 572	C++
<signal handler called>	
libcoreclr.so!MethodTable::GetFlag(const MethodTable * this, MethodTable::WFLAGS_HIGH_ENUM flag) Line 3705	C++
libcoreclr.so!MethodTable::HasComponentSize(const MethodTable * this) Line 1688	C++
libcoreclr.so!SVR::my_get_size(Object * ob) Line 9543	C++
libcoreclr.so!SVR::gc_heap::background_mark_simple1(SVR::gc_heap * this, uint8_t * oo, int thread) Line 19516	C++
libcoreclr.so!SVR::gc_heap::background_mark_simple(SVR::gc_heap * this, uint8_t * o, int thread) Line 19711	C++
libcoreclr.so!SVR::gc_heap::background_mark_object(SVR::gc_heap * this, uint8_t * o, int thread) Line 19723	C++
libcoreclr.so!SVR::gc_heap::background_drain_mark_list(SVR::gc_heap * this, int thread) Line 26190	C++
libcoreclr.so!SVR::gc_heap::background_mark_phase(SVR::gc_heap * this) Line 26982	C++
libcoreclr.so!SVR::gc_heap::gc1(SVR::gc_heap * this) Line 16683	C++
libcoreclr.so!SVR::gc_heap::bgc_thread_function(SVR::gc_heap * this) Line 28113	C++
libcoreclr.so!(anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::operator()(void*) const(void * argument) Line 1382	C++
libcoreclr.so!(anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(void*)(void * argument) Line 1362	C++
libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void * pvParam) Line 1845	C++
libpthread.so.0!start_thread(void * arg) Line 463	C
libc.so.6!thread_start() Line 78	C++

The device the crash occurred on is a Jetson AGX XAVIER running Jetpack SDK 4.5.1

Configuration

GC

<ServerGarbageCollection>true</ServerGarbageCollection>
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
<RetainVMGarbageCollection>false</RetainVMGarbageCollection>

OS DETAILS

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

RUNTIME DETAILS

The application is published self-contained on linux-arm64.

.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-arm64
 Base Path:   /home/youpark/dotnet/sdk/5.0.201/

Host (useful for support):
  Version: 5.0.4
  Commit:  f27d337295

.NET SDKs installed:
  5.0.201 [/home/youpark/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.1 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.2 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.1 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.2 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Environment

# Enable .net crash dump
export COMPlus_DbgEnableMiniDump=1
export COMPlus_DbgMiniDumpName=/media/youpark/memorycard/core.minidump.%p
export COMPlus_DbgMiniDumpType=4

# Disable managed debug/attach
export COMPlus_EnableDiagnostics=0
export COMPLUS_ReadyToRun=0

# GC STRESS
export COMPlus_StressLog=1
export COMPlus_LogLevel=6
export COMPlus_LogFacility=0x00080001
export COMPlus_StressLogSize=2000000
export COMPlus_TotalStressLogSize=40000000

gdbinit

set unwindonsignal on
set debug aarch64
add-auto-load-safe-path /lib/aarch64-linux-gnu/libthread_db-1.0.so
set auto-load safe-path /
handle SIGUSR1 nostop noprint pass
handle SIGSEGV nostop
handle SIGILL nostop noprint pass
handle SIGQUIT nostop noprint pass
handle SIG34 nostop noprint pass noignore

Other information

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

.NET Runtime crash in GC with stack.

libcoreclr.so!sigsegv_handler(int code, siginfo_t * siginfo, void * context) Line 572	C++
<signal handler called>	
libcoreclr.so!MethodTable::GetFlag(const MethodTable * this, MethodTable::WFLAGS_HIGH_ENUM flag) Line 3705	C++
libcoreclr.so!MethodTable::HasComponentSize(const MethodTable * this) Line 1688	C++
libcoreclr.so!SVR::my_get_size(Object * ob) Line 9543	C++
libcoreclr.so!SVR::gc_heap::background_mark_simple1(SVR::gc_heap * this, uint8_t * oo, int thread) Line 19516	C++
libcoreclr.so!SVR::gc_heap::background_mark_simple(SVR::gc_heap * this, uint8_t * o, int thread) Line 19711	C++
libcoreclr.so!SVR::gc_heap::background_mark_object(SVR::gc_heap * this, uint8_t * o, int thread) Line 19723	C++
libcoreclr.so!SVR::gc_heap::background_drain_mark_list(SVR::gc_heap * this, int thread) Line 26190	C++
libcoreclr.so!SVR::gc_heap::background_mark_phase(SVR::gc_heap * this) Line 26982	C++
libcoreclr.so!SVR::gc_heap::gc1(SVR::gc_heap * this) Line 16683	C++
libcoreclr.so!SVR::gc_heap::bgc_thread_function(SVR::gc_heap * this) Line 28113	C++
libcoreclr.so!(anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::operator()(void*) const(void * argument) Line 1382	C++
libcoreclr.so!(anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(void*)(void * argument) Line 1362	C++
libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void * pvParam) Line 1845	C++
libpthread.so.0!start_thread(void * arg) Line 463	C
libc.so.6!thread_start() Line 78	C++

The device the crash occurred on is a Jetson AGX XAVIER running Jetpack SDK 4.5.1

Configuration

GC

<ServerGarbageCollection>true</ServerGarbageCollection>
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
<RetainVMGarbageCollection>false</RetainVMGarbageCollection>

OS DETAILS

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

RUNTIME DETAILS

The application is published self-contained on linux-arm64.

.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-arm64
 Base Path:   /home/youpark/dotnet/sdk/5.0.201/

Host (useful for support):
  Version: 5.0.4
  Commit:  f27d337295

.NET SDKs installed:
  5.0.201 [/home/youpark/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.1 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.2 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [/home/youpark/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.1 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.2 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [/home/youpark/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Environment

# Enable .net crash dump
export COMPlus_DbgEnableMiniDump=1
export COMPlus_DbgMiniDumpName=/media/youpark/memorycard/core.minidump.%p
export COMPlus_DbgMiniDumpType=4

# Disable managed debug/attach
export COMPlus_EnableDiagnostics=0
export COMPLUS_ReadyToRun=0

# GC STRESS
export COMPlus_StressLog=1
export COMPlus_LogLevel=6
export COMPlus_LogFacility=0x00080001
export COMPlus_StressLogSize=2000000
export COMPlus_TotalStressLogSize=40000000

gdbinit

set unwindonsignal on
set debug aarch64
add-auto-load-safe-path /lib/aarch64-linux-gnu/libthread_db-1.0.so
set auto-load safe-path /
handle SIGUSR1 nostop noprint pass
handle SIGSEGV nostop
handle SIGILL nostop noprint pass
handle SIGQUIT nostop noprint pass
handle SIG34 nostop noprint pass noignore

Other information

Author:	peters
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[jkotas]

Is it possible for you to share the core dump of the crash with us?

[peters]

@jkotas Sure! I can upload it to Azure. Do you have an email where I can send the link privately?

[jkotas]

My email address is in my github profile.

[peters]

@jkotas I've sent 4 core dumps to your email. The latest core dump I sent you has the same backtrace as originally reported 2 days ago. Let me know if you require any additional details.

(gdb) thread 9
[Switching to thread 9 (Thread 0x7ead1c51c0 (LWP 12394))]
#0  0x0000007f84dbfd5c in __waitpid (pid=<optimized out>, stat_loc=0x7f83543954, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30	../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
(gdb) bt
#0  0x0000007f84dbfd5c in __waitpid (pid=<optimized out>, stat_loc=0x7f83543954, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000007f84702520 in PROCCreateCrashDump (argv=<optimized out>) at /__w/1/s/src/coreclr/src/pal/src/thread/process.cpp:3335
#2  PROCCreateCrashDumpIfEnabled () at /__w/1/s/src/coreclr/src/pal/src/thread/process.cpp:3450
#3  0x0000007f846ce218 in invoke_previous_action (action=<optimized out>, code=11, siginfo=0x7f83543da0, context=0x7f83543e20, signalRestarts=255)
    at /__w/1/s/src/coreclr/src/pal/src/exception/signal.cpp:334
#4  sigsegv_handler (code=11, siginfo=0x7f83543da0, context=0x7f83543e20) at /__w/1/s/src/coreclr/src/pal/src/exception/signal.cpp:571
#5  <signal handler called>
#6  MethodTable::GetFlag (this=0x0, flag=<optimized out>) at /__w/1/s/src/coreclr/src/vm/methodtable.h:3705
#7  MethodTable::HasComponentSize (this=0x0) at /__w/1/s/src/coreclr/src/vm/methodtable.h:1688
#8  WKS::my_get_size (ob=<optimized out>) at /__w/1/s/src/coreclr/src/gc/gc.cpp:9543
#9  WKS::gc_heap::background_mark_simple1 (oo=0x7c78e239f0 "p\022\340\020\177") at /__w/1/s/src/coreclr/src/gc/gc.cpp:19516
#10 0x0000007f84584d6c in WKS::gc_heap::background_mark_simple (o=<optimized out>) at /__w/1/s/src/coreclr/src/gc/gc.cpp:19711
#11 WKS::gc_heap::background_mark_object (o=0x7c78e239f0 "p\022\340\020\177") at /__w/1/s/src/coreclr/src/gc/gc.cpp:19723
#12 WKS::gc_heap::background_drain_mark_list (thread=<optimized out>) at /__w/1/s/src/coreclr/src/gc/gc.cpp:26190
#13 0x0000007f8456ec28 in WKS::gc_heap::background_mark_phase () at /__w/1/s/src/coreclr/src/gc/gc.cpp:26982
#14 0x0000007f8456dd38 in WKS::gc_heap::gc1 () at /__w/1/s/src/coreclr/src/gc/gc.cpp:16683
#15 0x0000007f84584ab0 in WKS::gc_heap::bgc_thread_function () at /__w/1/s/src/coreclr/src/gc/gc.cpp:28113
#16 0x0000007f8444aca0 in (anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::operator()(void*) const (this=<optimized out>, argument=0x7eb2baa5b0)
    at /__w/1/s/src/coreclr/src/vm/gcenv.ee.cpp:1382
#17 (anonymous namespace)::CreateSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(void*) (argument=0x7eb2baa5b0) at /__w/1/s/src/coreclr/src/vm/gcenv.ee.cpp:1362
#18 0x0000007f84704298 in CorUnix::CPalThread::ThreadEntry (pvParam=0x7ea801daa0) at /__w/1/s/src/coreclr/src/pal/src/thread/thread.cpp:1845
#19 0x0000007f84db5088 in start_thread (arg=0x7eb2baa39f) at pthread_create.c:463
#20 0x0000007f84a9effc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
(gdb) frame 8
#8  WKS::my_get_size (ob=<optimized out>) at /__w/1/s/src/coreclr/src/gc/gc.cpp:9543
9543	/__w/1/s/src/coreclr/src/gc/gc.cpp: No such file or directory.
(gdb) info locals
mT = 0x0

[jkotas]

Thanks. I will take a look when I get a chance.

[jkotas]

This is what's happening:

1. StateMachine = default; in AsyncTaskMethodBuilder is translated into a call to JIT_MemSet
2. JIT_MemSet writes a single byte into the destination to validate that it is writeable (https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/arm64/crthelpers.S#L15)
3. The first field of the StateMachine is GC reference. If the background GC sees the GC reference between the point of the first byte being overwritten and the actual memset clearning the rest, it will crash.