[x64/CentOS7] createdump failing with message: terminate called after throwing an instance of 'PAL_SEHException'

[xaviersolau]

Environment:

Dotnet runtime 2.1.4
CentOS 7

I am trying to create a memory dump of a live .Net Core process. Unfortunately createdump itself crashs:

Writing minidump to file /tmp/coredump.58593
terminate called after throwing an instance of 'PAL_SEHException'
Aborted

The crash is not dependent of the createdump option (same crash with -n and -h)

This issue is hard to reproduce since it doesn't occur on all .Net core applications. It may be related to the fact that I am using GRPC since when I remove the use of it from my code it looks like it works.... (but I could not isolate the issue on a single application using GRPC so I am not sure about that)

I used LLDB in order to investigate the issue and I find out that we have an invalid address use:

* thread dotnet/coreclr#1, name = 'createdump', stop reason = signal SIGSEGV: invalid address (fault address: 0x30)
    frame #0: 0x00007ffff7a17890 libmscordaccore.so`MetaSig::Reset()
libmscordaccore.so`MetaSig::Reset:
->  0x7ffff7a17890 <+0>:  movl   0x30(%rdi), %eax
    0x7ffff7a17893 <+3>:  movl   %eax, 0x40(%rdi)
    0x7ffff7a17896 <+6>:  movq   0x28(%rdi), %rax
    0x7ffff7a1789a <+10>: movq   %rax, 0x38(%rdi)

with the call stack:

* thread dotnet/coreclr#1, name = 'createdump', stop reason = signal SIGSEGV: invalid address (fault address: 0x30)
  * frame #0: 0x00007ffff7a17890 libmscordaccore.so`MetaSig::Reset()
    frame dotnet/coreclr#1: 0x00007ffff7967391 libmscordaccore.so`ClrDataFrame::ValueFromDebugInfo(MetaSig*, bool, unsigned int, unsigned int, IXCLRDataValue**) + 337
    frame dotnet/coreclr#2: 0x00007ffff7968438 libmscordaccore.so`ClrDataFrame::GetExactGenericArgsToken(IXCLRDataValue**) + 200
    frame dotnet/coreclr#3: 0x00007ffff78de62f libmscordaccore.so`ClrDataAccess::EnumMemWalkStackHelper(CLRDataEnumMemoryFlags, IXCLRDataStackWalk*, Thread*) + 2655
    frame dotnet/coreclr#4: 0x00007ffff78db1b2 libmscordaccore.so`ClrDataAccess::EnumMemDumpAllThreadsStack(CLRDataEnumMemoryFlags) + 3026
    frame dotnet/coreclr#5: 0x00007ffff78df14f libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWorkerSkinny(CLRDataEnumMemoryFlags) + 127
    frame dotnet/coreclr#6: 0x00007ffff78e14ca libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWrapper(CLRDataEnumMemoryFlags) + 122
    frame dotnet/coreclr#7: 0x00007ffff78e1750 libmscordaccore.so`ClrDataAccess::EnumMemoryRegions(ICLRDataEnumMemoryRegionsCallback*, unsigned int, CLRDataEnumMemoryFlags) + 224
    frame dotnet/coreclr#8: 0x000000000040776b createdump`CrashInfo::EnumerateMemoryRegionsWithDAC(_MINIDUMP_TYPE) + 219
    frame dotnet/coreclr#9: 0x00000000004063f7 createdump`CrashInfo::GatherCrashInfo(_MINIDUMP_TYPE) + 663
    frame dotnet/coreclr#10: 0x0000000000405693 createdump`CreateDumpCommon(char const*, _MINIDUMP_TYPE, CrashInfo*) + 179
    frame dotnet/coreclr#11: 0x00000000004054ee createdump`main + 606
    frame dotnet/coreclr#12: 0x00007ffff6cb7c05 libc.so.6`__libc_start_main + 245
    frame dotnet/coreclr#13: 0x00000000004051c9 createdump

[xaviersolau]

I guess this may be related to the issue dotnet/diagnostics#61

[xaviersolau]

After some investigation I noticed that in the class ClrDataFrame (file stack.cpp) and in the method ClrDataFrame::GetExactGenericArgsToken we call GetLocalSig that is supposed to create the MetaSig instance that is used after in the method ValueFromDebugInfo.
In my case, the GetLocalSig is actually failing to get this object but it is not taken into account in the if (FAILED(status = GetLocalSig(&sig, &numLocals))) because the status is not set to an error but is actually set to S_FALSE.

[jkotas]

cc @mikem8361

[mikem8361]

Thanks for the details. I'll try to get this fixed soon.

Is there anything you could send me that repros the problem?

[xaviersolau]

Well in fact, I have a fix. I will make a pull request tomorrow...

[mikem8361]

I was just working on fix too.

[mikem8361]

I changed GetLocalSig to return E_FAIL instead of S_FALSE. The upstream callers of all the functions that call this function handle actual failure better or the same.

[patricksuo]

will there be any chance to ship the fix with dotnet core 2.2?

[mikem8361]

Fixed in PR dotnet/coreclr#20084

[mikem8361]

I'll try to get it into 2.2 also.

[xaviersolau]

I changed GetLocalSig to return E_FAIL instead of S_FALSE. The upstream callers of all the functions that call this function handle actual failure better or the same.

yes I did the same on my side!

[patricksuo]

@mikem8361

I'll try to get it into 2.2 also.

any news about 2.2 backport?