createdump segfaults with ASP.NET Core 2.2 applications #11645

tmds · 2018-12-11T07:31:27Z

Using the 2.2 sdk:

Create a new ASP.NET Core app using the web template
Publish the app
Run it: dotnet <myapp.dll>
Find its pid and run createdump <pid>.

createdump segfaults:

Thread 1 "createdump" received signal SIGSEGV, Segmentation fault.
0x00007ffff7b00c43 in ClrDataAccess::EnumWksGlobalMemoryRegions(CLRDataEnumMemoryFlags) ()
   from /home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App/2.2.0/libmscordaccore.so
(gdb) bt
#0  0x00007ffff7b00c43 in ClrDataAccess::EnumWksGlobalMemoryRegions(CLRDataEnumMemoryFlags) () from /home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App/2.2.0/libmscordaccore.so
dotnet/coreclr#1  0x00007ffff7acf557 in ClrDataAccess::EnumMemCLRHeapCrticalStatic(CLRDataEnumMemoryFlags) () from /home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App/2.2.0/libmscordaccore.so
dotnet/coreclr#2  0x00007ffff7ad49ae in ClrDataAccess::EnumMemoryRegionsWorkerHeap(CLRDataEnumMemoryFlags) () from /home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App/2.2.0/libmscordaccore.so
dotnet/coreclr#3  0x00007ffff7adc4ce in ClrDataAccess::EnumMemoryRegionsWrapper(CLRDataEnumMemoryFlags) () from /home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App/2.2.0/libmscordaccore.so
dotnet/coreclr#4  0x00007ffff7adc739 in ClrDataAccess::EnumMemoryRegions(ICLRDataEnumMemoryRegionsCallback*, unsigned int, CLRDataEnumMemoryFlags) ()
   from /home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App/2.2.0/libmscordaccore.so
dotnet/coreclr#5  0x000000000040c16b in CrashInfo::EnumerateMemoryRegionsWithDAC(_MINIDUMP_TYPE) ()
dotnet/coreclr#6  0x000000000040adf7 in CrashInfo::GatherCrashInfo(_MINIDUMP_TYPE) ()
dotnet/coreclr#7  0x000000000040a093 in CreateDumpCommon(char const*, _MINIDUMP_TYPE, CrashInfo*) ()
dotnet/coreclr#8  0x0000000000409bbe in main ()

Dotnet info:

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.100
 Commit:    b9f2fa0ca8

Runtime Environment:
 OS Name:     fedora
 OS Version:  29
 OS Platform: Linux
 RID:         fedora.29-x64
 Base Path:   /home/tmds/Downloads/2.2/sdk/2.2.100/

Host (useful for support):
  Version: 2.2.0
  Commit:  1249f08fed

.NET Core SDKs installed:
  2.2.100 [/home/tmds/Downloads/2.2/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.0 [/home/tmds/Downloads/2.2/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.0 [/home/tmds/Downloads/2.2/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.0 [/home/tmds/Downloads/2.2/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

The text was updated successfully, but these errors were encountered:

jkotas · 2018-12-11T12:14:06Z

cc @mikem8361

mikem8361 · 2018-12-11T18:14:37Z

Is there anyway I could get the test app? The binary would be enough.

omajid · 2018-12-11T19:09:47Z

As @tmds said in the original post, this was just dotnet new webapp && dotnet run.

mikem8361 · 2018-12-11T22:42:10Z

I’ve been able to repro the problem and currently investigating.

The problem is the ClrDataModule Request faulted on a dynamic module getting the file layout flag. Fixed the Request code not get the file layout and in the crash dump code skip any dynamic modules.

Issue #21485: fix EnumProcessModules hPseudoCurrentProcess bug. Added handle reference. Issue #21484: createdump segfaults with ASP.NET app The problem is the ClrDataModule Request faulted on a dynamic module getting the file layout flag. Fixed the Request code not get the file layout and in the crash dump code skip any dynamic modules.

Issue #21484: createdump segfaults with ASP.NET app The problem is the ClrDataModule Request faulted on a dynamic module getting the file layout flag. Fixed the Request code not get the file layout and in the crash dump code skip any dynamic modules. Issue #21485: fix EnumProcessModules hPseudoCurrentProcess bug.

patricksuo · 2019-01-04T04:16:23Z

hi @mikem8361 ,

patched createdump/libmscordac.so still crash.

BT indicates segfault site is https://github.com/dotnet/coreclr/blob/b24eee89da14b63be629b5bab80d24990316e1ba/src/debug/daccess/request.cpp#L3808-L3808 , which happens before https://github.com/dotnet/coreclr/blob/b24eee89da14b63be629b5bab80d24990316e1ba/src/debug/createdump/crashinfo.cpp#L656

Since segfault happens before the ClrDataModule Request ,
how does dotnet/coreclr#21526 fix this issue?
I have limited knowledge about DAC.
Could you elaborate this fix?

BTW, createdump works well in MiniDumpWithFullMemory mode.

mikem8361 · 2019-01-04T05:10:30Z

The segfault I repro'ed locally was in the DAC task.cpp RequestGetModuleData on a "dynamic" module the asp.net app had. It was on Ubuntu 14.04 instead of Fedora and it was the version 2.2.101 SDK instead of 2.2.100. I fixed a real problem but just not the one you ran into.

I'll try to repro on a Fedora docker image.

mikem8361 · 2019-01-04T05:40:46Z

I'm curious how you built the patched version of 2.2 with my fixes? Did you build and use all of the coreclr repo binaries? Because you can't just take createdump and libmscordaccore.so. You need the rest (matching libcoreclr.so, etc.).

patricksuo · 2019-01-04T05:41:15Z

Did you build and use all of the coreclr repo binaries?

yes

GC heap globals like ephemeral_heap_segment and finalize_queue were null/invalid for the simple webapp's heap. Added validity checks before dereferencing them. The dynamic module fix is still needed, createdump would have still hit this problem after the GC heap fix.

mikem8361 · 2019-01-04T18:39:28Z

@sillyousu, I just pushed a fix for your actually problem.

@noahfalk, can you review the latest commit?

mikem8361 · 2019-01-04T18:40:49Z

@noahfalk, see PR dotnet/coreclr#21534

GC heap globals like ephemeral_heap_segment and finalize_queue are null/invalid for a server GC. Add a check to skip the workstation GC memory enumeration if server. The server memory enumeration already skips if workstation GC. The dynamic module fix is still needed, createdump would have still hit this problem after the GC heap fix.

GC heap globals like ephemeral_heap_segment and finalize_queue are null/invalid for a server GC. Add a check to skip the workstation GC memory enumeration if server. The server memory enumeration already skips if workstation GC. The dynamic module fix is still needed, createdump would have still hit this problem after the GC heap fix. Report the entire generation table in EnumWksGlobalMemoryRegions and EnumSvrGlobalMemoryRegions (dotnet#20233).

GC heap globals like ephemeral_heap_segment and finalize_queue are null/invalid for a server GC. Add a check to skip the workstation GC memory enumeration if server. The server memory enumeration already skips if workstation GC.

patricksuo · 2019-01-05T05:12:28Z

@sillyousu, I just pushed a fix for your actual problem.

Thanks, @mikem8361.

~~Do you happen to know how can I keep debug symbol in Release build?~~

mikem8361 · 2019-01-05T18:44:57Z

Add the “stripSymbols” option to the “build.sh” command and it should generate *.dbg files that contain the symbols.

GC heap globals like ephemeral_heap_segment and finalize_queue are null/invalid for a server GC. Add a check to skip the workstation GC memory enumeration if server. The server memory enumeration already skips if workstation GC.

Issue #21484: createdump segfaults with ASP.NET app The problem is the ClrDataModule Request faulted on a dynamic module getting the file layout flag. Fixed the Request code not get the file layout and in the crash dump code skip any dynamic modules. Issue #21485: fix EnumProcessModules hPseudoCurrentProcess bug.

GC heap globals like ephemeral_heap_segment and finalize_queue are null/invalid for a server GC. Add a check to skip the workstation GC memory enumeration if server. The server memory enumeration already skips if workstation GC. The dynamic module fix is still needed, createdump would have still hit this problem after the GC heap fix. Report the entire generation table in EnumWksGlobalMemoryRegions and EnumSvrGlobalMemoryRegions (#20233).

mikem8361 · 2019-01-14T18:59:56Z

Fixed.

This was recently fixed upstream. See : https://github.com/dotnet/coreclr/issues/21484

This was recently fixed upstream. See: https://github.com/dotnet/coreclr/issues/21484

mikem8361 self-assigned this Dec 11, 2018

mikem8361 closed this as completed Jan 14, 2019

omajid referenced this issue in omajid/dotnet-regular-tests Feb 18, 2019

Verify that createdump works on ASP.NET Core

75c7df2

This was recently fixed upstream. See : https://github.com/dotnet/coreclr/issues/21484

omajid referenced this issue in omajid/dotnet-regular-tests Feb 18, 2019

Verify that createdump works on ASP.NET Core

b4d4066

This was recently fixed upstream. See : https://github.com/dotnet/coreclr/issues/21484

omajid referenced this issue in omajid/dotnet-regular-tests Feb 20, 2019

Verify that createdump works on ASP.NET Core

7a78c2a

This was recently fixed upstream. See: https://github.com/dotnet/coreclr/issues/21484

RheaAyase referenced this issue in redhat-developer/dotnet-regular-tests Feb 20, 2019

Verify that createdump works on ASP.NET Core

11b73d2

This was recently fixed upstream. See: https://github.com/dotnet/coreclr/issues/21484

msftgits transferred this issue from dotnet/coreclr Jan 31, 2020

msftgits added this to the 3.0 milestone Jan 31, 2020

patricksuo mentioned this issue Jan 31, 2020

Is createdump runtime independent? #11710

Closed

dotnet locked as resolved and limited conversation to collaborators Dec 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

createdump segfaults with ASP.NET Core 2.2 applications #11645

createdump segfaults with ASP.NET Core 2.2 applications #11645

tmds commented Dec 11, 2018

jkotas commented Dec 11, 2018

mikem8361 commented Dec 11, 2018

omajid commented Dec 11, 2018

mikem8361 commented Dec 11, 2018 via email

patricksuo commented Jan 4, 2019 •

edited

mikem8361 commented Jan 4, 2019

mikem8361 commented Jan 4, 2019

patricksuo commented Jan 4, 2019 •

edited

mikem8361 commented Jan 4, 2019

mikem8361 commented Jan 4, 2019

patricksuo commented Jan 5, 2019 •

edited

mikem8361 commented Jan 5, 2019 via email

mikem8361 commented Jan 14, 2019

createdump segfaults with ASP.NET Core 2.2 applications #11645

createdump segfaults with ASP.NET Core 2.2 applications #11645

Comments

tmds commented Dec 11, 2018

jkotas commented Dec 11, 2018

mikem8361 commented Dec 11, 2018

omajid commented Dec 11, 2018

mikem8361 commented Dec 11, 2018 via email

patricksuo commented Jan 4, 2019 • edited

mikem8361 commented Jan 4, 2019

mikem8361 commented Jan 4, 2019

patricksuo commented Jan 4, 2019 • edited

mikem8361 commented Jan 4, 2019

mikem8361 commented Jan 4, 2019

patricksuo commented Jan 5, 2019 • edited

mikem8361 commented Jan 5, 2019 via email

mikem8361 commented Jan 14, 2019

patricksuo commented Jan 4, 2019 •

edited

patricksuo commented Jan 4, 2019 •

edited

patricksuo commented Jan 5, 2019 •

edited