Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

createdump crash while writing minidump #61

Closed
patricksuo opened this issue Aug 21, 2018 · 8 comments
Closed

createdump crash while writing minidump #61

patricksuo opened this issue Aug 21, 2018 · 8 comments
Assignees
@mikem8361

Comments

@patricksuo
Copy link

patricksuo commented Aug 21, 2018 •
edited
Loading

setup info:

docker container `microsoft/dotnet:2.1-sdk`
host VM: Centos 7

I tried to create mindump for a live dotnet core app.
But createdump itself crash...

createdump output:

Writing minidump with heap to file /workspace/SceneServer/SceneServer/core.123
terminate called after throwing an instance of 'PAL_SEHException'
Aborted (core dumped)

createdump stack trace:

(lldb) bt
* thread #1: tid = 249, 0x00007f4ef8667fff libc.so.6`gsignal + 207, name = 'createdump', stop reason = signal SIGABRT
  * frame #0: 0x00007f4ef8667fff libc.so.6`gsignal + 207
    frame #1: 0x00007f4ef866942a libc.so.6`abort + 362
    frame #2: 0x00007f4ef8f800ad libstdc++.so.6`__gnu_cxx::__verbose_terminate_handler() + 349
    frame #3: 0x00007f4ef8f7e066 libstdc++.so.6`___lldb_unnamed_symbol56$$libstdc++.so.6 + 6
    frame #4: 0x00007f4ef8f7e0b1 libstdc++.so.6`std::terminate() + 17
    frame #5: 0x00007f4ef8f7e2c9 libstdc++.so.6`__cxa_throw + 105
    frame #6: 0x00007f4ef943c9af libmscordaccore.so`ThrowExceptionHelper + 79
    frame #7: 0x00007f4ef940f870 libmscordaccore.so
    frame #8: 0x00007f4ef935f391 libmscordaccore.so`ClrDataFrame::ValueFromDebugInfo(MetaSig*, bool, unsigned int, unsigned int, IXCLRDataValue**) + 337
    frame #9: 0x00007f4ef9360438 libmscordaccore.so`ClrDataFrame::GetExactGenericArgsToken(IXCLRDataValue**) + 200
    frame #10: 0x00007f4ef92d662f libmscordaccore.so`ClrDataAccess::EnumMemWalkStackHelper(CLRDataEnumMemoryFlags, IXCLRDataStackWalk*, Thread*) + 2655
    frame #11: 0x00007f4ef92d31b2 libmscordaccore.so`ClrDataAccess::EnumMemDumpAllThreadsStack(CLRDataEnumMemoryFlags) + 3026
    frame #12: 0x00007f4ef92d0fe3 libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWorkerHeap(CLRDataEnumMemoryFlags) + 643
    frame #13: 0x00007f4ef92d94d9 libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWrapper(CLRDataEnumMemoryFlags) + 137
    frame #14: 0x00007f4ef92d975f libmscordaccore.so`ClrDataAccess::EnumMemoryRegions(ICLRDataEnumMemoryRegionsCallback*, unsigned int, CLRDataEnumMemoryFlags) + 239
    frame #15: 0x000000000040776b createdump`CrashInfo::EnumerateMemoryRegionsWithDAC(_MINIDUMP_TYPE) + 219
    frame #16: 0x00000000004063f7 createdump`CrashInfo::GatherCrashInfo(_MINIDUMP_TYPE) + 663
    frame #17: 0x0000000000405693 createdump`CreateDumpCommon(char const*, _MINIDUMP_TYPE, CrashInfo*) + 179
    frame #18: 0x00000000004054ee createdump`main + 606
    frame #19: 0x00007f4ef86552e1 libc.so.6`__libc_start_main + 241
    frame #20: 0x00000000004051c9 createdump
@mikem8361
Copy link
Member

I need a little more information because from the above stack frame only COR_E_OPERATIONCANCELED exceptions should escape the EnumMem* APIs. Is there anyway to package a repo that I can use debug createdump?

Source/line info might be useful. The dotnet-symbol would be a way to get the symbols for your runtime (mainly libcoredaccore.so).

Dumping the PAL_SEHException object in lldb might give me some more info too.

@patricksuo
Copy link
Author

Is there any alt way to get the symbols? dotnet symbol stuck in downloading libmscordaccore.so

Writing: ./out/libmscordaccore.so
ERROR: The operation was canceled. -> Unable to read data from the transport connection: Operation canceled.

full log:

root@localhost:/workspace/SceneServer/SceneServer# dotnet symbol -o ./out -d  ./core.685  
Downloading from http://msdl.microsoft.com/download/symbols/
Writing files to ./out
WARNING: ELF .gnu_debuglink section in /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/createdump: Virtual address range is not mapped 00000000004159A8 4
Writing: ./out/createdump
Writing: ./out/createdump.dbg
WARNING: ELF .gnu_debuglink section in /usr/lib/x86_64-linux-gnu/liburcu-common.so.4.1.0: Virtual address range is not mapped 00007F40133B4188 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/liburcu-common.so.4.1.0/elf-buildid-43cc70757f315199defee4f7578970e359ade3f9/liburcu-common.so.4.1.0'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-43cc70757f315199defee4f7578970e359ade3f9/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4.1.0: Virtual address range is not mapped 00007F40135BC238 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/liburcu-cds.so.4.1.0/elf-buildid-534e34009698400889543010e406a9483c49ca79/liburcu-cds.so.4.1.0'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-534e34009698400889543010e406a9483c49ca79/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4.1.0: Virtual address range is not mapped 00007F40137C43F8 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/liburcu-bp.so.4.1.0/elf-buildid-2cffdc49cbe561a4c8cdc7bb3e301b1e691943fc/liburcu-bp.so.4.1.0'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-2cffdc49cbe561a4c8cdc7bb3e301b1e691943fc/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0: Virtual address range is not mapped 00007F40139D02F0 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/liblttng-ust-tracepoint.so.0.0.0/elf-buildid-af27ed28887b5fb21e253ad2d06238acba04c002/liblttng-ust-tracepoint.so.0.0.0'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-af27ed28887b5fb21e253ad2d06238acba04c002/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0.0.0: Virtual address range is not mapped 00007F4013C54968 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/liblttng-ust.so.0.0.0/elf-buildid-74b641914e8298fbda8fbba805d8087bbc59a629/liblttng-ust.so.0.0.0'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-74b641914e8298fbda8fbba805d8087bbc59a629/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/libcoreclrtraceptprovider.so: Virtual address range is not mapped 00007F4013F09850 4
Writing: ./out/libcoreclrtraceptprovider.so
Writing: ./out/libcoreclrtraceptprovider.so.dbg
WARNING: ELF .gnu_debuglink section in /lib/x86_64-linux-gnu/libdl-2.24.so: Virtual address range is not mapped 00007F40140E91B0 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/libdl-2.24.so/elf-buildid-b895f0831f623c5f23603401d4069f9f94c24761/libdl-2.24.so'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-b895f0831f623c5f23603401d4069f9f94c24761/_.debug'
WARNING: ELF .gnu_debuglink section in /lib/x86_64-linux-gnu/librt-2.24.so: Virtual address range is not mapped 00007F40142F13C0 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/librt-2.24.so/elf-buildid-5d83e0642e645026dbb11f89f7df7106bd821495/librt-2.24.so'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-5d83e0642e645026dbb11f89f7df7106bd821495/_.debug'
WARNING: ELF .gnu_debuglink section in /lib/x86_64-linux-gnu/libpthread-2.24.so: Virtual address range is not mapped 00007F4014512750 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/libpthread-2.24.so/elf-buildid-4285cd3158dde596765c747ae210ab6cbd258b22/libpthread-2.24.so'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-4285cd3158dde596765c747ae210ab6cbd258b22/_.debug'
WARNING: ELF .gnu_debuglink section in /lib/x86_64-linux-gnu/libc-2.24.so: Virtual address range is not mapped 00007F40148AA550 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/libc-2.24.so/elf-buildid-aa889e26a70f98fa8d230d088f7cc5bf43573163/libc-2.24.so'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-aa889e26a70f98fa8d230d088f7cc5bf43573163/_.debug'
WARNING: ELF .gnu_debuglink section in /lib/x86_64-linux-gnu/libgcc_s.so.1: Virtual address range is not mapped 00007F4014AC42A8 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/libgcc_s.so.1/elf-buildid-51ad5fd294cd6c813bed40717347a53434b80b7a/libgcc_s.so.1'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-51ad5fd294cd6c813bed40717347a53434b80b7a/_.debug'
WARNING: ELF .gnu_debuglink section in /lib/x86_64-linux-gnu/libm-2.24.so: Virtual address range is not mapped 00007F4014DC8220 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/libm-2.24.so/elf-buildid-1b95e3a8b8788b07e4f59ee69b1877f9deb42033/libm-2.24.so'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-1b95e3a8b8788b07e4f59ee69b1877f9deb42033/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22: Virtual address range is not mapped 00007F4015146D98 4
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/libstdc%2B%2B.so.6.0.22/elf-buildid-0972dabed9d271753baf023aca990613ddb217b6/libstdc%2B%2B.so.6.0.22'
WARNING: HttpSymbolStore: 404 Not Found 'http://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-0972dabed9d271753baf023aca990613ddb217b6/_.debug'
WARNING: ELF .gnu_debuglink section in /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/libmscordaccore.so: Virtual address range is not mapped 00007F4015719088 4
Writing: ./out/libmscordaccore.so
ERROR: The operation was canceled. -> Unable to read data from the transport connection: Operation canceled.

@patricksuo
Copy link
Author

stack trace with symbols info:

(lldb) bt
* thread #1: tid = 685, 0x00007f4014741fff libc.so.6`gsignal + 207, name = 'createdump', stop reason = signal SIGABRT
  * frame #0: 0x00007f4014741fff libc.so.6`gsignal + 207
    frame #1: 0x00007f401474342a libc.so.6`abort + 362
    frame #2: 0x00007f401505a0ad libstdc++.so.6`__gnu_cxx::__verbose_terminate_handler() + 349
    frame #3: 0x00007f4015058066 libstdc++.so.6`___lldb_unnamed_symbol56$$libstdc++.so.6 + 6
    frame #4: 0x00007f40150580b1 libstdc++.so.6`std::terminate() + 17
    frame #5: 0x00007f40150582c9 libstdc++.so.6`__cxa_throw + 105
    frame #6: 0x00007f40155169af libmscordaccore.so`::ThrowExceptionHelper(ex=<unavailable>) + 79 at seh.cpp:202
    frame #7: 0x00007f40154e9870 libmscordaccore.so at siginfo.cpp:0
    frame #8: 0x00007f4015439391 libmscordaccore.so`ClrDataFrame::ValueFromDebugInfo(this=<unavailable>, sig=<unavailable>, isArg=false, sigIndex=<unavailable>, varInfoSlot=<unavailable>, _value=0x00007ffde8741ef0) + 337 at stack.cpp:1364
    frame #9: 0x00007f401543a438 libmscordaccore.so`ClrDataFrame::GetExactGenericArgsToken(this=0x00000000024bac20, genericToken=0x00007ffde8741ef0) + 200 at stack.cpp:1093
    frame #10: 0x00007f40153b062f libmscordaccore.so`ClrDataAccess::EnumMemWalkStackHelper(this=<unavailable>, flags=<unavailable>, pStackWalk=<unavailable>, pThread=<unavailable>) + 2655 at enummem.cpp:934
    frame #11: 0x00007f40153ad1b2 libmscordaccore.so`ClrDataAccess::EnumMemDumpAllThreadsStack(this=0x000000000249bb20, flags=CLRDATA_ENUM_MEM_DEFAULT) + 3026 at enummem.cpp:1241
    frame #12: 0x00007f40153b114f libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWorkerSkinny(this=0x000000000249bb20, flags=CLRDATA_ENUM_MEM_DEFAULT) + 127 at enummem.cpp:1588
    frame #13: 0x00007f40153b34ca libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWrapper(this=0x000000000249bb20, flags=CLRDATA_ENUM_MEM_DEFAULT) + 122 at enummem.cpp:1842
    frame #14: 0x00007f40153b3750 libmscordaccore.so`ClrDataAccess::EnumMemoryRegions(this=0x000000000249bb20, callback=0x00000000024774a0, miniDumpFlags=0, flags=<unavailable>) + 224 at enummem.cpp:1954
    frame #15: 0x000000000040776b createdump`CrashInfo::EnumerateMemoryRegionsWithDAC(this=0x00000000024774a0, minidumpType=MiniDumpNormal) + 219 at crashinfo.cpp:643
    frame #16: 0x00000000004063f7 createdump`CrashInfo::GatherCrashInfo(this=<unavailable>, minidumpType=<unavailable>) + 663 at crashinfo.cpp:209
    frame #17: 0x0000000000405693 createdump`CreateDumpCommon(dumpPathTemplate="", minidumpType=MiniDumpNormal, crashInfo=0x00000000024774a0) + 179 at createdump.cpp:47
    frame #18: 0x00000000004054ee createdump`main(argc=<unavailable>, argv=<unavailable>) + 606 at main.cpp:81
    frame #19: 0x00007f401472f2e1 libc.so.6`__libc_start_main + 241
    frame #20: 0x00000000004051c9 createdump`_start + 41

@patricksuo
Copy link
Author

patricksuo commented Aug 22, 2018 •
edited
Loading

@mikem8361 ,

Is there anyway to package a repo that I can use debug createdump?

I can't reproduce with a trivial dotnet app yet.

Source/line info might be useful. The dotnet-symbol would be a way to get the symbols for your runtime (mainly libcoredaccore.so).

done.

Dumping the PAL_SEHException object in lldb might give me some more info too.

I don't know how to.
i tried to dump ex in frame 6, but no luck.

(lldb) fr s 6
frame #6: 0x00007f40155169af libmscordaccore.so`::ThrowExceptionHelper(ex=<unavailable>) + 79 at seh.cpp:202
(lldb) fr v ex
(PAL_SEHException *) ex = <variable not available>

@patricksuo
Copy link
Author

patricksuo commented Aug 22, 2018 •
edited
Loading

BTW, can I mirror the http://msdl.microsoft.com/download/symbols ?
I have trouble downloading symbols from this website in China.

@mikem8361
Copy link
Member

I have no idea if you can or should mirror Microsoft's websites.

But the "The operation was canceled. " errors the symbol downloader was hitting are probably network problems. I look into see if I can increase the read time outs.

@patricksuo
Copy link
Author

patricksuo commented Aug 23, 2018 •
edited
Loading

@mikem8361

you can debug createdump with the following program and docker image :

you also need consul to run this program.

@mikem8361 mikem8361 self-assigned this Aug 23, 2018
mikem8361 added a commit to mikem8361/coreclr that referenced this issue Sep 20, 2018
@mikem8361
Fix issue #20076 createdump faults.
f8e81e2 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
@mikem8361 mikem8361 mentioned this issue Sep 20, 2018
Merged
mikem8361 added a commit to dotnet/coreclr that referenced this issue Sep 21, 2018
@mikem8361
Fix issue #20076 createdump faults. (#20084)
f1cf6e7 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
@mikem8361
Copy link
Member

Fixed in PR dotnet/coreclr#20084

xaviersolau pushed a commit to criteo-forks/coreclr that referenced this issue Sep 21, 2018
@mikem8361
Fix issue #20076 createdump faults. (dotnet#20084)
4d7898a 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
mikem8361 added a commit to mikem8361/coreclr that referenced this issue Oct 12, 2018
@mikem8361
Fix issue #20076 createdump faults.
81ff03f 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
@mikem8361 mikem8361 mentioned this issue Oct 12, 2018
Merged
mikem8361 added a commit to mikem8361/coreclr that referenced this issue Oct 15, 2018
@mikem8361
Fix issue #20076 createdump faults.
6b444de 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
mikem8361 added a commit to mikem8361/coreclr that referenced this issue Oct 23, 2018
@mikem8361
Fix issue #20076 createdump faults.
5cf3d27 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
mikem8361 added a commit to dotnet/coreclr that referenced this issue Oct 25, 2018
@mikem8361
Fix issue #20076 createdump faults. (#20403)
fd1893f 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
crntn pushed a commit to criteo-forks/coreclr that referenced this issue Nov 12, 2018
@mikem8361 @crntn
Fix issue #20076 createdump faults. (dotnet#20084)
45c1ee3 
Return E_FAIL instead of S_FALSE from ClrDataFrame::GetLocalSig().

Also issue dotnet/diagnostics#61
@ghost ghost locked as resolved and limited conversation to collaborators Jun 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mikem8361 mikem8361

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@patricksuo @mikem8361