Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Which LLDB I should use for analysis crash dump (app run on .NET CORE runtime 2.1.3-servicing-26807-02) from Linux ARM32 #58

Open
shaojun opened this Issue Aug 16, 2018 · 13 comments

Comments

Projects
None yet
6 participants
@shaojun
Copy link

shaojun commented Aug 16, 2018

I've got several my app crash under my linux iot boards (RaspberryPi3B+ and ASUS Tinker Board S), they are all Linux (Debian based) ARM32 system, I've tried varies of LLDB(from 3.5 to 4.0) for loading libsosplugin.so to analysis the dump but all failed, either could not load the core dump, or after I loaded the plugin, after I typed in sos command, it just crashes itself with Segmentation fault, any suggestion for this?

@mikem8361 mikem8361 self-assigned this Aug 16, 2018

@mikem8361

This comment has been minimized.

Copy link
Member

mikem8361 commented Aug 16, 2018

Your symptoms usually indicate a mismatch between the version of lldb and the libsosplugin.so.

Since this is 2.1.x, lldb 3.9 or greater should work with the libsosplugin.so (built in the coreclr repo) that comes with the runtime. Linux arm32 (arm64) lldb/sos support is still a work in progress. There is no testing yet in our test repo for the coreclr sos and I have got to arm on any OS yet, in this diagnostics repo. It is a high priority.

You could try building the coreclr repo (the diagnostics repo doesn't build yet for Linux arm) (the release/2.1 branch) with the version of lldb installed you want to use. I have no idea how hard this is because my experience with arm so far has been weak.

I've also heard that lldb itself has problems on arm.

/cc: @sywhang (he is having similar problems)

@sywhang

This comment has been minimized.

Copy link
Member

sywhang commented Aug 17, 2018

Thanks for reporting this issue! Also adding @hoyosjs who is testing out the arm bits of diagnostics area with me.
As far as I can tell, LLDB on ARM is pretty broken (i.e. can't set breakpoints, occasionally has random segfaults, etc.) even without SOS. The segfault might be an SOS thing though, but I haven't investigated too much into it. I've only tried out lldb 3.9, but haven't looked much into more recent versions of lldb so I can't say if it's also broken on more recent versions of lldb. We hope to get to this issue soon but don't have a committed timeline for it yet.

@shaojun

This comment has been minimized.

Copy link
Author

shaojun commented Aug 17, 2018

with lldb 3.9 on my ASUS Tinker board S (linux ARM32), it failed on loading the dump file with Segmetation fault.
So any suggestion I can take for now for debug my .NET CORE 2.1 application crash issue? it's really urgent for me.
BTW, I've opened the dump file with text editor, can see in last rows, some readable info was there, some exceptions were logged, like stackoverflow and etc.

@mikem8361

This comment has been minimized.

Copy link
Member

mikem8361 commented Aug 17, 2018

I've heard that gdb works better on arm that lldb currently, but you can use SOS with gdb.

@sywhang

This comment has been minimized.

Copy link
Member

sywhang commented Aug 17, 2018

I've heard that gdb works better on arm that lldb currently, but you can use SOS with gdb.

I think what @mikem8361 meant is that you can't use SOS with lldb.

@mikem8361

This comment has been minimized.

Copy link
Member

mikem8361 commented Aug 17, 2018

@sywhang

This comment has been minimized.

Copy link
Member

sywhang commented Aug 17, 2018

@shaojun Since SOS is broken you won't be able to read that dump file. Sorry that we can't be of much help here... :/ With GDB, you won't be able to do managed debugging, but it is still possible to debug at the runtime source level. Make sure the symbols for coreclr is loaded, and there is a good chance you will be able to see where things are going off the rails.

I believe @kouvel has recently tried out repro'ing and fixing an issue on a Raspberry Pi (ARM64 but it runs an ARM32 OS) so he might be able to provide some insights on debugging on Raspberry Pi?

@kouvel

This comment has been minimized.

Copy link
Member

kouvel commented Aug 17, 2018

I was mostly using gdb for native debugging, I didn't have much luck with lldb3.6-3.9 on arm32 for native or managed debugging. I haven't checked the latest versions available from their feed and it's possible that a more recent version if available might work.

@shaojun

This comment has been minimized.

Copy link
Author

shaojun commented Aug 20, 2018

thank you guys, so for now (say Aug of 2018), there's no solution for Linux ARM of managed debugging and diagnostic (like memory dump analysis)?? then when I can expect this?

@sywhang

This comment has been minimized.

Copy link
Member

sywhang commented Aug 20, 2018

That is correct. @tommcdon would have a better idea regarding general timeline on when we would want to start officially supporting ARM diagnostics.

@mikem8361

This comment has been minimized.

Copy link
Member

mikem8361 commented Aug 20, 2018

Is lldb 6.0 available on your ARM platform? The first step is to find/get a version of lldb that works and contact the llvm foundation at https://bugs.llvm.org if it doesn't.

@tommcdon

This comment has been minimized.

Copy link
Member

tommcdon commented Aug 20, 2018

According to the Official LLDB Debugger Status page, "ARM architectures on Linux are untested". As @mikem8361 points out, we are looking for a stable version pf LLDB that runs on ARM32. Once this occurs we can provide a more accurate estimate when Linux ARM32 managed core dump debugging will be available/supported.

@janvorli

This comment has been minimized.

Copy link
Member

janvorli commented Feb 22, 2019

I finally got some success with LLDB 7.0.1 that I've cross-built from sources. This version works fine for live debugging and the SOS plugin works with it too. However, the core dump debugging is still broken. While it doesn't crash while loading the core, it gets incorrect contexts for all threads (most of the registers including PC and SP being 0). So there is no way to get even a call stack.

@mikem8361 mikem8361 added this to the Future milestone Feb 23, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.