Unable to read all frames even V8 ones in core dump taken

[eokoneyo]

Hi I have been interested in debugging nodejs in prod for a while but having to install mdb was quite an issue, it’s really awesome that there is llnode.

I installed llnode on my mac with brew for practice taking core dumps, analyzing points of failure etc it all worked, so I tried setting up with “a real project” using the simple code here so everything is predictable but with a docker container. The idea was to get the core dump move it to my machine and debugging, like I would hope to do in production

This is what I got after moving it to my machine, I’d like to know what I might have done wrong

[hhellyer]

@maziey93 - Can you post the command line you started lldb with and let me know whether the docker image was Mac or Linux?
I suspect that in the first instance you need to make sure lldb can find the node executable that generated the core dump, otherwise we can't find the symbols we need to understand the v8 object layout.

[eokoneyo]

@hhellyer I started lldb using lldb node -c <path-to-core>, the docker image is the official node image for 5.12.0 I know it's some sort of variant of Linux but I'm not sure which it is.

[hhellyer]

Ah, so lldb was running in a Mac terminal? The node executable on your path would have been a Mac one rather than a Linux one?

[eokoneyo]

@hhellyer yes I ran lldb in a Mac terminal, but got the core dump from the node docker image which is more or less Linux, but shouldn't the native C binding stack at least show?

[hhellyer]

Ok, we could probably use some better examples for this case. But here's what happens for me, using a linux VM that mounts my Mac file system so I can access files in either environment:

To see the same as you:
> lldb -c ~/work/core.linux.core.on.mac.dmp
lldb can load this but not find symbols, so even lldb commands like "bt" give very little information.

To make it work, point lldb at the exact linux executable you were running when the dump was created:
> lldb ~/work/node-sdks/node-v6.5.0-linux-x64/bin/node -c ~/work/core.linux.core.on.mac.dmp
(You can zip it up and move it with the dump if you want.)
With this bt will give more output and when you load the llnode plugin you should see that work too. (v8 bt is probably the best test command to start with as it fills in the stack and gives you reference points to start with.)

[eokoneyo]

@hhellyer I see what you mean now, I'm actually running the node v6.9.1 on my machine and the docker's node version is 5.12.0. So do you think that using the same version and most definitely compiling from the same source solve this problem??

[hhellyer]

No - you need the exact same node executable that was used inside docker. You can't use a Mac executable with a Linux core dump. Trying to rebuild the same level from source (even on the same platform) is unlikely to work - you probably won't get exactly the same binary.

The core file and node exe that created it should be treated as a matching pair. When you copy the core file from your docker image copy the node executable as well.

[eokoneyo]

@hhellyer Alright thanks alot, I will give it a try

[hhellyer]

@maziey93 - Let me know if it does/doesn't work. I shall leave this issue open so I can use it to update the docs. If it works I'll base them off what I've written, if it doesn't I'll try and help you get it going so I can write some docs which do work!

[eokoneyo]

@hhellyer it works 🕺🏾🕺🏾

[hhellyer]

Re-opening to remind me to update the documentation!

[bnoordhuis]

@hhellyer Ping, did that documentation update happen?

[hhellyer]

@bnoordhuis - Sorry, got sidetracked - I've raised PR #86.

[brendangregg]

can this ticket be closed?

[hhellyer]

PR #86 has been landed and published to https://www.npmjs.com/package/llnode - Closing this now.

@brendangregg - Thanks for the reminder, sorry I didn't want to close this until the README.md update was visible on npmjs.org.

[hhellyer]

@maziey93 - I know this is closed but I've also published an npm package which can generate a core dump and tar it up with the executable and libraries you need for lldb. It should simplify this sort of thing in the future. (We needed it for some situations where gcore was blocked - mainly inside docker containers.)
It's here:
https://www.npmjs.com/package/gencore
and I've put up a blog article about using it here:
https://developer.ibm.com/node/2017/06/26/using-gencore-module-obtain-core-dumps-use-llnode/

[eokoneyo]

@hhellyer thank you!! I am taking a look at it