Proposal for Javascript API for Node core-dump analysis

[rnchamberlain]

This is a proposal for a Javascript API for reading native and Node/V8 information from a core dump. The idea is to provide an alternative to using a line-mode debugger to analyze the dump, and to allow analysis routines to be written in Javascript. The Javascript API would be supported by native libraries, re-using the existing debugger + plugin solutions to read the core dump and extract the basic V8 structures (eg stack frames, objects) from the dump.

Here is a basic prototype, built on the llnode/lldb project:
indutny/llnode@master...rnchamberlain:llnode_api

The prototype provides the following Javascript APIs to load a named core dump and print the thread stacks (showing both native and Javascript stack frame information):

NODE_SET_METHOD(exports, "loadDump", LoadDump);
NODE_SET_METHOD(exports, "getThreadCount", GetThreadCount);
NODE_SET_METHOD(exports, "getFrameCount", GetFrameCount);
NODE_SET_METHOD(exports, "getFrame", GetFrame);

It also shows how the API could be used to provide a core dump viewer application on Node

[rnchamberlain]

Here's an architecture picture for the proposed Javascript API:

[rnchamberlain]

And a screenshot of how it could be used

[davidmarkclements]

hey @rnchamberlain this is an awesome direction,
it inspired me to put some thought into this:

https://github.com/rnchamberlain/llnode/compare/llnode_api...davidmarkclements:llnode_api

tl;dr - works on OS X now + intermediate JSON format ftw

1. now works on OSX as well - assumes brew install homebrew/versions/llvm36 has been run (and all the crazy code signing stuff you need on OS X) - maybe a better way to get the dylib - also assumes OS X user has lldb cloned into llnode dir (as per install instructions)
2. If you're using OS X (or indeed nvm) the node binary won't be /usr/bin/node, added a second arg to loadDump
3. removed the newlines from all frames - let's think about the frame as data, rather than output
4. exposed some of the unknown frames, because there's still info there
5. standardised labelling (added Native: and Unknown)
6. added an lldb-json-demo - run node lldb-json-demo <my-core-file> this is really just to show an idea, what if we could generate entire core dumps in a singular JSON format, this would make viewing and analysis tools easier to make
7. disabled all fprintf's to stdout, since it conflicts with node writing to stdout

I'd like to be able to extend this further so the JSON output looks something like

[{"type": "js", "invocation": "emit(this=0x00001b....etc"}]

On top of that, I'm really excited about moving further with llnode_api.cc to fetch memory addresses

cc @lucamaraschi

[rnchamberlain]

@davidmarkclements Excellent! Many thanks for adding OSX, and the extra loadDump arg.

(3.) Yes, the API should return data rather than print-ready output. Way to do this could be to have getFrame() return an object containing the frame data. That object could then have an fprint() convenience method. Likewise other elements in the dump.

(4.) Agree, the unknown frames are useful

(6.) Output to a JSON representation is a really good idea and came up in the Post-mortem WG call yesterday. See also @davepacheco #13

(7.) Yes, API should be silent!

For memory addresses/V8 heap objects, @hhellyer has made very good progress adding the underlying API we need into LLDB, see https://github.com/indutny/llnode/pull/16 for detail. Looks like there is a 3.9 branch of LLDB coming up, so this may appear soon in a public release.

OK if I pull your changes into my fork?

cc @indutny

[davidmarkclements]

OK if I pull your changes into my fork?

Yes absolutely! :)

In terms of getFrame returning an object, if it's a legitimate JavaScript object that we bubble up there's no need for fprint (we can just JSON.stringify to serialize). Also, in terms of performance, if getFrame actually returns raw JSON (as a string) this would work better for writing directly out to a file - it would be nice to have both JS objects, and JSON strings

[yjhjstz]

@rnchamberlain can you provide the core file? thanks.

[rnchamberlain]

Hi @yjhjstz
The core files are quite large, so I have not included one as a sample in the project. It is quite easy to create one though. For example on Linux:
ulimit -c unlimited
node --abort-on-uncaught-exception unknown.js

That should produce a core file in the current directory when the "Cannot find module" exception is thrown.

Then to see the llnode API demo, run the following command, and open the URL in a browser as indicated:
node llnode/test/lldbdemo.js

[rnchamberlain]

Slightly different approach from @yjhjstz. This is a 'command handler' addition to the JavaScript API. It allows LLDB commands (including Fedor's v8 plugin extension commands, so 'v8 bt' for example) to be passed in to the NPM module and results returned.

See: https://github.com/rnchamberlain/llnode/pull/1#issuecomment-240725570

[yjhjstz]

@rnchamberlain I make a docker to run demo, easy to deploy.
https://github.com/yjhjstz/llnode/tree/llnode-docker/docker

[rnchamberlain]

Update to the prototype API. Added methods to run the heap scan and display an object histogram (Linux only, using LLDB 3.9):
indutny/llnode@master...rnchamberlain:llnode_api

Builds with node-gyp, demo 'node llnode/test/lldbdemo.js', run in directory containing a core dump.

Next step perhaps to formalize the API, returning JS objects for frames etc, and including the JSON and direct 'command handler' proposals from @davidmarkclements and @yjhjstz

[richardlau]

I believe the work for this is being done over in https://github.com/nodejs/llnode (tracked in nodejs/llnode#14). If for some reason this needs to be tracked outside of the llnode repository, please open a new issue over in https://github.com/nodejs/diagnostics.