Please document the current debugging protocol between an IDE and Node.js (--inspect)

[eranif]

Is your feature request related to a problem? Please describe.
Yes. My name is Eran and I am the author of CodeLite IDE. Up until version ~7 we had a working Node.js debugger using the old API. Since the version 8.X (I think) the API was deprecated and it is recommended to use the new DevTools protocol :/

HOWEVER, for the love of god, I searched the web for 2 days and could not find an example of how to establish a connection to the Node.js or something that describes how the debug session is executed (flow of messages between Node.js and the IDE)

Describe the solution you'd like
Can you please provide such a document? or maybe a simple working example? (can be in any language you choose)
Here is for example, how the XDebug DBGP protocol is documented (using this protocol could have made Node.js Debuggable from many existing IDEs, thus providing a great choice of tools to your user base)

https://xdebug.org/docs-dbgp.php#description

Thanks!

[addaleax]

How close comes https://chromedevtools.github.io/devtools-protocol/ to what you are looking for? It sounds like you’re interested more in the underlying message format for transport than the protocol itself? (I think the answer in that case would be pretty close to “WebSocket messages”, but I’m not sure.)

[eranif]

Close to none?
I understood that its using WebSocket, I was able to connect over the ws://... URL, however, I don't get any feedback from Node.js, no errors, maybe I am sending wrong messages?
There is not a single document I could find that will explain how to communicate with NodeJS. No log? No stderr messages?

[bnoordhuis]

You can look at https://github.com/nodejs/node-inspect for a simple working debug client.

https://github.com/nodejs/node/blob/98819dfa5853d7c8355d70aa1aa7783677c391e5/deps/v8/src/inspector/js_protocol.json describes the protocol.

[eranif]

If we put a side the debugger client implementation language (in my case its C++).
I tried something simple:

• I used wscat (a JS script that uses websockets)
• Run a simple test script under Node.js

So it goes something like this:

C:\Users\PC\Documents\CodeLite\NODEJS\TestDebug>node --inspect-brk Tester.js
Debugger listening on ws://127.0.0.1:9229/7bfa75b1-2502-4da1-985b-dc3d5c92cb58
For help see https://nodejs.org/en/docs/inspector

and on another terminal, I used wscat like this:

wscat -c ws://127.0.0.1:9229/7bfa75b1-2502-4da1-985b-dc3d5c92cb58

A connection is established successfully.
What I would like to understand now is the sequence of messages that Node.js is expecting. Unless I am really misunderstood how the debugger protocol works.
Also, is there a log / verbose mode I can enable to Node.js to help me understand whats going on?

Thanks!

[AyushG3112]

@eranif perhaps this guide is what you are looking for: https://nodejs.org/en/docs/guides/debugging-getting-started/ ? It has a section for Inspector Clients which provide various options to use the inspector.

[eranif]

@AyushG3112 thanks, this page is about how to debug your Node.js application. This is not what I am looking for. I am trying to implement a debugger interaface in my IDE (see my initial comment).

[sagitsofan]

@eranif VSCode is doing what you are looking for, debugging node js apps, so i guess he is using the new api.
Maybe it can help you: https://github.com/Microsoft/vscode

[refack]

@eranif You are right in that we don't have a step by step guide for howto use the inspector protocol. We do have some direct examples in our test suite:

1. a Helper (esseptialy a WS parser/generator) https://github.com/nodejs/node/blob/master/test/common/inspector-helper.js
2. An example with some usage of the protocol - https://github.com/nodejs/node/blob/master/test/common/test/parallel/test-inspector-multisession-ws.js
3. https://github.com/nodejs/node/blob/master/test/common/test/sequential/test-inspector-* and specificaly test/sequential/test-inspector-break-e.js

If you find this information useful, you could submit a PR, as I assume others will find it useful in the future.

P.S. you could also use a MITM proxy and see how DevTools etc. talk to node.

[refack]

Ping @jkrems

[jkrems]

You can run node inspect my-script.js with debug logging. This will print all communication between the debug client and node:

NODE_DEBUG=inspect node inspect alive.js

The code for the inspect protocol connection can be found here, including the websocket decoding: https://github.com/nodejs/node-inspect/blob/master/lib/internal/inspect_client.js#L60.

The websocket code in the command line debugger is actually reverse engineered from its C++ counterpart here: https://github.com/nodejs/node/blob/8b4af64f50c5e41ce0155716f294c24ccdecad03/src/inspector_socket.cc

[eranif]

Thank you!
I already got the websocket code working, I just wasn't sure what kind of messages the inspector is waiting for. Using the mentioned environment variable, I can see the message exchange clearly.

It's now just matter of replacing the old driver with the new one and I am good to go :)

I will upload a working C++ example of how to connect to Node.js and execute a debug session for other people who might find it useful as I did.

Here is the snippet that allows me to connect to the Node.js and initiate the debugging session (putting here for future / search references for other people):

// No boost or other external libraries, just plain old good C++11
#define ASIO_STANDALONE 1
#define _WEBSOCKETPP_CPP11_THREAD_ 1

#include <websocketpp/client.hpp>
#include <websocketpp/config/asio_no_tls_client.hpp>

#include <iostream>
#include <sstream>

typedef websocketpp::client<websocketpp::config::asio_client> client;

using websocketpp::lib::bind;
using websocketpp::lib::placeholders::_1;
using websocketpp::lib::placeholders::_2;

// pull out the type of messages sent by our config
typedef websocketpp::config::asio_client::message_type::ptr message_ptr;

// This message handler will be invoked once for each incoming message
void on_message(client* c, websocketpp::connection_hdl hdl, message_ptr msg)
{
    std::cout << "on_message called with hdl: " << hdl.lock().get() << " and message: " << msg->get_payload()
              << std::endl;
}

static void send_startup_command(client* c, websocketpp::connection_hdl hdl, const std::string& method)
{
    static int i = 0;
    std::stringstream ss;
    ss << "{\"id\":" << (++i) << ",\"method\":\"" << method << "\"}";
    websocketpp::lib::error_code ec;
    c->send(hdl, ss.str(), websocketpp::frame::opcode::TEXT, ec);
}

void on_open_handler(client* c, websocketpp::connection_hdl hdl)
{
    std::cout << "on_open_handler called" << std::endl;

    // Send upgrade request
    send_startup_command(c, hdl, "Runtime.enable");
    send_startup_command(c, hdl, "Debugger.enable");
    send_startup_command(c, hdl, "Runtime.runIfWaitingForDebugger");
}

int main(int argc, char* argv[])
{
    // Create a client endpoint
    client c;
    if(argc < 2) {
        std::cerr << "Usage: " << argv[0] << " <WebSocker URI>" << std::endl;
        return 1;
    }
    std::string uri = argv[1];

    try {
        // Set logging to be pretty verbose (everything except message payloads)
        c.set_access_channels(websocketpp::log::alevel::all);
        c.clear_access_channels(websocketpp::log::alevel::frame_payload);

        // Initialize ASIO
        c.init_asio();

        // Register our message handler
        c.set_message_handler(bind(&on_message, &c, ::_1, ::_2));
        c.set_open_handler(bind(&on_open_handler, &c, ::_1));

        websocketpp::lib::error_code ec;
        client::connection_ptr con = c.get_connection(uri, ec);
        if(ec) {
            std::cout << "could not create connection because: " << ec.message() << std::endl;
            return 0;
        }

        // Note that connect here only requests a connection. No network messages are
        // exchanged until the event loop starts running in the next line.
        c.connect(con);

        // Start the ASIO io_service run loop
        // this will cause a single connection to be made to the server. c.run()
        // will exit when this connection is closed.
        c.run();
    } catch(websocketpp::exception const& e) {
        std::cout << e.what() << std::endl;
    }
}

[hthomann]

HI! I agree that the DevTools Protocol View docs (https://chromedevtools.github.io/devtools-protocol/v8) leave much to be desired. Don't get me wrong, now that I've gotten over a few hurdles and have been using it, I'm mostly happy with the docs. But as a newbie, it was a challenge. First, an example use case of a web socket client showing how to connect and how to make a few of the basic methods would be good. Second, technical accuracy would be good. I've found some things are returned that aren't documented, and other times things are not returned that are documented. Finally, getting the variable values has not be clear. One would think there would be a dedicated method just to get the variable values associated with a stack. But it seems like I have to call a not-so-obvious Runtime domain method, parse a ton of json and then use some of the values from the json to make more Runtime domain calls. As I keep digging maybe I'll find something obvious. Anyway, I don't want to sound overly negative, overall the docs are good, but going forward more focus on details and accuracy would be good.

[jkrems]

Docs can always be better. :) Though I think that documenting it on the node side is dangerous because the protocol isn't controlled by node. It's a V8 protocol and/or a Chrome protocol. The good news is that you can use the same implementation to allow users to remotely debug a Chrome instance (and some other kinds of processes by now). The bad news is that we're "just" consumers of this protocol and node isn't really where the biggest experts can be found. :)

In general issues with the documentation of the Chrome protocol could be filed here: https://github.com/ChromeDevTools/devtools-protocol/issues

[eranif]

@hthomann I plan on writing a beginners document once I have my implementation done. About the variables: I am still not sure how to get list of all locals + function arguments, but once you have the variable name (or expression) it basically using 2 methods:
Debugger.evaluateOnCallFrame - this will return an object ID
Next you can keep calling Runtime.getProperties with the object ID returned by the previous call (each returned property can be examined by its object-id (incase its an object))
Its actually pretty similar to gdb's MI interface (its called variable-objects in GDB terminology)
For me, the hard part was not connecting via websocket, but actually that I needed to call Runtime.runIfWaitingForDebugger to get things running - so it's basically the lack of documentation that was blocking me (I have experience of writing multiple debugger interfaces lldb, gdb, cdb, xdebug) and I can say with confident that nodes protocol was the hardest to crack :)

[hthomann]

HI! OK, I'm finding a spot where the docs are very much lacking. :) Take a call to Runtime.getProperties. This will return an array or PropertyDescriptor (PD). If we look at PD, it can contain a RemoteObject (RO), where the RO seems to be at the heart of the PD. BUT, RO is listed as optional. It would be nice to know under what conditions it might not exist, and if it does’t exist what we should look at instead.
Next, if there is a RO, and if it contains an 'object', the object might contain a subtype of about 17 different types. It would be nice to know how each type is defined. For example. So far I’ve looked at ‘null’ and ‘array’. Null is an object BUT has not ObjectId, so no further refinement needed. An array however has an objectID so we can use it to dig further to get its values. How am I to know if other subtypes have an ObjectID, or what additional data fields they might have? I suppose I can assume that if a subtype doesn’t contain an objectID I need not do further processing (e.g. Runtime.getProperties)??
If there is a doc (aside from Chrome DevTools Protocol Viewer docs) I’m missing please let me know. Thanks for your time and consideration!!

[eranif]

Thanks for this, however, I have already completed my debugger implementation :)

[Trott]

Though I think that documenting it on the node side is dangerous because the protocol isn't controlled by node.

I think that improved documentation is always desirable, but this isn't Node.js's protocol to document. I'm going to close this. (A pull request putting some of the links above in appropriate places in our docs would be welcome.)

[Trott]

(And even though this is closed, continued relevant comments and questions are fine. Not trying to shut down conversation about the protocol. Just trying to be realistic about whether or not Node.js is going to document it.)

[lostsource]

You can also use built in Protocol Monitor within DevTools while connecting to the NodeJS debugger.

https://twitter.com/ChromeDevTools/status/983768645967818753