Debugger breakpoints using incorrect URL for a remote debugging session and therefore failing to pause the process

[treffynnon]

Symptoms

This is related to the closed ticket #99, which describes the symptoms of this very problem.

You set a breakpoint, but the process isn't paused when that breakpoint is encountered and carries merrily on.

Environment

This is a remote debugging session between a Windows client (running Chrome devtools) and a remote Linux server (running node with --inspect=0.0.0.0).

What's wrong

By watching the websocket communication with Wireshark it is apparent that the Chrome Dev tools are using an incorrect filepath/URL when calling Debugger.setBreakpointByUrl.

What I am seeing

Request

{
  "id": 17,
  "method": "Debugger.setBreakpointByUrl",
  "params": {
    "lineNumber": 67,
    "url": "EFS\\DEV-efsbastion\\username\\application\\projects\\service\\module\\index.js",
    "columnNumber": 18,
    "condition": ""
  }
}

Response

{
  "id": 17,
  "result": {
    "breakpointId": "EFS\\DEV-efsbastion\\username\\application\\projects\\service\\module\\index.js:67:18",
    "locations": []
  }
}

Note the empty locations array. I am no node debugger expert, but it feels like this locations array should be checked to ensure it is not empty after setting a new breakpoint marker before then showing the marker as active in the UI.

What I should be seeing

Request

{
  "id": 17,
  "method": "Debugger.setBreakpointByUrl",
  "params": {
    "lineNumber": 67,
    "url": "/EFS/DEV-efsbastion/username/application/projects/service/module/index.js",
    "columnNumber": 18,
    "condition": ""
  }
}

Response

{
  "id": 17,
  "result": {
    "breakpointId": "/EFS/DEV-efsbastion/username/application/projects/service/module/index.js:67:18",
    "locations": [{
      "scriptId":"104",
      "lineNumber":67,
      "columnNumber":18
    }]
  }
}

Note the Linux path separators instead of Windows and the initial root slash on the path.

But isn't this a node bug?

No, it is not from what I can tell because of the following two discoveries.

PHPStorm

PHPStorm/WebStorm debugger continues to work just fine using it's URL regexes

Request

{
  "id": 17,
  "method": "Debugger.setBreakpointByUrl",
  "params": {
    "lineNumber": 67,
    "urlRegex": "[/\\\\][iI][nN][dD][eE][xX]\\.[jJ][sS]([;?#!].*)?$"
  }
}

Response

{
  "id": 17,
  "result": {
    "breakpointId": "/[/\\\\][iI][nN][dD][eE][xX]\\.[jJ][sS]([;?#!].*)?$/:67:18",
    "locations": [{
      "scriptId":"104",
      "lineNumber":67,
      "columnNumber":18
    }]
  }
}

Manually recreating the request successfully

If I manually set the breakpoint myself I see a success response back over the websocket. For example consider the following code run from the Chrome console:

let id, key, socket

id = 1000
key = '<debugger-key-goes-here>'

const connect = userSuppliedKey => {
	socket = new WebSocket(`ws://<ip-address-goes-here>:9229/${userSuppliedKey || key}`)
	socket.addEventListener('message', e => {
		const d = JSON.parse(e.data)
		if (d) {
			if (d.method === 'Debugger.scriptParsed') return console.log('scriptParsed', JSON.stringify({ scriptId: d.params.scriptId, url: d.params.url }))
			if (d.method === 'Debugger.breakpointResolved') return console.log('breakpointResolved', JSON.stringify(Object.assign({ breakpointId: d.params.breakpointId }, d.params.location)))
		}
		console.info(`${e.data}`)
	})
}
const send = (method, x) => socket.send(JSON.stringify({ id: ++id, method: `Debugger.${method}`, params: x }))
const _getPossibleBps = (start, end) => send('getPossibleBreakpoints', { start, end })
const getPossibleBps = (scriptId, start, end) => _getPossibleBps({ scriptId, lineNumber: start }, { scriptId, lineNumber: end })
const setBp = (lineNumber, url) => send('setBreakpointByUrl', { lineNumber, url, })
const removeBp = (lineNumber, url, columnNumber) => send('removeBreakpoint', { breakpointId: `${url}:${lineNumber}:${columnNumber || 0}` })
const enable = () => send('enable')

Which gives us this when run in the Chrome console

connect('<debugger-key-goes-here>')
// '<debugger-key-goes-here>'

enable()
// ...
// scriptParsed {"scriptId":"104","url":"/EFS/DEV-efsbastion/username/application/projects/service/module/index.js"}
// ...

getPossibleBps('104', 9, 15)
// {"id":1016,"result":{"locations":[{"scriptId":"104","lineNumber":9,"columnNumber":15},{"scriptId":"104","lineNumber":11,"columnNumber":15},{"scriptId":"104","lineNumber":12,"columnNumber":16},{"scriptId":"104","lineNumber":14,"columnNumber":4},{"scriptId":"104","lineNumber":15,"columnNumber":0},{"scriptId":"104","lineNumber":17,"columnNumber":4},{"scriptId":"104","lineNumber":18,"columnNumber":14}]}}

setBp(11, '/EFS/DEV-efsbastion/username/application/projects/service/module/index.js')
// {"id":1024,"result":{"breakpointId":"/EFS/DEV-efsbastion/username/application/projects/service/module/index.js:11:0","locations":[{"scriptId":"104","lineNumber":11,"columnNumber":15}]}}

removeBp(11, '/EFS/DEV-efsbastion/username/application/projects/service/module/index.js')
// {"id":1019,"result":{}}

Possible introduction of the issue

https://chromium.googlesource.com/chromium/src/+/2140ffeadf3e081e2e0821fc8e1ee040373b205d%5E%21/third_party/WebKit/Source/devtools/front_end/sdk/DebuggerModel.js

in combination with https://chromium.googlesource.com/chromium/src/+blame/master/third_party/blink/renderer/devtools/front_end/common/ParsedURL.js#103

[treffynnon]

Also referenced on https://bugs.chromium.org/p/chromium/issues/detail?id=841697

[alexkozy]

Thanks for great report!

Inside V8 on inspector backend side we try to match breakpoint url and script url. In this case we send incorrect url from frontend side and when we receive new parsed script on backend side we do not set breakpoint properly.

I will do following two points next week:

• check why we send incorrect url from frontend side,
• consider some kind of breakpoint url normalization on backend side to make our protocol a little more flexible.

I will update this issue as soon as I get some results.

(PhpStorm actually uses great regexp)

[mathieulj]

Just to corroborate that this is due to a change on the chrome side and provide a workaround. We experienced this recently running windows workstations with a linux development VM. We had 3 versions of chrome(64, 66, 68) connecting one at a time to a debug session and only 64 was working normally. This was true for node 6, 8 and 10.

As a workaround for those who need to debug, we found that using the id and manually visiting the link chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=${ip}:${port}/${id} works as expected even in the newer chrome versions. If, for example, you get the message Debugger listening on ws://0.0.0.0:3020/4dacc23a-f296-4b6d-bb47-18af21dfef67 during inspect start, the URL would be chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=${ip}:3020/4dacc23a-f296-4b6d-bb47-18af21dfef67 where ip is the ip of the remote machine. We also found that the plugin nodejs-v8-inspector-manag(https://chrome.google.com/webstore/detail/nodejs-v8-inspector-manag/gnhhdgbaldcilmgcpfddgdbkhjohddkj) seems to still work.

[brucejo75]

@ak239

Also see: https://bugs.chromium.org/p/chromium/issues/detail?id=844070
& meteor/meteor#9875

[tonymet]

I'm experiencing the lack-of-breakpoints with Windows 10 + WSL (aka Ubuntu Bash for Windows) + Node 8.11.3

@mathieulj 's workaround by manually building the chrome-devtools:// URL works for me.

Thanks for the help guys this was driving me bonkers for a while :D

Node v8.11.3
Windows 10 PRo 1803
Version 67.0.3396.87 (Official Build) (64-bit)

[tonymet]

Also mentioned in the chromium bug, Chrome Extensions "Inspector Manager" is a simpler workaround for now

https://chrome.google.com/webstore/detail/nodejs-v8-inspector-manag/gnhhdgbaldcilmgcpfddgdbkhjohddkj

[treffynnon]

@ak239 is there any progress being made with this issue?

Unfortunately, [Web/PHP]Storm has it's own issues with simply choosing the first file it sees with the name the breakpoint is set against. In the case of files named index.js this is an issue as so many modules use this filename that the breakpoint often ends up being set against the wrong file.

[alexkozy]

I landed following CL recently. It should fix provisional breakpoints in Node.js targets. At least it helps ndb.
I have another CL to land that fixes breakpoints in experimental Node.js modules: CL. I am going to land it soon.
It would be nice if someone could verify that issue is fixed in latest Canary.

[treffynnon]

I downloaded the canary with version: Version 70.0.3508.0 (Official Build) canary (64-bit) and it appears to still have the same problem if that helps.

[mathieulj]

It seems to have gotten worse with the release of chrome 69. The workarounds I posted above no longer work with chrome 69.0.3497.81(stable) and with chrome 71.0.3548.0 (canary). Workstations still running 68.0.3440.106 do work with both workarounds.

[mathieulj]

For those with a Webstorm licence, version 2018.2.3 seems to break correctly on breakpoints.

[treffynnon]

For those without a WebStorm licence you can use the Early Access Programme (EAP) builds for free with the following notice:

This is an early access version of the product. You expressly acknowledge that this version of the product may not be reliable, may not work as intended and may contain errors. Any use of the EAP product is at your own risk.

-- JetBrains

_{NOTE: I am not affiliated with JetBrains}

[alexkozy]

The root of this issue is using platform-specific file paths in our protocol as was mentioned before. I landed V8 patch and close to merging Node patch that will migrate Node starting Node 11 to file urls: nodejs/node#22251.
At the same time I am working on better solution for remote debugging as part of ndb project. I will update this issue as soon as it is ready for Node 8 and Node 10.

[alexkozy]

fileURL patches were merged to Node 10. So I expect this issue to be fixed in next Node 10 release, I will update this bug when release happens.

[GummyDonut]

@ak239 Will this be updated for node 8 at well? As I just recently encountered the bug after Chrome updated XD

[june07]

The most recent version of NiM actually does work as a fix to this. Different DevTools versions can be selected along with a custom version as the video shows. https://youtu.be/hjf6esnqOJQ

So in essence when using NiM it doesn't matter which version of Node or Chrome you use.

https://june07.com/nim

[sdd]

@june07 ?I'm using NiM 1.1.0, Node 10.12 (remote, linux), Chrome (69, windows) and I still get this problem

[june07]

You need to change the DevTools version under NiM settings...

…

On Fri, Oct 12, 2018, 8:41 AM Scott Donnelly ***@***.***> wrote: @june07 <https://github.com/june07> ?I'm using NiM 1.1.0, Node 10.12 (remote, linux), Chrome (69, windows) and I still get this problem — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#103 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK099hKyPKXOBT0d5avdzEfaKwUHigwkks5ukLgagaJpZM4UNjlM> .

[sdd]

@june07 thanks for the quick response.

I'v tried changing NiM settings but it is not clear what to use for the value of custom devtools.

I have tried a few values but none work, and now if I try to turn off "use alternate devtools", it does not open an inspector at all.

[alexkozy]

@sdd, could you please try DevTools frontend in Canary with Node 10.12?

[smeijer]

For those with a Webstorm licence, version 2018.2.3 seems to break correctly on breakpoints.

Strange, just upgraded from 2018.1 to 2018.2, but doesn't change anything here. Also tried the 2018.3, but still not hitting any breakpoint.

Using NiM turned out to work, but it doesn't work as sweet as the Webstorm debugger.

[june07]

I've just tried NiM 1.1.0 with Node 10.12 and Version 69.0.3497.100 (Official Build) (64-bit) and can confirm that: https://chrome-devtools-frontend.appspot.com/serve_file/@a10b9cedb40738cb152f8148ddab4891df876959/inspector.html works. Which is the appspot option. [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality6&> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality6&> 10/12/18, 12:31:15 PM

…

On Fri, Oct 12, 2018 at 12:24 PM Stephan Meijer ***@***.***> wrote: For those with a Webstorm licence, version 2018.2.3 seems to break correctly on breakpoints. Strange, just upgraded from 2018.1 to 2018.2, but doesn't change anything here. Also tried the 2018.3, but still not hitting any breakpoint. Using NiM turned out to work, but it doesn't work as sweet as the Webstorm debugger. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#103 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AK099oAGrx5HcpX-hvgBJjehy6nbJkGcks5ukOx1gaJpZM4UNjlM> .

[smeijer]

Okay, let me rephrase that. WebStorm does kind of work. Some breakpoints can be hit, other ones not at all. And others don't seem to be hit, but my frontend does block. So node is pausing and waiting for the debugger, but I don't have control over it.

So I basically have 3 types of breakpoints.

1. The ones that function as expected
2. Those that cannot be hit
3. Those that are hit, but not given control over?

[brucejo75]

@ak239, Should I assume that Node 8 will not be supported?

I've just tried NiM 1.1.0 with devtools set to: https://chrome-devtools-frontend.appspot.com/serve_file/@a10b9cedb40738cb152f8148ddab4891df876959/inspector.html

on Node 8.11.3 and Version 69.0.3497.100
(Official Build) (64-bit)
Result: breakpoints are not hit.

[alexkozy]

I definitely would like to get if fixed, but I need your help.
@brucejo75, could you provide more details about your setup, I assume that you use Windows and can reproduce this bug with any node script with node 8? (I tried on mac and it works).
Could you confirm that with latest node 10.12 it works to you? And that it does not work with dedicated frontend from latest Google Chrome Canary?

[june07]

@brucejo75

https://youtu.be/ZfpZ3wGvEm4

I literally just setup my environment to match yours and confirmed again that things are working with NiM. Maybe you'll see some changes between our environments from the video?

[alexkozy]

Could we focus on fixing DevTools and DevTools protocol in this issue please? A lot of workarounds were already mentioned and I believe that now we can focus on finding solution for root issue.

[brucejo75]

@june07,

I had the devtools set correctly but failed to turn them on, D'oh.

When I use it now, I notice that the debug console uses v66 of chrome and all works as expected.

[brucejo75]

@ak239,

Looks like canary (v72) fixes the problem! (at least for me).

Environment

Windows 10: Version 1803 (OS Build 17134.345)
node: 8.11.3 & 10.12 (using WSL)
chrome: 69.0.3497.100 & 72.0.3580.0 (Official Build) canary (64-bit)

Note: I am running WSL on windows, which is the Ubuntu linux subsystem.

node script index.js:

1 console.log('hello World');
2
3 console.log('Hit 1?');
4 debugger;
5 console.log('Hit 2?');

Repro steps

1. node --inspect-brk index.js
2. in chrome browser go to chrome://inspect, click on Open dedicated DevTools for Node
3. Set breakpoint on Line 3.
4. in dedicated DevTools, resume script execution (e.g. hit F8)

Fails with chrome 69, tested with node 8.11.3 & node 10.12

Result: debugger stops on line 4, missing breakpoint at line 3.

Workaround with NIM tools works because essentially it uses v66 of the devTools.

Verified fixed with chrome 72, tested with node 8.11.3 & node 10.12

Result: debugger stops on breakpoint at line 3
Yay!

[june07]

@june07,

I had the devtools set correctly but failed to turn them on, D'oh.

When I use it now, I notice that the debug console uses v66 of chrome and all works as expected.

Actually updated NiM (1.2.0) to use the most up to date version of DevTools (Chrome 71) as well as adding info on how to better use the custom setting https://june07.com/blog/nim-custom-devtools-url.

@ak239 This will be my last post to this thread. Cheers.

[brucejo75]

Ignore?

This may be expected behavior. Before I can set a breakpoint on a required file, the file has to be loaded. Sorry for the clutter.

Entered issue

Oops! spoke too soon. devTools fails to stop on a breakpoint in a required module.
@ak239, fails with Canary :-(

Environment

Windows 10: Version 1803 (OS Build 17134.345)
node: 8.11.3 (using WSL)
chrome: 72.0.3580.0 (Official Build) canary (64-bit)

Note: I am running WSL on windows, which is the Ubuntu linux subsystem.

node script index.js:

1 let foo = require('./foo');
2 console.log('hello World');
3 foo();

node script foo.js

1 function foo() {
2   console.log('Hit 1?');
3   debugger;
4   console.log('Hit 2?');
5 }
6
7 module.exports = foo;

Repro steps

1. node --inspect-brk index.js
2. in chrome browser go to chrome://inspect, click on Open dedicated DevTools for Node
3. Set breakpoint on foo.js Line 2.
4. in dedicated devTools, resume script execution (e.g. hit F8)

Result: devTools fails to stop on the breakpoint, stops on foo.js line 3.

Workaround

1. node --inspect-brk index.js
2. in chrome browser go to chrome://inspect, click on Open dedicated DevTools for Node
3. set breakpoint on index.js line 2
4. set breakpoint on foo.js line 2.
5. in dedicated devTools, resume script execution (e.g. hit F8)

Result: devTools will now stop on both breakpoints.

[cronvel]

On Chrome/Chromium v69, UI breakpoints (not debugger statement) doesn't work anymore since node v10.
It was working on node v8 (and I confirm it's still working when I downgrade). I believe it's a bug on the devTools side nonetheless. Also in Node v10, the devTools doesn't skip Node.js source code anymore, which is pretty annoying...

[brucejo75]

I verified that Chrome release version: Version 71.0.3578.80 (Official Build) (64-bit) now has this repaired. Maybe should be closed?

[daniele-orlando]

I confirm that Chrome 71.0.3578.98 works fine.

[louderspace]

I'm using v8 8.0.426 and am still seeing this issue with the latest Chrome and VS Code. Installing NiM did fix the problem on the mac, but not on windows (though, I'm not convinced NiM is correctly installed on windows because I'm not seeing the syntax coloring). I did verify that all of my paths are correctly specified within the v8::ScriptOrigin.

Here's an example:

Ticks = 23110 <17356> <5> messageFromDebugger : /1, payload : {"id":17,"method":"Debugger.setBreakpointByUrl","params":{"urlRegex":"[Dd]:\\dev\\apd\\<...redacted...>\.js|file:\/\/\/[Dd]:\/dev\/apd\/<...redacted...>\.js","lineNumber":430,"columnNumber":13}}

[june07]

What version of DevTools, NiM, and Node.js are you running on Windows?

[louderspace]

Node.js v12.16.3
NiM 2.3.4
Google Chrome | 81.0.4044.138
VS Code 1.45.1

Note, I've embedded v8 into a desktop application directly.

Updating Node.js yesterday helped resolve one of my problems on windows, but I'm still seeing the same problem within VSCode.

In the failure case, I'm unable to step into some functions that are in other .js files.

[june07]

Well regarding

(though, I'm not convinced NiM is correctly installed on windows because I'm not seeing the syntax coloring)

I can confirm running with the same versions as you that syntax coloring is working in both Chrome 81.0.4044.138 and Edge Version 83.0.478.28 (using both currently).

If that is still an issue you may want to double check your actual DevTools version, as in the case of NiM it's possible you have it set several different ways.:

Of course this same point could also very well affect your not being able to step into some functions, as that is a function of the DevTools debugger.

[louderspace]

I'm getting a new machine today and will try setting it up again from scratch. Here's what it currently looks like.

[alexkozy]

@louderspace If you are embedding V8 directly, your ScriptOrigin might contain properly formed file URLs like file://... - in this case, everything should work out of the box.

The solution that was implemented on the DevTools side only works for Node.js. So how do you start DevTools to debug your embedded V8?

[louderspace]

I am setting the script origins:

Added tracing shows "SetScriptPath = D:\dev\p4<redacted>\nml\nml_include.js

Should this be in the form file://dev/... ?

[louderspace]

I tried various flavors and haven't had any success:

CompileAndRun script : file:///D:/dev/p4/nml/nml_include.js

I do have NiM working on windows now and running Chrome 8.3