Unexpected stoppage on remote debugging connection

[st0012]

Recently I mostly use remote debugging and this trap (https://github.com/ruby/debug/blob/master/lib/debug/server.rb#L229) means the program (usually Rails server) will be stopped upon connection, even with the nonstop flag.

And I'll need to manually skip the first few frames every time the connection is established, which is annoying. One example:

❯ ~/projects/debug/exe/rdbg -A
# No sourcefile available for /Users/st0012/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/puma-5.6.2/lib/puma/single.rb
=>#0    [C] Thread#join at ~/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/puma-5.6.2/lib/puma/single.rb:61
  #1    Puma::Single#run at ~/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/puma-5.6.2/lib/puma/single.rb:61
  # and 16 frames (use `bt' command for all frames)

Stop by SIGURG
(rdbg:remote)

So my questions are:

1. Why does the debugger need to stop for this signal?

2. If it's needed in some cases, can we skip this behavior when nonstop flag is used?

3. Why should the debuggee be paused with signal by itself upon activation? The stoppage is triggered by the debugger server itself (pause source)

[ko1]

Why does the debugger need to stop for this signal?

This behavior is same as gdb -p.
It is useful to analyze what happens on the program.

If it's needed in some cases, can we skip this behavior when nonstop flag is used?

Now non-stop flag is debuggee's flag.
rdbg -An can be more useful to control the behavior?

[st0012]

rdbg -An can be more useful to control the behavior?

I used rdbg -A for demonstrating the issue but usually we use it with VSCode. So ideally it should be set from the debuggee along with nonstop too.

It is useful to analyze what happens on the program.

For the use cases I see (mostly attaching to a Rails server), I don't see much need for that. But it stopping at places like puma's internal (as shown above) is confusing and annoying.

Let me also explain how we use the debugger with our development tooling:

1. Define a server service with debugger enabled:

server: rdbg -O -n --sock-path=$RUBY_DEBUG_SOCK_DIR/ruby-debug-server -- bin/rails s

2. Start the server service for developer server.
3. When we want to add breakpoints, we write debug statements (binding.b) in code and restart the server service. (Most people are still used this way to add bps)
4. Attach to the debuggee from VSCode.

And ideally, the server should run normally until we hit the breakpoint after reloading the page or making a request. But we always end up stopping at this place and then need to click continue multiple times.

So IMO, it'd be great to make it optional and also non-default.

[st0012]

I noticed that when in console mode, it traps SIGINT

=>#0    [C] Thread#join at ~/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/puma-5.6.2/lib/puma/single.rb:61
/Users/st0012/projects/debug/lib/debug/session.rb:1527:in `enter_subsession'
/Users/st0012/projects/debug/lib/debug/session.rb:231:in `process_event'
/Users/st0012/projects/debug/lib/debug/session.rb:194:in `session_server_main'
/Users/st0012/projects/debug/lib/debug/session.rb:164:in `block in activate'
  #1    Puma::Single#run at ~/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/puma-5.6.2/lib/puma/single.rb:61
  # and 16 frames (use `bt' command for all frames)

Stop by SIGINT
#<Proc:0x00000001127363c0 /Users/st0012/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/puma-5.6.2/lib/puma/launcher.rb:486> is registered as SIGINT handler.
`sigint` command execute it.

But the stoppage only happened when I try to exit the debugger, which I think it's what you expect analyze is for?

However in remote debugging case, SIGURG is sent and trapped "right after" the connection happened. I think it may be because SIGURG is for both urgent signal and out-of-band data transferring? In that case, maybe switch to SIGINT will be better?

[st0012]

I think this sentence from gnu manual explains the stoppage:

When a socket finds that out-of-band data are on their way, it sends a SIGURG signal to the owner process or process group of the socket.

So when the VSCode makes a socket connection to the debuggee and sends the first message (greeting), it'll also send trigger SIGURG signal to the debuggee process, which will immediately trigger a debugger stoppage with ThreadClient.current.on_trap. And because at that moment, the program is just the puma server waiting for a request, the program always stops at puma's internal code.

[st0012]

Ah the signal is always triggered by the debugger server itself (pause source). So which signal it uses doesn't matter for this issue.
This behavior seems to be introduced in bb5e5c3

[bfad]

For me, it also seems to lose the attached debugger sometimes on subsequent requests. The server would say "wait for debugger connection..." while the client debugger that was attached would be blocked waiting — even ctrl-C would do nothing.

[ko1]

I understand the scenario on #629 (comment).
If we do "nonstop" at attach, we need to show "Enter Ctrl-C to enter REPL mode" or something like that (now it is not clear only hangs or connected without any messages).

I used rdbg -A for demonstrating the issue but usually we use it with VSCode. So ideally it should be set from the debuggee along with nonstop too.

We can consider separately with 'rdbg -A` and attach on VSCode (or not sure it should be same or not). For example, we can add another setting (like "nonstop": true) in launch.json.

But we always end up stopping at this place and then need to click continue multiple times.

At least, it should be one click.

---

• Stop on attach:
  • We can see what kind of code are working at attach timing. It will help when the program seems hangs up.
  • We can use REPL immediately.
  • We can set breakpoints on running process with clicking on VSCode (and so on).
  • Other debuggers VS, gdb, lldb does this behavior, because maybe they need to setup breakpoints and so on for debugging.
• Nonstop on attach:
• Do not need to enter "c" for continue or continue button on the VSCode, and wait for the breakpoints.
• This is default behavior of "launch" mode on VSCode (do not stop until breakpoints).

Could you check other debugger's (python, go, ...) attach behavior on VSCode?
(or I'll check)

---

There are another aspect: breakpoints (set by command or debugger method) should stop or not when the remote debugger client is not connected.
Now (1) it waits for client connection, but it is possible to (2) skip breakpoints if the client is not connected.
If we set debuggee's -n as nonstop at attach, I think (2) can be applied (I don't say it should be).

[ko1]

So I prefer:

• Attach clients should setup nonstop:
  • rdbg -nA
  • "nonstop": true in launch.json on VSCode attach configuration.

I think the same default behavior with other debuggers is good, so it it is my current consideration.

[st0012]

we need to show "Enter Ctrl-C to enter REPL mode" or something like that

👍 good call

Attach clients should setup nonstop:

I like the client configuration idea, but I think for users it'll be hard to understand the difference between server's nonstop and the client's nonstop. Perhaps a different name will be better, like --wait. It'll mean waiting for breakpoints to stop.

I also hope this behavior can also be configured on the server side, with something like RUBY_DEBUG_CLIENT_CONNECTION_STOP=false.

[ko1]

I checked the following GUI tool to attach and all of them do not stop the debuggee:

• VSCode
  • Go
  • JavaScript
  • Python
  • C
• Visual Studio with C (asm)
• (I can not use attach on RubyMine)

So I changed my mind for nonstop on attaching with DAP.

On the CUI attachment, I confirmed that gcc/lldb/dlv (Golang) do stop at attach, so I remain the current behavior.

[ko1]

On the JavaScript with nodejs, I found interesting that it doesn't stop at debugger statement when client is not attached.
So NodeJS is one example of:

There are another aspect: breakpoints (set by command or debugger method) should stop or not when the remote debugger client is not connected.

[st0012]

With nodejs's default debugger, the CUI doesn't stop at connection. It only stops when it hits the breakpoint.

// test.js
i = 0
while (i >= 0) {
  console.log(i);
  i++;
  if (i >= 3000000) {
    debugger;
  }
}

Terminal1

$ node --inspect test.js

Terminal2

Initially it'll be:

$ node inspect -p <pid>
connecting to 127.0.0.1:9229 ... ok
>

And when it hits the breakpoint, the source will be shown and the program will only stop then:

$ node inspect -p <pid>
connecting to 127.0.0.1:9229 ... ok
break in test.js:6
  4   i++;
  5   if (i >= 3000000) {
> 6     debugger;
  7   }
  8 }
debug> repl
Press Ctrl+C to leave debug repl
> i
3000000
>

So I changed my mind for nonstop on attaching with DAP.

Does that mean DAP won't stop even without -n specified?