Feature request: run rr under cgdb

[dkasak]

Is it possible to run rr under the cgdb TUI?

I know about rr gdbinit and gdb's -ix argument so I tried passing rr's setup script like that, and it starts and seemingly configures cgdb successfully. However, then I've never called rr replay and I don't know how to do it from inside gdb and whether it is even possible.

It seems the support for this is already almost there so it would be nice to support it explicitly, maybe through a rr flag.

[rocallahan]

I don't know anything about cgdb off the top of my head. Some rr features that may be useful:

• rr replay -d <debugger> will spawn <debugger> instead of gdb. rr expects it to support the same command-line parameters as gdb.
• rr replay -s <port> will spawn rr listening on a specific port. You can then use target remote :<port> in gdb to connect to that rr instance.

[dkasak]

These don't appear to work correctly so this may be more complex than I initially thought.

Some observations:

• rr replay -c cgdb starts the cgdb TUI successfully, but the programs output is displayed in its upper pane where the source code should be displayed and destroys it. The program's output should be echoed in the lower pane, like in gdb. Running rr replay -c cgdb --no-redirect-output works around it, but then no output is displayed, of course.

• Running the program seems to work, but the backtrace is then wrong, mentioning librrpreload.so:

  (rr) bt
  #0  0x0000000070000002 in ?? ()
  #1  0x00007f7d751f3158 in ?? () from /usr/bin/../lib/librrpreload.so
  #2  0x00007f7d751f005d in ?? () from /usr/bin/../lib/librrpreload.so
  #3  0x00007f7d751f10a0 in ?? () from /usr/bin/../lib/librrpreload.so
  #4  0x00007f7d751f318a in ?? () from /usr/bin/../lib/librrpreload.so
  Backtrace stopped: previous frame identical to this frame (corrupt stack?)
  

• rr replay -s <port> followed by connecting cgdb to the port manually displays similar behaviour.

• cgdb should support all gdb arguments, intercepting only --help, --version and -d (but not even that if they are passed after --).

EDIT: I meant rr replay -d cgdb above, the -c was a typo.

[rocallahan]

rr replay -d cgdb (not -c) mostly works for me. What doesn't work, specifically, is that when the debugee produces a line of output, it gets echoed to the gdb pane, but the whole screen scrolls down one line as well so the source code moves up one line and isn't updated correctly. I'm not sure if that's what you're seeing or not.

[rocallahan]

(Resizing the window fixes everything.)

[dkasak]

The -c was a typo in my comment, I used -d when running it. It is still functionally almost completely broken for me, though. I am only able to run the program successfully, with most of the other commands seeming broken. For instance, running next after the program initially stops at _start will immediately run the program to the very end, terminating with SIGKILL. The backtrace is initially

(rr) bt
#0  0x00007f97c739dd70 in _start () from /lib64/ld-linux-x86-64.so.2
#1  0x0000000000000003 in ?? ()
#2  0x00007ffda36ae59b in ?? ()
#3  0x00007ffda36ae5a1 in ?? ()
#4  0x00007ffda36ae5a4 in ?? ()
#5  0x0000000000000000 in ?? ()

but after entering the next command, it becomes

(rr) bt
#0  0x0000000070000002 in ?? ()
#1  0x00007f97c7174158 in ?? () from /usr/bin/../lib/librrpreload.so
#2  0x00007f97c717105d in ?? () from /usr/bin/../lib/librrpreload.so
#3  0x00007f97c71720a0 in ?? () from /usr/bin/../lib/librrpreload.so
#4  0x00007f97c717418a in ?? () from /usr/bin/../lib/librrpreload.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Trying to run reverse-next at that point gives me

(rr) reverse-next
Cannot find bounds of current function

If I enter step instead of next when the program is stopped at _start, I get the following

(rr) step
Single stepping until exit from function _start,
which has no line number information.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?

The program then seems to stall for a long time until I press C-c, at which point lines like this begin to be printed

Reply contains invalid hex digit 84
warning: Invalid remote reply: 48
Reply contains invalid hex digit 84
warning: Invalid remote reply: 4c
Reply contains invalid hex digit 84

Pressing C-c a few more times will eventually output something like this

Program stopped.
0x00007f97c73ad6c5 in match_symbol () from /lib64/ld-linux-x86-64.so.2

But the particular function it stops in will be unpredictable. At this point, next will work but with the effect of running until the end of the current function. Eventually it reaches

Single stepping until exit from function _start,
which has no line number information.
0x00007f97c66371a0 in __libc_start_main () from /usr/lib/libc.so.6

and running another next at this point will run the debugee until the end, like before.

Regarding the output issue, yes, that is what appears to be happening. My debugee produced a lot of output, so the UI seemed to be completely broken at first. Resizing does fix it.

[rocallahan]

Try break main then c. That should get you into a good state. It's true that _start is not a useful place to start debugging from, but gdb has the same issue for me.

[dkasak]

You're right, that does fix it. It's surprising that gdb behaves exactly the same for everything I described above. I could have sworn it did not, but I guess I must have always set a breakpoint somewhere in the program before running it, when doing real debugging.

Do you have an idea what causes output to be echoed in the upper pane? I'd love to fix this since then it would be perfect, though it's already usable with the resizing trick.

[rocallahan]

When you run cgdb without rr, it uses gdb's tty command to switch output to a special pty set up by cgdb for this purpose. That's how cgdb takes control of the debugee's output. When gdb uses a remote target, which it does with rr, this isn't supported. E.g. if you set up a remote session with gdbserver, all program output is echoed to gdbserver's tty even after you connect with cgdb/gdb. I'm pretty sure there's no remote protocol support for changing this.

[rocallahan]

So I think the options are

• rr replay -d cgdb and deal with the debugee output interfering with cgdb's interface
• rr replay -q -d cgdb and live without any debuggee output
• rr replay -s <port> in one window and then running cgdb in another window, with the debugee output appearing in the former window. We can improve this option a little bit by having rr replay -s <port> -d cgdb replace gdb with cgdb in its example Launch gdb with command line.

[rocallahan]

Done that in 66118a3. Just a minor tweak.

[dkasak]

That sounds quite reasonable and having the output in a separate window might even be nicer than in the same window. Also, thanks for the nice UI improvement! I think this can be closed.

[jrmuizel]

It seems like when using 'rr replay -s ' you don't get rr's overrides of commands like 'checkpoint'. Is there a way to fix that?

[rocallahan]

Yes. Pipe the output of rr gdbinit to a file and load it when you start gdb by passing the parameters -x <file>.

[NickHu]

When I use rr replay -d cgdb, it only displays assembly code in the top pane. Running cgdb on the binary works ok, as does the list source code l command within rr replay -d cgdb.