gdb thread number changes, when replaying thread_create

[roquo]

Today I debugged an application with multiple threads, with a bug in a code segment which gets called from almost every thread in the application, but only one thread triggers a bug.
While replaying, I saw the thread number shown in gdb changes everytime a thread gets created. This behavior really messed up with me, because while running back and forward I only looked at the thread number, not the real tid.
Some code to explain the behavior:

void foo(){ 
    printf("Thread!\n");
}

int main() {
    pthread_t p;
    int r = pthread_create(&p, NULL, &foo, NULL);
    if(r!=0)
      return -1;
    pthread_join(p, NULL);
    printf("Finish!\n");
    return 0;
}

Is it possible to change this behavior in rr, that the thread number stays the same while debugging?

[rocallahan]

Interesting. rr doesn't control that thread number directly. That's something gdb makes up. As you can see, the tid numbers reported by rr are correct and remain stable. I suspect gdb assigns new a new thread number because we go through a state where thread 2 doesn't exist, and then it does again, so gdb treats it as a new thread. I don't see any obvious way to avoid this.

[roquo]

For testing I implemented a function in gdb set-thread-id <source-id> <destination-id> .
This function can change the given thread numbers in gdb, if the source-id exists and the destination-id does not exist.
Example:

(rr) b foo
Breakpoint 1 at 0x4005f1
(rr) c
Continuing.
[New Thread 94800.94801]
[Switching to Thread 94800.94801]

Thread 2 hit Breakpoint 1, NameError: Installation error: gdb.execute_unwinders function is missing

0x00000000004005f1 in foo ()
(rr) info threads
  Id   Target Id                                  Frame 
  1    Thread 94800.94800 (mmap_hardlink_3_a.out) NameError: Installation error: gdb.execute_unwinders function is missing

0x00007fe05d061f47 in pthread_join () from /lib64/libpthread.so.0
* 2    Thread 94800.94801 (mmap_hardlink_3_a.out) NameError: Installation error: gdb.execute_unwinders function is missing

0x00000000004005f1 in foo ()
(rr) rc
Continuing.

Thread 1 stopped.
[Switching to Thread 94800.94800]
NameError: Installation error: gdb.execute_unwinders function is missing
0x00007fe05d4a2ff0 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) c
Continuing.
[New Thread 94800.94801]
[Switching to Thread 94800.94801]

Thread 3 hit Breakpoint 1, NameError: Installation error: gdb.execute_unwinders function is missing

0x00000000004005f1 in foo ()
(rr) set-thread-id 3 2
(rr) info threads
  Id   Target Id                            Frame 
  1    Thread 94800.94800 (mmap_hardlink_3) NameError: Installation error: gdb.execute_unwinders function is missing

0x00007fe05d061f47 in pthread_join () from /lib64/libpthread.so.0
* 2    Thread 94800.94801 (mmap_hardlink_3) NameError: Installation error: gdb.execute_unwinders function is missing

0x00000000004005f1 in foo ()
(rr) rc
Continuing.

Thread 1 stopped.
[Switching to Thread 94800.94800]
NameError: Installation error: gdb.execute_unwinders function is missing
0x00007fe05d4a2ff0 in _start () from /lib64/ld-linux-x86-64.so.2
(rr) c
Continuing.
[New Thread 94800.94801]
[Switching to Thread 94800.94801]

Thread 4 hit Breakpoint 1, NameError: Installation error: gdb.execute_unwinders function is missing

0x00000000004005f1 in foo ()
(rr) set-thread-id 4 1
Thread ID 1 already exist. Cannot set it!
(rr) 

If gdb implements this function (it needs still more testing and until now it can not handle thread id's with multiple debugged procceses, though. It was only for testing, what we could do in gdb), then I think, we could use this function to set the "correct" thread ID's while replaying.

My Idea is, that when we are replaying a thread creation, we put the thread number in a map with the thread-id key, so we can track all the threads, and if it already exists, we could set the correct thread ID.
What is your opinion on that?

[rocallahan]

How would you send the set-thread-id command to gdb?

If you're going to patch gdb then I think a better approach would be to have gdb remember the (tid, thread number) pairs associated with exited threads and automatically reuse a thread number if it sees a new thread with the old tid.

[roquo]

I added the ability to gdb, that it tracks over all threads and sets correct number and id, if tid already existed.
Until now everything works fine and I saw no problems. We don't have to change anything in rr. So we can close this issue.
Hoepfully GNU accepts the changes ( I don't think so) so everyone can use this "feature". I am going to host this version of gdb, maybe other rr users are interested in the debugger.
Thanks

[rocallahan]

Did you submit your changes upstream or at least file a bug in gdb Bugzilla? If so, can you post a link here? Thanks!

[roquo]

Bug report: https://sourceware.org/bugzilla/show_bug.cgi?id=23454
gdb repository with changes: https://github.com/roquo/gdb_for_rr
Update: does not work, when rr is ued within eclipse
newest version from the gdb mentioned above works with eclipse, when the script mentioned on Using rr in an IDE is changed to this:

#!/bin/bash
exec rr replay -- "$@" -ex=rr_running_under_eclipse

I sent Sourceware a patch request, maybe they patch the changes in to the gdb

[sjindel-google]

Any luck getting your changes upstreamed? I'm hitting the same problem.

[roquo]

No sorry, the patch I sent was not accepted. You can use the gdb copy from my site, which has the changes but its not an actual gdb version.