Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ustack not symbolicated if traced process exits first #246

Open
ahupp opened this issue Nov 10, 2018 · 2 comments

Comments

@ahupp
Copy link

commented Nov 10, 2018

I have process that allocates once and then sleeps:

#include <unistd.h>
#include <stdlib.h>

int main() {
  void* a = malloc(2);
  sleep(1000);
  return 0;
}

And I trace it with this command:

sudo bpftrace -e 'uprobe:/lib64/libc.so.6:malloc /comm=="test_malloc"/ { @[ustack] = count(); }'

If I exit test_malloc first before exiting bpftrace, I get unsymbolicated output:

@[
0x7fc578a18800
0x7fc5789b5445
]: 2

But if I exit bpftrace first it is symbolicated:

@[
__libc_malloc+0
__libc_start_main+245
]: 2

So it seems like the symbolication step depends on having the process running. It would be convenient for the things I'm doing to have it symbolicate regardless of whether the process has exited.

@ahupp ahupp changed the title ustack not symbolicated if traced process exists first ustack not symbolicated if traced process exits first Nov 10, 2018

@tyroguru

This comment has been minimized.

Copy link
Contributor

commented Nov 10, 2018

I've just taken a quick look at this and we don't appear to attempt to resolve the stacks until bpftrace exits in this case (at least that's how it appears under a debugger). If I'm interpreting the code correctly then this would be the wrong thing to do. We should try and resolve symbols as soon as we are handed the array of PCs.

Maybe someone who understands this code better will chip in but I'll take a look next week if nobody does in the meantime.

@mmarchini

This comment has been minimized.

Copy link
Member

commented Feb 25, 2019

We should try and resolve symbols as soon as we are handed the array of PCs.

As @tjfontaine said in #286, if we try to resolve symbols too soon we'll introduce extra overhead into the traced application. I guess we could try to resolve symbols in a separate bpftrace thread, but even then there might be some unforeseen overhead.

What we can do is to keep the memory mapped information of traced processes cached in bpftrace, and use this cached information if the process has exited when we're resolving symbols. This might be more laborious than it sounds though, since symbol resolution is implemented in bcc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.