New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error finding location for probe (usdt) #328

Open
DavidVentura opened this Issue Jan 9, 2019 · 10 comments

Comments

Projects
None yet
6 participants
@DavidVentura
Copy link

DavidVentura commented Jan 9, 2019

I started playing with bpftrace and bcc tools, but I cannot get bpftrace to work with usdt probes. bcc's trace works fine.

Issue:

$ sudo ./build/src/bpftrace  -e 'usdt:/home/david/git/bpftrace/a.out:testprobe { time("%H:%M:%S  "); printf("%-6d %d\n", pid, arg0); }'
Attaching 1 probe...
Error finding location for probe: usdt:/home/david/git/bpftrace/a.out:testprobe

bcc on the same binary

$ sudo ../../bcc/tools/trace.py   'u:/home/david/git/bpftrace/a.out:testprobe "%d" retval'
PID     TID     COMM            FUNC             -
7442    7442    a.out           testprobe        1547067909
7442    7442    a.out           testprobe        1547067910

example code from here

$ cat test.c                                                                                                                                                                                                                                                                                
#include <sys/sdt.h>                                                                                                                                                                                                                                                                                                         
#include <sys/time.h>                                                                                                                                                                                                                                                                                                        
#include <unistd.h>                                                                                                                                                                                                                                                                                                          
#include <stdio.h>                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                             
long int myclock() {                                                                                                                                                                                                                                                                                                         
    struct timeval tv;                                                                                                                                                                                                                                                                                                       
    gettimeofday(&tv, NULL);                                                                                                                                                                                                                                                                                                 
    //printf("%ld\n", tv.tv_sec);
    DTRACE_PROBE1(test-app, testprobe, tv.tv_sec);
    return tv.tv_sec;
}

int main(int argc, char **argv)
{

    while(1) {
        myclock();
        sleep(1);
    }
    return 0;
}

Using an uretprobe on the function works though

$ sudo ./build/src/bpftrace  -e 'uretprobe:/home/david/git/bpftrace/a.out:myclock { time("%H:%M:%S  "); printf("%-6d %d\n", pid, retval); }'
Attaching 1 probe...
22:13:17  7660   1547068397
22:13:18  7660   1547068398

What am I doing wrong? I am on HEAD on both projects..

@mmarchini mmarchini added the bug label Jan 14, 2019

@mmarchini mmarchini added this to the 1.0 milestone Jan 14, 2019

@FooBarWidget

This comment has been minimized.

Copy link
Contributor

FooBarWidget commented Jan 30, 2019

I have a similar problem, but instead of tracing my own binary I am trying to trace glibc on Ubuntu 18.04. According to the glibc malloc source there should be a probe called memory_arena_new:

 LIBC_PROBE (memory_arena_new, 2, a, size);

Indeed, readelf tells me that it is there:

  stapsdt              0x0000003c       NT_STAPSDT (SystemTap probe descriptors)
    Provider: libc
    Name: memory_arena_new
    Location: 0x0000000000091bd0, Base: 0x00000000001bdd48, Semaphore: 0x0000000000000000
    Arguments: 8@%rdx 8@%rbp

And yet:

$ sudo bpftrace -e 'usdt:/lib/x86_64-linux-gnu/libc-2.27.so:memory_arena_new { printf("memory arena new"); }'
Attaching 1 probe...
Error finding location for probe: usdt:/lib/x86_64-linux-gnu/libc-2.27.so:memory_arena_new

Am I doing something wrong? bpftrace commit ccac69c.

@FooBarWidget

This comment has been minimized.

Copy link
Contributor

FooBarWidget commented Jan 30, 2019

@DavidVentura After debugging bpftrace's source code, I found out why your example doesn't work. Your provide's provider name is 'test-app'. However, bpftrace looks for a probe whose provider name equals that of the binary, i.e. bpftrace is looking for your 'testprobe' under the provider name 'a.out'. This is apparent from attached_probe.cpp AttachedProbe::attach_usdt:

std::string provider_name = GetProviderFromPath(probe_.path);
err = bcc_usdt_get_location(ctx, provider_name.c_str(), probe_.attach_point.c_str(), 0, &loc);
if (err)
    throw std::runtime_error("Error finding location for probe: " + probe_.name);

I fixed your issue by renaming your binary so that it matches the provider name (you could also do it the other way around, i.e. renaming the provider name to the binary's name).

In my case, the provider name is "libc". I fixed my problem my symlinking /lib/x86_64-linux-gnu/libc-2.27.so to /tmp/libc and then attaching the usdt probe to /tmp/libc instead.

This was not at all obvious. I think bpftrace should provide a way to specify the provider name, and making it more clear in which provider namespace it searched for a probe.

@dalehamel

This comment has been minimized.

Copy link
Contributor

dalehamel commented Feb 11, 2019

I have also run into this issue recently, and came up with similar conclusions to @FooBarWidget.

A more forgiving / flexible behavior might be to check for a given probe name against all providers, and only request that the provider be specified if there is duplication. This is less explicit though, and I could see there being issues with this.

I think that being able to manually specify the provider name, rather than depending on it being the basename of the path provided, is probably the best way forward here, as it isn't unreasonable to require the user to explicitly indicate the provider name. In this approach it would be harder to maintain backwards compatibility. One option would be to default to the current approach with a deprecation warning if no provider name is specified, and then deprecate in a future release.

So for example, where I currently write a probe as:

bpftrace -p PID -e 'usdt:/usr/bin/ruby:method__entry { printf("here\n"); }

Or, generically:

usdt:PATH:PROBE, which provider being inferred from the final segment of the path.

The new interface would be:

bpftrace -p PID -e 'usdt:/usr/bin/ruby:ruby:method__entry { printf("here\n"); } (explicitly indicating the ruby provider, even though it's the same as the final path segment in this case)

Or, generically:

usdt:PATH:PROVIDER:PROBE

I'm not sure if this would be an issue for how probes are generally handled though (having 4 arguments instead of 3). I'll need to check the code for that.

@brendangregg

This comment has been minimized.

Copy link
Member

brendangregg commented Feb 11, 2019

Thanks for the debugging (I just ran into this too). So how about:

bpftrace -p PID -e 'usdt:/usr/bin/ruby:method__entry { printf("here\n"); }

would instrument ruby:method__entry

bpftrace -p PID -e 'usdt:/usr/bin/ruby:hello:method__entry { printf("here\n"); }

would instrument hello:method__entry

ie, if the provider name is not provided, it defaults to the binary name. Sound ok?

@mmarchini

This comment has been minimized.

Copy link
Member

mmarchini commented Feb 11, 2019

Sounds good to me.

@brendangregg

This comment has been minimized.

Copy link
Member

brendangregg commented Feb 11, 2019

Anyone already made a start at coding it? :-)

@dalehamel

This comment has been minimized.

Copy link
Contributor

dalehamel commented Feb 11, 2019

ie, if the provider name is not provided, it defaults to the binary name. Sound ok?

i like that suggestion @brendangregg

Anyone already made a start at coding it? :-)

Not quite, but I did start poking around the code to see how this works and I think I have a rough idea of what needs to be done.

At the surface, it looks like the changes would just be to AttachPoint and a few related files:

https://github.com/iovisor/bpftrace/blob/master/src/ast/ast.h#L205-L220
https://github.com/iovisor/bpftrace/blob/master/src/parser.yy#L145-L150

And probably a couple of other minor changes elsewhere related to the conditional handling of the provider name. I'll keep pulling at this thread and see if i can come up with a rough cut for a PR.

@brendangregg

This comment has been minimized.

Copy link
Member

brendangregg commented Feb 12, 2019

This also needs to update bcc_usdt_get_argument() in ast/irbuilderbpf.cpp.

dalehamel added a commit to dalehamel/bpftrace that referenced this issue Feb 12, 2019

Fix for bpftrace iovisor#328
Allows specifying a provider namespace for USDT probes

@dalehamel dalehamel referenced this issue Feb 12, 2019

Merged

Usdt specify provider ns #413

2 of 2 tasks complete
@dalehamel

This comment has been minimized.

Copy link
Contributor

dalehamel commented Feb 12, 2019

I have #413 up and ready for review

@williangaspar

This comment has been minimized.

Copy link
Member

williangaspar commented Feb 19, 2019

did #413 solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment