libprobe: Add a probe! macro for static instrumentation

[cuviper]

This adds a new libprobe with a probe! macro, for adding static
instrumentation to any rust crate. Please see the docstrings for simple
probe examples and how they might be used by tools.

This initial version only implements SystemTap SDT, with a voided macro
for other platforms, so probes can be freely written anywhere. I
believe the macro is generic enough that a DTrace USDT probe! could be
implemented for freebsd and macos too.

---

This is related to issue #6816 requesting DTrace USDT, although I only implemented SystemTap SDT. I enabled this for both Linux and Android target_os, although I only tested the former. Since it's simply void everywhere else, it's safe to insert probes in any code.

Please let me know if anything needs to change, and I'm also happy to answer questions about debugging with SDT. Thanks!

[alexcrichton]

Wouldn't this have to opt-in to the asm feature as well?

[cuviper]

You mentioned that before, but this seems to work fine as-is. I've tried it in a separate file, and rustdoc --test works too. Hopefully this is not just an accident, because it will be nicer if users don't need to know it's asm underneath.

[alexcrichton]

Ah, I forgot about #12122!

It's unclear how much of a bug that is, but this is an experimental crate so it's ok to be susceptible to breakage.

[alexcrichton]

Being unfamiliar with these sorts of probes, is it required that these be an identifier? Could they be an arbitrary string?

[sanxiyn]

1. C uses identifiers too: STAP_PROBE(provider, name). 2) Is there a way to accept only strings, not other expressions? $name:ident accepts only identifiers, but $name:expr accepts too much.

[alexcrichton]

The other option could be $name:expr with stringify!($name), but that may be too accepting still. If only identifiers are supported, that's not so bad!

[cuviper]

On SystemTap's part, it's not strictly necessary to be an identifier. The C implementation uses them to name a variable for the semaphore, but that's an internal detail could be totally different if/when we add semaphores here. And they are stringified in the final macro already, so it could conceivably let expr through if you really want.

I'm not sure if GDB is prepared for arbitrary strings in break -probe name though. At first glance, its line parser seems to be breaking it up on spaces, regardless of quoting. That might be fixable.

DTrace USDT uses them in a function name - see USDT Macros. I'm not sure how necessary it is to be named like that, because I don't know USDT internals.

So sticking to identifiers seems best. Even better would be just C-identifiers, no Unicode, but I don't know how to enforce that in the macro. Also note that it did allow a keyword loop in my example, which is fine by me, just surprising that :ident allowed it.

[alexcrichton]

Ok! Sounds like we should stick to identifiers for now. We could expand to expressions in general later if necessary.

[cuviper]

FYI, I just noted on reddit that Unicode does work. I had no idea that even C99 allows this! :)

[olivren]

I'm not familiar with SystemTap, either. If I stumble upon a piece of code using probe!, I'd be a bit puzzled by the use of identifiers.

First, I will ask myself whether these identifiers make reference to a previously declared variable, and I'd scan the code to find them with no luck. Then, I'd wrongly assume this macro introduces a new variable or another item.

Would it be possible to use a more specific macro syntax instead of a coma-separated list of identifiers ? This would hint me that these identifiers are somewhat interpreted specially by the macro.

fn main() {
    probe!(main@foo);
    let x = 42;
    probe!(show_x@foo: x);
    let y = Some(x);
    probe!(show_y@foo: match y {
        Some(n) => n,
        None    => -1
    });
}

fn main() {
    probe!(begin@foo);
    let mut total = 0;
    for i in range(0, 100) {
        total += i;
        probe!(loop@foo: i, total);
    }
    assert_eq!(total, 4950);
    probe!(end@foo);
}

[cuviper]

@olivier-renaud, that's a fair point about identifier confusion. These are totally outside the normal code namespace, and I like the idea of changing up the syntax to make it look special. (That was never an option in CPP macros.) The chosen syntax won't matter much to the actual implementation - just tweaking the macro patterns. Now we get to bikeshed on what might look better. :)

I don't really care for name@provider, just because that's inverted from the order that the existing tools refer to these fields. Flipping to provider@name could be better, except '@' has the wrong hierarchy implication when read like "at", and there's also the past association with '@' pointers. Maybe some other symbol wouldn't have as much baggage, like provider$name. (I don't think any QWERTY-keyboard symbols are left without any rust meaning at all...)

SystemTap is relatively verbose with process.provider("provider").mark("name"), so that's not helpful for macro inspiration. GDB refers to probes by provider:name. DTrace uses provider:module:function:name (module being the binary, or crate in rust terms), and you can omit fields to leave just provider:::name. So both GDB and DTrace resemble rust paths - maybe we should embrace that and go with provider::name? Or maybe that's confusing if it looks like a real path but isn't, so just settle on provider:name?

If the colon is the initial separator, I think think it's fine to stick with commas for the optional arguments, probe!(foo:bar) and probe!(foo:baz, arg1, arg2,...). But since placing a probe is vaguely like a function call,maybe we could also use parentheses like probe!(foo:bar()) and probe!(foo:baz(arg1, arg2, ...)).

Opening up syntax leaves endless possibilities. Opinions and suggestions are welcome...

[olivren]

provider:name seems good to me, even better if you say that other tools use this syntax. I like the parenthesized form, too.

[sanxiyn]

This looks good to me.

Ideally, we should have tests, but even if this PR contained tests, before enabling them SystemTap would need to be installed to Linux buildbots etc. I am fine with adding tests later, as long as they are added.

[cuviper]

Do those buildbots have at least GDB 7.5? I see that src/test/debuginfo does a lot with GDB, and I could probably make similar tests for probe! breakpoints and their arguments.

[alexcrichton]

I've been thinking about this, and I'm wary to starting including a crate in the standard rust distribution that's only supported on linux/android, and also doesn't have many tests to prevent it from regressing. I think this is a fantastic crate and it's very well written, but it may not be the right time to include in the standard distribution.

I would love to see this on the wiki page and rust-ci, however!

[cuviper]

It's sort of a weird case, because the "normal" behavior of a probe! is transparent, to do nothing, and the "unsupported" platforms can do nothing perfectly well -- that's why I included platform/default.rs. :) The operation of the program is fine in any case, and those platforms will only miss out on the added support for external tools. So I disagree that such platform support should be a blocker, because then you'll be forever limited to the least-common-denominator of tooling.

On the testing point, I was hoping for an answer whether GDB 7.5 can be relied on in the buildbots, then I'm happy to write tests targeting that. Or if only some buildbots have GDB 7.5+, maybe there's a way to mark the tests ignored on others. If GDB break -probe tests pass, I claim that's enough to say SystemTap would be fine too, since they use the same metadata. (If not, that bug would belong on SystemTap anyway.)

If you still think it's not suitable, then I'll gladly publish it separately, but I think there's more value to having this included in standard rust, and little-to-no downside. The worst case for rust is that it might produce SDT metadata that the tools can't understand, but rust programs will still work regardless.

[alexcrichton]

I agree that this shouldn't be blocked solely based on the fact that it's only implemented for linux, but this is also a library that hasn't had much time to bake in the community to see how it works out and how it gets used. This is a perfect candidate for a cargo package to be super easy to install, but cargo is still in the works at this time.

For now, I think it's best to publish separately to get some experience with it getting used in the community. If this turns out to be the de-facto standard used throughout rust libraries, then I think it'd be a great fit for being part of the distribution.

Also, sorry, I thought I commented (bad memory), but the bots don't have GDB 7.5 right now. They're all on 7.4 or below. We could update them if necessary, however.

If it's ok with you, I'm going to close this for now, but I highly encourage you to publish in your own repo. This is a high-quality library which definitely deserves attention!

[No9]

@cuviper did you look into publishing it as your own repo?
Is there anything I could do to support you with it?

[cuviper]

Thank you for the nudge, @No9 -- I've now published libprobe. I haven't yet looked at cargo or rust-ci, so if you care to contribute those, it would nice. I see you also have some dtrace experience, so if you can broaden the platform support thus, that would be welcome too. :)