Add message receiving USDTs (PoC)

[viraptor]

This requires the systemtap development package as well as dtrace tool
to be present during building.

The specific notifications are configured in radicle_link.d and the
names can be used to filter the results. Currently the notifications
work only on Linux, but a different crate can be used for the same
result on MacOS.

The result can be tested on a running seed with:

sudo bpftrace -p $(pgrep -f radicle-seed-node) -e '
  usdt:*/radicle-seed-node:radicleusdt:have_recv {
    printf("have, peer %s, urn %s\n", str(arg0), str(arg1));
  }
  usdt:*/radicle-seed-node:radicleusdt:want_recv {
    printf("want, peer %s, urn %s\n", str(arg0), str(arg1));
  }'

The solution uses sonde-rs for actual notifications. Unfortunately the upstream version doesn't compile now and the patch is needed wasmerio/sonde-rs#2 (already referenced in Cargo.toml)

This is a working proof-of-concept - more monitoring points can be added over time.

[viraptor]

ping @xla
This can be tested on seed with https://github.com/viraptor/radicle-bins/tree/viraptor/with-usdt

[kim]

This is very exciting.

Wondering how hard it would be to automate probe generation so uprobe/uretprobe could be simulated for Rust functions (like tracing’s #[instrument] macro).

[viraptor]

Technically that's possible.
The hard part would be to create the correct arguments automatically. usdt can only accept basic types and it would be nice to send something better than Debug string since the bpftrace tool can then do clever processing on the values. (current logs for example are a bit verbose with all the object details)

I think this may be better as an explicit call. Or maybe a combination of macro annotation pointing at a function used to process the args?

Alternatively it could be interesting to add a few probes explicitly and then look for commonalities?

[kim]

The hard part would be to create the correct arguments automatically. usdt can only accept basic types

Ya true, although maybe we could get away with strings, or key/value pairs of strings?

and it would be nice to send something better than Debug string. (current logs for example are a bit verbose with all the object details)

That's largely due to limitations of the tracing crate. We're not actually interested in distributed tracing, so their whole shenanigans around object safety are very unnecessary for our purposes. The problem obviously remains that all approaches to structured observability require to know _a priori_ what is needed _post mortem_, which lies in the theological domain. I suppose we could enhance the quality of our lives by just making it easier to attach `gdb` (gdbserver inside docker, anyone?), and limit logging / tracing / metrics to just the highest level sufficient to deduce a repro.

[kim]

I also see you are aware of usdt. Their macro approach seems quite pleasant, and also that arguments are actually lazily evaluated.

[viraptor]

The usdt create wouldn't be terribly hard to enhance with Linux support, but it's both more time than I have now and would likely permanently require nightly for asm support. (which I understand you want to move away from at some point?)

The lazy parameters may be possible with sonde, but I'm not sure. There doesn't seem to be any macro available for it and I'm not sure Linux pokes the right bit in memory. (https://github.com/oss-mirror/systemtap/blob/b367613c3bb88549dbb9944d306fe561dea03f04/includes/sys/sdt.h#L240 seems to Yolo over the whole section where the is-enabled bit is reserved in flags in https://github.com/oxidecomputer/usdt/blob/bd095fbbf712e9377fbf21589cc26d4221e2e3f2/usdt-impl/src/no-linker.rs#L143 )

Re. GDB, i guess you could do that, but debugger attaching + threads is hell in my experience. But yeah, with debug symbols and a set of reasonable macros it could definitely work. Usdt still has the advantage of low overhead and lightweight scripting though... GDB can be scripted with python, but I hate it with passion :-)

[kim]

permanently require nightly for asm support

Yeah that'd be a pita for packagers.

The lazy parameters may be possible with sonde, but I'm not sure. There doesn't seem to be any macro available for it and I'm not sure Linux pokes the right bit in memory. (https://github.com/oss-mirror/systemtap/blob/b367613c3bb88549dbb9944d306fe561dea03f04/includes/sys/sdt.h#L240 seems to Yolo over the whole section where the is-enabled bit is available in https://github.com/oxidecomputer/usdt/blob/bd095fbbf712e9377fbf21589cc26d4221e2e3f2/usdt-impl/src/no-linker.rs#L143 )

Interesting. This means that the overhead is comparable to logging, which is quite bad I think, because the level is evaluated at runtime. But maybe we could exploit this fact and dispatch to either a "base" or "debug" probe based on the log level?: ``` fn report_something<F>(args: F) where F: FnOnce() -> (&str, &str) { if log_enabled!(Level::Debug) { let (arg0, arg1) = args(); usdt::something_debug(arg0.as_ptr() as *mut _, arg1.as_ptr() as *mut _); } else { usdt::something() } } ```

Re. GDB, i guess you could do that, but debugger attaching + threads is hell in my experience.

Yes it's a pain. But defining up front which types have a traceable representation is a pain, too. Maybe it's simply all public types of a crate?

[xla]

Also curious how we can turn this into something that allows ergonomics similar to the instrument attribute macro.

[xla]

What's the significance of this file in the setup?

[viraptor]

It defines the probe signatures. It's used to transform lines like probe have_recv(char*, char*); into:

#include <sys/sdt.h>
...
#define RADICLE_LINK_HAVE_RECV(arg1, arg2) \
DTRACE_PROBE2 (radicle_link, have_recv, arg1, arg2)

(or similar, depending on your system and implementation) in the header.

This gets consumed to compile a library which can be extern'ed in the rust code.

Behold artist representation:

             build.rs                                           build.rs
                |                                                  |
radicle_link.d -> dtrace -h -> radicle_link-(rand).h -\            |
             \--> dtrace -G -> radicle_link-(rand).o -+--> $SONDE_RUST_API_FILE -> pub mod usdt;