Tracking issue for USDT Integration

[smklein]

https://crates.io/crates/usdt is super sick, and it's about time we start integrating! This issue tracks integration into a variety of potential spots within omicron:

• Sled Agent (device management)
• Sled Agent (interfacing w/Propolis)
• Sled Agent's bootstrapping
• Nexus (APIs)
• Nexus (DB) interface
• ... other?

[bnaecker]

Just a thought. We'd talked previously about getting the actual SQL strings at various points -- this might be a good candidate. We might not want to log them all the time, but a probe spitting it out could be very helpful.

[davepacheco]

I think we could get pretty far with:

• dropshot per-request probes
• slog probes for every message logged (even ones being filtered out)
• probes for database operations (is there a central place we can put this?)

[bnaecker]

@davepacheco USDT requires nightly, and there's no easy way to instruct it to generate empty probes if nightly isn't opted into. Are you cool with transferring that requirement to Dropshot?

[smklein]

async-bb8-diesel implements batch_execute_async with a query string - we could fire a probe for each one within that crate?

https://github.com/oxidecomputer/async-bb8-diesel/blob/dc5ff98304efe98dfb9736a4634e5841108690bb/src/connection_manager.rs#L115-L123

(We could also do the same thing for the per-connection implementation: https://github.com/oxidecomputer/async-bb8-diesel/blob/dc5ff98304efe98dfb9736a4634e5841108690bb/src/connection.rs#L39-L46 )

This would effectively look like:

    async fn batch_execute_async(&self, query: &str) -> ConnectionResult<()> {
        let diesel_conn = Connection(self.0.clone());
        let query = query.to_string();
        task::spawn_blocking(move || {
               /* ++ FIRE PROBE HERE, RECORD QUERY ++ */
               diesel_conn.inner().batch_execute(&query)
            })
            .await
            .unwrap() // Propagate panics
            .map_err(ConnectionError::from)
    }

[davepacheco]

@bnaecker What about adding a Dropshot (or usdt) feature such that:

• if the feature is enabled, you get probes and must be on nightly
• if the feature is not enabled, then you don't get probes and can use whatever toolchain you want

It's not as slick as automatically detecting this but it might even be better. It avoids a problem I've run into in the past where I thought something should have DTrace probes, but a bug in the build process failed to detect support, and we didn't discover that until we needed the probes in production.

[davepacheco]

@smklein That sounds good.

[bnaecker]

@davepacheco That should be fairly easy to add into USDT. We already have a no-op implementation for platforms without DTrace, so adding a feature that allows specifically selecting it would be straightforward. Dropshot could then re-export that feature.

[bnaecker]

Just an update on some of the upstream work. Probes for log messages can be inserted via the slog-dtrace crate. Probes for Dropshot requests are part of this PR. Probes for database operations (via async-bb8-diesel) are part of this PR. More nuanced probes around database operations would be a bit more work, but this should allow to us to instrument every query.

[bnaecker]

The database-related probes, for connections and queries, has been implemented as part of #488.

[bnaecker]

Log messages are emitted to DTrace as part of #489.