Include internal trace context with internal metrics

[binarylogic]

#1953 introduced a new internal_metrics source for Vector's internal metrics. Metrics are defined within events, you can see an example here. A caveat noted in #1953 is the ability to carry over the internal tracing context:

A big benefit of tracing is its ability to propagate contextual information about where you are in the system. We use this extensively for our logging to tell you what type and the name of the current component without having to manually pass that data around everywhere just for log messages. This same functionality would be very beneficial for naming and tagging metric data, so a tracing based solution is what we started with and what I spent quite a bit of time exploring.

Unfortunately, the state of using tracing to collect metrics is extremely early right now. What we want to do is definitely possible, but both the exact implementation and the public API are still undergoing a lot of experimentation. This seems like an area we could jump in and help advance the state of the art, but my current read is that we need a metrics system sooner than that work will be ready, even with our help.

This issue represents the work to solve this.

[binarylogic]

@lukesteensen is this something @ktff could work on? This would require upstream work but I think it might be worth it.

[lukesteensen]

I've chimed in on metrics-rs/metrics#70 to see if there's anything we can help with.

[binarylogic]

@MOZGIII as discussed, I think this is the highest priority change outside of any remaining k8s works. I've added this to your sprint. It might be worth chiming in on metrics-rs/metrics#70 to ensure your approach will be accepted.

[binarylogic]

@MOZGIII how's this coming? Just noting this is priority after the source-level line merging.

[MOZGIII]

I've started with the implementation.

[lukesteensen]

@MOZGIII Could you explain a bit about what your rough plan is here? I'd like to do as much of this work as possible in the open so that we can make sure we're aligned with the upstream projects.

[binarylogic]

To add more context here. We chatted with the maintainer of metrics-rs in an email thread which then moved to a Discord channel. Mike has been chatting with him there about the direction. But I agree, I would like to hear a rough plan, and also double verify that his plan is blessed by the maintainer.

[lukesteensen]

Ah, gotcha. I can follow along in Discord.

[MOZGIII]

Yep, here's what we're planning to do:

• we'll add a Layer for metrics crate that would inject labels;
• this, in turn, would require two things:
  • in metrics crate ecosystem:
    • either disabling the static labels optimization, or
    • changing the APIs to decouple the static and dynamic label sets (we're leaning to this option currently);
  • in tracing ecosystem:
    • as I discovered today, we'll need to add a way to enumerate all fields a Scope has (no such API currently).

That's about it, roughly. I hope it goes smoothly and without any more surprises.

[MOZGIII]

A tiny status update:

• turns out no changes are required in tracing crate - there's a (non-trivial) way to do what we need with the current API;
• there's work ongoing on the API changes to allow dynamic labels passing; the macros might change a bit too.

For further progress, check out the metrics-rs/metrics#87.
We're also discussing the progress in chat at https://discord.gg/RqTk6bq - feel free to join if you're interested.