Metrics redesign: labels via context

[tobz]

High Level Goals

Take advantage of existing, contextual data to provide labels for metrics.

Notes

This one is fairly straightforward in concept, and could go many ways in implementation.

More or less, we'd like to be able to do something akin to tracing-error where we simply check if we're currently in a tracing span, and if so, extract its fields and use them as labels for a given metric measurement via counter!, etc.

There's obviously specifics to deal with in terms of which fields should be used, or how should the fields be formatted, but let's first consider what other contextual data might exist and how many any of it in particular is useful.

While the idea of reusing the span context is obvious, what's less obvious to me is how much flexibility there needs to be i.e. is it all fields or nothing, or do we allow picking and choosing?

[lukesteensen]

Hi! We're starting to rework how we do metrics in Vector (RFC here) and this feature would be super, super valuable for us.

For our use case, we'd like to specify a set of labels to be applied from the surrounding tracing spans whenever available. We could also handle all span data being applied, but I'm not sure it'd all be useful and we may choose to filter it downstream in that case.

[tobz]

For our use case, we'd like to specify a set of labels to be applied from the surrounding tracing spans whenever available. We could also handle all span data being applied, but I'm not sure it'd all be useful and we may choose to filter it downstream in that case.

Would the fields you want to copy over change on a per-span basis, or would a global static list be enough?

[lukesteensen]

A global static list would be perfect.

[lukesteensen]

@tobz is there anything we can do to help move this work forward? We'd love to have this available and are happy to contribute however would be helpful.

[binarylogic]

To echo @lukesteensen, we’d be happy to sponsor this feature or assign one of our engineers to take it to the finish line. However we can help. :)

[tobz]

Hey y'all!

I've just released alphas of the metrics v2 redesign, so this is the perfect time to work on this stuff!

I think the "assign an engineer" route is likely the more fruitful one, because sponsorship or not, I don't think I have enough time to dedicate here.

Going back to the original goals, just so we can all orient here: we want to be able to siphon off fields from the current span and use them as labels. In principle, this should be doable, but likely we want a way to filter these fields (the aforementioned global static list) because a lot of spans will have garbage data, in the context of metrics emitted within them.

My unexplored thoughts/options here:

• we can have a global static list, set by a free function, and macros will read that and slurp in span fields at the callsite
• Recorder methods can read the global static list (or recorders can require the list to be passed in) and do it there

I personally lean more towards pushing this into exporters because a global static list/free function is fragile and imposes unnecessarily on downstream exporters. I've been exploring the idea of composable exporters, things like layering and fan-out, and so this might also be a good use case for layering: design a single layer that wraps an existing Recorder and does the span field label bits and passes them down.

How do y'all feel about the above?

[tobz]

I'm going to close this issue as #87 was merged, which adds support for extracting tracing::Span fields and applying them as labels to metrics! 🎉