Support for Statsd Tags/Dimensions

[tehranian]

Greetings! Here's a feature request to track. Other folks that use Datadog for monitoring would likely be interested in this as well.

The statsd metrics that Envoy produces are geared towards the generic statsd implementation. For folks that use Datadog for monitoring this is suboptimal, because Datadog's statsd implementation supports metric tags (ie. dimensions). See:

• https://docs.datadoghq.com/guides/dogstatsd/
• https://www.datadoghq.com/blog/the-power-of-tagged-metrics/
• https://help.datadoghq.com/hc/en-us/articles/204312749-Getting-started-with-tags

Ex: For a given metric like a cluster's upstream_cx_rx_bytes_buffered, a single Envoy instance currently produces metrics with names:

envoy.cluster.CLUSTER_A.upstream_cx_rx_bytes_buffered
envoy.cluster.CLUSTER_B.upstream_cx_rx_bytes_buffered
...
envoy.cluster.CLUSTER_N.upstream_cx_rx_bytes_buffered

... for each cluster. In Datadog best practices, one would prefer a single metric-name like envoy.cluster.upstream_cx_rx_bytes_buffered which is tagged by the respective cluster name. This makes it possible to aggregate & monitor across many clusters, and be able to find anomalies or get alerts from any misbehaving cluster(s).

(And yes, part of the problem here is that Datadog doesn't support templating/string interpolation of metrics names in their monitors & dashboards.)

[mattklein123]

Tags/dimensions is something we would like to support. Also for Prometheus/Wavefront/Influx, etc. Needs some thinking on where to do this and how to keep back compat for statsd.

[dnoe]

@mrice32 This might be relevant to your interests.

[tehranian]

Didn't know that Prometheus/Wavefront/Influx supported tags as well. Changed the title of this issue to be more generic.

Thanks folks!

[mrice32]

+1 We will definitely need this feature for our internal monitoring as well.

[mattklein123]

This one will need some design thought. When someone firmly signs up to work on it we can figure out what to do here. @mrice32 if that's you, let's chat at some point.

[mrice32]

Yes, that will be me. Sounds good.

[mattklein123]

OK ping me offline when you are ready to work on this and we can chat and hopefully come back to this thread with a design proposal. Anyone else feel free to weigh in here with thoughts.

[mattklein123]

I just synced up with @mrice32 and after some brainstorming this is what I'm thinking about at a high level for this:

• We will always need to retain backcompat for stat backends that don't supporting tagging.
• Instead of having tagging/not-tagging be built throughout the code, it would be better if tagging is orthogonal to the main stats.
• We propose adding the ability to specify tag extraction via effectively regex capture expressions. For example cluster..foo.bar.baz would capture the cluster name as a tag.
• Because of how heavily cached stats are, we can do tag extraction when stats are first created (for some of the dynamic stats we might want to have them pre-cached on the scope also).
• Then when stats are sent to sinks, sinks will have access to full hierarchical name, as well as tag extracted name.

I think this approach balances efficiency, user configuration, and backcompat.

Please comment!