xds: improve how envoy metrics are emitted

[rboyer]

Since generated envoy clusters all are named using (mostly) SNI syntax
we can have envoy read the various fields out of that structure and emit
it as stats labels to the various telemetry backends.

I changed the delimiter for the 'customization hash' from ':' to '~'
because ':' is always reencoded by envoy as '_' when generating metrics
keys.

Fixes #6227

An updated grafana/prometheus dashboard for connect is available here: https://gist.github.com/rboyer/f8a3a35feec3a27010192bda5796769e

[banks]

LGTM.

One bikeshed on label name but not a big deal.

It would be awesome if you have this running locally if you could paste some sample metrics from the prometheus output so we can get a feel for the way the labelling comes out (I looked at the grafana dash so I have a good idea but would be nice to see the real thing as a sanity check).

Also, did you look at what happens to statsd metrics with this config? I'm not super clear on that but maybe could use the existing statsd integration test and make it fail and dump the raw statsd metrics sent or something?

[rboyer]

I scraped the /metrics endpoints on a sidecar proxy and a mesh gateway proxy. I also force crashed the dogstatsd-udp and statsd-udp integration tests to extract their logs:

https://gist.github.com/rboyer/9b880e0fb29c5f2f215741bb9fddc707

The stats extraction, understandably, does nothing to the plain statsd output because there's no place in the protocol IIRC to send tags. The dogstatsd one however does what you'd expect.

[banks]

envoy.cluster.membership_change:1|c|#foo:bar,local_cluster:s1,envoy.cluster_name:1a47f6e1~s2,consul.custom_hash:1a47f6e1,consul.service_subset:,consul.service:s2,consul.namespace:default,consul.datacenter:primary,consul.routing_type:internal,consul.trust_domain:d245a980-5b33-d9ee-839a-2c9860a74e62,consul.target:1a47f6e1~s2.default.primary,consul.full_target:1a47f6e1~s2.default.primary.internal.d245a980-5b33-d9ee-839a-2c9860a74e62

I assume it's intentional that envoy.cluster_name:1a47f6e1~s2 remains the prefixed name? It think this is reasonable as it allows distinguishing the cluster name if you care while using consul.service_name if you'd really just want to see which Consul service is being addressed.

👍 Thanks RB this looks great to me.

[mkeeler]

This looks good to me. I just have the one question about some regexes.

[mkeeler]

LGTM