linkerd-viz helm: add support for metric_relabel_configs

[hvoigt]

It is currently not possible to easily drop metrics with the Prometheus configuration that is included with the linkerd-viz helm chart. To enable this we extend the Prometheus configuration template to add a metric_relabel_config section. This configuration is usually used when dropping metrics and thus allows users to customize their recorded metrics to avoid high cardinality.

E.g. there is a blog post that describes the problem in depth: https://itnext.io/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b

With this change we are able to deploy the helm chart without the need to do custom modifications to the templates.

Fixes #12194

[adleong]

Thanks for the PR, @hvoigt, and thanks for sharing that blog post. That's really cool. I would hesitate a bit to make the metric relabeling configurable like this because the linkerd-viz CLI and dashboard rely on certain labels and the exact set of required labels isn't exactly documented. Linkerd-viz is intended to be something that "just works" and we've been encouraging users who need to tune their prometheus to use an external promethus: https://linkerd.io/2.15/tasks/external-prometheus/#prometheus-scrape-configuration

[hvoigt]

Thanks for the PR, @hvoigt, and thanks for sharing that blog post. That's really cool. I would hesitate a bit to make the metric relabeling configurable like this because the linkerd-viz CLI and dashboard rely on certain labels and the exact set of required labels isn't exactly documented. Linkerd-viz is intended to be something that "just works" and we've been encouraging users who need to tune their prometheus to use an external promethus: https://linkerd.io/2.15/tasks/external-prometheus/#prometheus-scrape-configuration

Hey @adleong thanks for taking a look at my PR. The thing is we actually have our own separate Prometheus. We intentionally do not directly scrape linkerd metrics from there because we had problems with cardinality in the past. As there are some potentially unbounded label values we kept the linkerd Prometheus and federate the metrics we need to our external Prometheus. That way if we experience problems with cardinality not all metrics are affected. I do not see harm in adding a configuration option with a default that "just works". We could add a warning in a comment that you need to "know what you are doing". If you bring your own Prometheus you would have the same problem that the labels are not documented, right?

[adleong]

@hvoigt This sounds good. Would you mind adding that warning to the comment? Note that you'll also need to run bin/helm-docs to regenerate the helm docs with your changes.

[hvoigt]

@hvoigt This sounds good. Would you mind adding that warning to the comment? Note that you'll also need to run bin/helm-docs to regenerate the helm docs with your changes.

@adleong Great! Here is the updated PR. helm-docs also updated.