linkerd-viz helm: add support for metric_relabel_configs (linkerd#12248)

It is currently not possible to easily drop metrics with the Prometheus
configuration that is included with the linkerd-viz helm chart. To
enable this we extend the Prometheus configuration template to add a
metric_relabel_config section. This configuration is usually used when
dropping metrics and thus allows users to customize their recorded
metrics to avoid high cardinality.

E.g. there is a blog post that describes the problem in depth:
https://itnext.io/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b

With this change we are able to deploy the helm chart without the need
to do custom modifications to the templates.

Fixes linkerd#12194

Signed-off-by: Heiko Voigt <heiko.voigt@jimdo.com>
Signed-off-by: Mark S <the@wondersmith.dev>

viz/charts/linkerd-viz/README.md

@@ -141,6 +141,7 @@ Kubernetes: `>=1.22.0-0`
 | prometheus.image.tag | string | `"v2.48.1"` | Docker image tag for the prometheus instance |
 | prometheus.logFormat | string | defaultLogLevel | log format (plain, json) of the prometheus instance |
 | prometheus.logLevel | string | defaultLogLevel | log level of the prometheus instance |
+| prometheus.metricRelabelConfigs | string | `nil` | A metricRelabelConfigs section allows to drop high cardinality metrics. *NOTE:* Please use with caution. Some metrics are needed for linkerd-viz to function properly. |
 | prometheus.nodeSelector | object | `{"kubernetes.io/os":"linux"}` | NodeSelector section, See the [K8S documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) for more information |
 | prometheus.podAnnotations | object | `{}` | annotations for the prometheus pod |
 | prometheus.proxy | string | `nil` |  |

viz/charts/linkerd-viz/templates/prometheus.yaml

@@ -137,6 +137,10 @@ data:
       # Copy tmp labels into real labels
       - action: labelmap
         regex: __tmp_pod_label_(.+)
+      {{- if .Values.prometheus.metricRelabelConfigs }}
+      metric_relabel_configs:
+      {{- toYaml .Values.prometheus.metricRelabelConfigs | trim | nindent 6 }}
+      {{- end}}
 
     {{- if .Values.prometheus.scrapeConfigs }}
     {{- toYaml .Values.prometheus.scrapeConfigs | trim | nindent 4 }}

viz/charts/linkerd-viz/values.yaml

@@ -499,6 +499,18 @@ prometheus:
   #   - action: labelmap
   #     regex: __meta_kubernetes_node_label_(.+)
 
+  # -- A metricRelabelConfigs section allows to drop high cardinality metrics.
+  # *NOTE:* Please use with caution. Some metrics are needed for linkerd-viz to
+  # function properly.
+  metricRelabelConfigs:
+  # This allows us to fine tune prometheus cardinality by dropping certain
+  # metrics as suggested here: https://itnext.io/optimizing-linkerd-metrics-in-prometheus-de607ec10f6b
+  #
+  # Ex:
+  # - action: keep
+  #   source_labels: [le]
+  #   regex: "(?i)(|10|50|100|500|1000|10000|30000|\\+Inf)"
+
   # -- A sidecarContainers section specifies a list of secondary containers to run
   # in the prometheus pod e.g. to export data to non-prometheus systems
   sidecarContainers: