Allow custom metric_relabel_config for Prometheus

[hvoigt]

What problem are you trying to solve?

Linkerd proxy metrics tend to have quite high cardinality. E.g. see this Medium article suggests to drop some le values to reduce cardinality:

metric_relabel_configs:
  - action: drop
    source_labels: [le]
    regex: "2.*|3.*|4.*|5.*"

We also experience that response_latency_ms_bucket has too high cardinality.

How should the problem be solved?

Provide a custom configuration for metric_relabel_configs in the linkerd-viz helm chart. E.g.

prometheus:
[...]
  proxyMetricRelabelConfigs:
  - action: keep
    source_labels: [le]
    regex: "(?i)(|10|50|100|500|1000|10000|30000|\\+Inf)"

Any alternatives you've considered?

We are currently copying the linkerd-viz helm chart and manually applying this change locally. This is annoying in case of an update as we have to reapply this change and release our own local version.

How would users interact with this feature?

If they use the Prometheus that comes with linkerd they define their wanted metric_relabel_configs using the helm charts values.

Would you like to work on this feature?

yes

[hvoigt]

@mateiidavid, @kflynn I attached a PR that solves this issue. What do you think?

[kflynn]

@hvoigt This is one of those requests that feels simple, but hides a fair amount of complexity. 😐 The challenge here is that metrics are pretty central to Linkerd's operation, and relabelings can actually break things that many people rely on.

As @alpeb wrote in #11445, "If you know exactly what you're doing, like reducing the cardinality of response_latency_ms_bucket, my advise is to not use the stock linkerd PodMonitors, and instead provide your own with your custom changes, pluging that in your pipeline via the standard mechanisms (kustomize, helm post-rendering, sub-charts, or a separate chart)." This feels like a better route than a broad control in the stock chart.

I'm going to close this one, but thank you for digging into this! Please feel free to reach out here or on Slack if you have more questions.

[kflynn]

Whoops, I also meant to link to the instructions for running your own Prometheus with Linkerd because another excellent tack here is doing that, then giving your Prometheus whatever configuration you want.

[hvoigt]

Sorry I missed that this issue has been closed. This whole issue -> PR discussion feels a bit disconnected.

@hvoigt This is one of those requests that feels simple, but hides a fair amount of complexity. 😐 The challenge here is that metrics are pretty central to Linkerd's operation, and relabelings can actually break things that many people rely on.

Could you elaborate what things other than linkerd-viz this could break? And likely it only breaks for people that already have a broken Prometheus because of high cardinality memory usage. If we document it that way I do not see big harm in allowing this config.

As @alpeb wrote in #11445, "If you know exactly what you're doing, like reducing the cardinality of response_latency_ms_bucket, my advise is to not use the stock linkerd PodMonitors, and instead provide your own with your custom changes, pluging that in your pipeline via the standard mechanisms (kustomize, helm post-rendering, sub-charts, or a separate chart)." This feels like a better route than a broad control in the stock chart.

Bringing your own Prometheus adds unnecessary complexity in my opinion. What is so bad allowing users to fine-tune the stock Prometheus setup? E.g. we added a warning to the documentation of this option over at the PR.

I'm going to close this one, but thank you for digging into this! Please feel free to reach out here or on Slack if you have more questions.

I hope we can revisit this decision as it seems there are multiple people suffering from high cardinality which could be solved simpler by a stock tuning option.