[kube-prometheus-stack] hack: fix etcd mixin

[jkroepke]

What this PR does / why we need it

In etcd-io/etcd#16246, etcd maintainers refactor the mixins. Instead having one mixin file, the distribute all the declaration across multiple files.

Running sync_prometheus_rules results into this error:

Generating rules from https://raw.githubusercontent.com/etcd-io/etcd/main/contrib/mixin/mixin.libsonnet
Traceback (most recent call last):
  File "/Users/jok/IdeaProjects/prometheus-community-helm-charts/charts/kube-prometheus-stack/hack/./sync_prometheus_rules.py", line 445, in <module>
    main()
  File "/Users/jok/IdeaProjects/prometheus-community-helm-charts/charts/kube-prometheus-stack/hack/./sync_prometheus_rules.py", line 426, in main
    alerts = json.loads(_jsonnet.evaluate_snippet(chart['source'], raw_text + '.prometheusAlerts'))
RuntimeError: RUNTIME ERROR: couldn't open import "./config.libsonnet": no match locally or in the Jsonnet library paths.
        https://raw.githubusercontent.com/etcd-io/etcd/main/contrib/mixin/mixin.libsonnet:1:2-29  

Addition, jsonnet-builder is required for proceed the jsonnet now. I added a switch which allows to clone the source and run jb + jsonnet to generate the prometheus rules.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[QuentinBisson]

@jkroepke can you fix the imports? :)

[jkroepke]

@QuentinBisson done

[dragoangel]

Hi @QuentinBisson @jkroepke

After getting dashboard from this PR there is issue with all panels as they use wrong filters:
sum(etcd_server_has_leader{job=~".*etcd.*", job="$cluster"})

when it should be:
sum(etcd_server_has_leader{job="$job"})

as:

• we already limit scope of $job to ~".*etcd.*"
• we don't have $cluster variable, we have $job variable which just have "display name" of cluster

[jkroepke]

@dragoangel I double check this now, but it seems like that someone manually removes the job=~".*etcd.*", label. I can validate the this limitation exists since 2021 in etcd repository and someone modify the dashboard and rules locally by hand.

It seems like that the etcd maintainer enforcing that the job label contains etcd somewhere.

@QuentinBisson whats your opinion? Restore the initial behavior by hand or replicate the new behavior? While creating the PR, I expect the changes and double check them with upstream.

[QuentinBisson]

The issue is actually coming from this line upstream:
https://github.com/etcd-io/etcd/blob/252f5a6f4d452b63c091ed73315bd3612a890e07/contrib/mixin/dashboards/targets.libsonnet#L8

[QuentinBisson]

But as you can see here it was already set that way

[QuentinBisson]

IMO the issue is that when upstream migrated to grafonnet instead of grafonnet-lib, they screwed the mixin variable definition here https://github.com/etcd-io/etcd/blob/252f5a6f4d452b63c091ed73315bd3612a890e07/contrib/mixin/dashboards/variables.libsonnet#L12

@jkroepke how about we manually fix this with a comment and create an issue upstream ? Can you take care ?

[jkroepke]

@QuentinBisson I bit of tracking the changes in this PR compared to the Changes linked the commit you mentioned.
etcd-io/etcd@f564436#diff-007651fd1693f51e6c72b237590b65d2a1a72fa3bba2d24993c2bf417bb25792L332

I have the option that the definition of the template variables is equal, except that label_values(etcd_server_has_leader, job) has been changed to label_values(etcd_server_has_leader{job=~".*etcd.*"}, job). And the queries of the panel are changes from sum(etcd_server_has_leader{job="$cluster"}) to sum(etcd_server_has_leader{job=~".*etcd.*",job="$cluster"})

Saving the label values of job label to a variable named $cluster, and re-use job="$cluster" in Panel Query seems legit to me. (Except the naming of the cluster label feels a bit odd). This logic has not been changed in this PR here. The only thing that is new here is the filter job=~".*etcd.*" appears everyone. At the moment, I have not the feeling that upstream does something wrong here.

To restore the old behavior, the can adjust the jsonnet config, like set etcd_selector: 'job=~".*"' ( I guess empty string is not support here)

Edit: I saw your upstream issue now. Looking forward there. 👍

[QuentinBisson]

I think the issue is not really that the variable upstream is called job instead of cluster so the dashboard variable templating is incorrect.

Sadly, setting etcd_selector: 'job=~".*"' will not fix the fact the $cluster variable is not set :)

As a temporary fix (i.e. if the etcd fix is not merged in a decent timeline), we can edit the value in our template (with a comment that would be seen during a review) to name the variable cluster instead of job in the etcd dashboard. WDYT @jkroepke ?

[QuentinBisson]

It's now fixed by #3917