Using Kubernetes labels/annotation as Prometheus alert labels and annotations is limiting

[ArthurSens]

The strategy of using kubernetes labels and annotations, prefixed with pyrra.dev/, as a strategy to generate Prometheus Alert labels and annotations works great for most cases! However, Kubernetes labels and annotations need to pass through different regular expressions than Prometheus requires, limiting the possible label values that one could use.

One common case is passing URLs, e.g. Runbook URL and Dashboard URL, as alert labels, but Kubernetes rejects such cases, another example is the 63 characters limitation:

ServiceLevelObjective.pyrra.dev "pyrra-pyrra-reconciliations" is invalid: [metadata.labels: Invalid value: "https://grafana.xxx.xxx/d/ccssRIenz/pyrra-slos?orgId=1&var-CX_ENV_ID=xxxx&from=now-1d&to=now": must be no more than 63 characters, metadata.labels: Invalid value: "https://grafana.xxx.xxx/d/ccssRIenz/pyrra-slos?orgId=1&var-CX_ENV_ID=xxx&to=now": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]

[ArthurSens]

We could easily fix that by adding an option for adding extra labels/annotations in the Alerting spec

[metalmatze]

Oh, that's an interesting point with the URLs. I haven't run into this myself yet.
I guess in that case your proposal makes sense.

The original reasoning for having them as CRD labels is to make filtering via kubectl get slo -l pyrra.dev/team=backend possible. I wonder how much usage that feature has in the first place.

[ArthurSens]

The original reasoning for having them as CRD labels is to make filtering via kubectl get slo -l pyrra.dev/team=backend possible. I wonder how much usage that feature has in the first place.

I usually search for SLOs using ArgoCD UI or Pyrra UI. I haven't used kubectl in a while, honestly 😬. If people rely on that feature, I guess they could keep their labels there, but I understand the duplicated effort to keep labels in sync in two different places.

I can totally understand that I don't represent all users! Do you see other people requesting the original use case?

[PabloPie]

In our case, we use annotations for runbooks and dashboard links, they are also propagated when prefixed with pyrra.dev/ and allow for much larger texts. I think labels are meant to be used more to filter, which is not something we usually require for those links.