Using k8s labels in prometheus rules expr

[vlogachev-bd]

Hello,

I am new to Prometheus so I would ask for some advice.

I am trying to use custom labels from k8s deployment in the prometheus rules expression.

Here is my setup:

k8s labels:

labels:
app: nginx-test
ignore-desired-replicas: "true"
ignore-ram-usage: "true"
ignore-cpu-usage: "true"
ignore-container-restarts: "true"
team: xxx

kube-state-metrics deployment.

args:
- '--metric-labels-allowlist=pods=[team, ignore-ram-usage, ignore-cpu-usage, ignore-container-restarts],deployments=[team, ignore-desired-replicas],persistentvolumeclaims=[team]'

Prometheus rule
expr: kube_deployment_labels * on (deployment, namespace) group_right(label_team) ((kube_deployment_status_replicas_available / kube_deployment_spec_replicas) < 1 )

I also tried to add group_right(label_team, ignore-desired-replicas) and ( (label_team, label_ignore-desired-replicas), but I am not even able to startup the container, I assume due to the fact that it does not recognize the label.

It looks to me that I am making a mistake somewhere in the kube-state-metrics configuration.

Currently, when I receive an alert I can see that it groups only be team and it is ignoring my second label ignore-desired-replicas.

Any thoughts, folks?

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale