Filter out non-running pods in Prometheus

[acondrat]

Prometheus job generated by the PodMonitor does not exclude non-running pods. All the "completed" Pods are still going to be listed as targets in Prometheus and marked as down. This issue is related to PodMonitor implementation and is discussed in prometheus-operator/prometheus-operator#4816

Signed-off-by: Arcadie Condrat arcadie.condrat@gmail.com

[acondrat]

Same change for the Flux2 helm chart - fluxcd-community/helm-charts#121

[stefanprodan]

@acondrat would this prevent users from being notified when a Flux controller is in crash loop?

[acondrat]

A pod that is in a crashloop would still be captured by the up == 0 query. I suppose most(if not all users) have an alerting rule built around that. This is also something that is captured by the kube-state-metrics exporter with a KubeDeploymentReplicasMismatch alert.

[stefanprodan]

LGTM

Thanks @acondrat

[SuperQ]

You may also want to include Pending. I think in some cases, the Pod IP has been assigned, but not all containers have started.

Including Pending should also shorten the time between a Pod starting and Prometheus getting the configuration updated for the first scrape.