revert failed pod marked as NotReady

[pascalwhoop]

This issue
#215
turned a "Failed" pod into a "PodNotReady". The creator of the PR wanted this for his evicted pods. I believe this is not correct. I would suggest looking at

kube_pod_container_status_terminated_reason{reason!="Completed"}
OR
kube_pod_container_status_terminated

to build a metric if people need it. Why are we reverting? Because any failed pod will now trigger this alert, since pods stay in the API for quite a long time and therefore any failed pod == PodNotReady.

Failed pods occur all the time. Build systems, batch processing, etc.

[pascalwhoop]

@metalmatze what's your take on this? Currently, any failed pod triggers this alarm and the alarm doesn't disappear until the pod is removed from the API

[Capitrium]

It would be great to get this merged in soon, the KubePodNotReady alert is currently very noisy and isn't the right way to track Evicted pods.

From the docs (emphasis added):

The kubelet can proactively monitor for and prevent total starvation of a compute resource. In those cases, the kubelet can reclaim the starved resource by proactively failing one or more Pods. When the kubelet fails a Pod, it terminates all of its containers and transitions its PodPhase to Failed. If the evicted Pod is managed by a Deployment, the Deployment will create another Pod to be scheduled by Kubernetes.

The original issue in #215 (emphasis added) was:

Evicted pods can be in phase failed and I would expect this alert to catch this.

I don't think this expectation is correct; Evicted pods are always in phase=Failed.

@metalmatze @brancz thoughts?

[wu0407]

pod not ready it look at "kube_pod_container_status_ready" and "kube_pod_status_ready"