Ignore failed pods for KubePodNotReady

[gouthamve]

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase

Essentially is something is evicted, or exits with non-zero, it gets rescheduled. Now, the failed pod sticks around until --terminated-pod-gc-threshold.

The only exception to this rule is that Pods with a phase of Succeeded or Failed for more than some duration (determined by terminated-pod-gc-threshold in the master) will expire and be automatically destroyed

--terminated-pod-gc-threshold int32     Default: 12500
Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If <= 0, the terminated pod garbage collector is disabled.

This is causing us some alerts like:

[gouthamve]

@brancz @tomwilkie

[brancz]

Yeah we actually had a similar case with Jobs that were causing lots of Completed pods.

This lgtm 👍