internal/store/pod.go: Only create waiting_reason series if pods are in waiting state

[lilic]

Due to the extremely high cardinality of this metric and with 2.0 release being breaking, this now changes that the waiting_reason metrics only include series that have state Waiting. We also now can include all the reasons as cardinality will only always be O(count of containers waiting). I made the adjustment for both containers and init containers.

The reduction of series can be seen in the unit tests, but I also tried it on my cluster and this is the result we only see the waiting containers:

# HELP kube_pod_container_status_waiting_reason Describes the reason the container is currently in waiting state.
# TYPE kube_pod_container_status_waiting_reason gauge
kube_pod_container_status_waiting_reason{namespace="default",pod="nginx-deployment-6b99d49f64-km96g",container="nginx",reason="ImagePullBackOff"} 1
kube_pod_container_status_waiting_reason{namespace="default",pod="nginx-deployment-6b99d49f64-2ghjb",container="nginx",reason="ImagePullBackOff"} 1
kube_pod_container_status_waiting_reason{namespace="default",pod="nginx-deployment-6b99d49f64-pgj2t",container="nginx",reason="ImagePullBackOff"} 1

Will open PRs to adjust the 0 values for other metrics here as well in another PR.

@brancz @smarterclayton @vsxen can you please have a look if this metric continues to work as you want.

Fixes #1321

[brancz]

Seems like CI is not happy, but strategy and implementation is sound.

/lgtm

[lilic]

Thanks @brancz I always forget about our main tests :D PTAL, thanks!

/hold

Put a hold until one of the reporters confirms this fixes it 👍

[brancz]

/lgtm

[smarterclayton]

Looks great, thanks!

[lilic]

/hold cancel

@brancz PTAL, I adjusted the docs so lgtm got dropped.

[vsxen]

LGTM

[brancz]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, lilic

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [brancz,lilic]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment