[prometheus-kube-stack] "Error on ingesting out-of-order result from rule evaluation"

[antoineozenne]

Describe the bug
There is some warn errors in logs:

level=warn ts=2021-07-19T12:40:08.145Z caller=manager.go:651 component="rule manager" group=kube-apiserver.rules msg="Error on ingesting out-of-order result from rule evaluation" numDropped=8
level=warn ts=2021-07-19T12:40:48.951Z caller=manager.go:651 component="rule manager" group=kube-apiserver-burnrate.rules msg="Error on ingesting out-of-order result from rule evaluation" numDropped=1

This seems to trigger the PrometheusMissingRuleEvaluations alert.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.10", GitCommit:"98d5dc5d36d34a7ee13368a7893dcb400ec4e566", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.10", GitCommit:"98d5dc5d36d34a7ee13368a7893dcb400ec4e566", GitTreeState:"clean", BuildDate:"2021-04-15T03:20:25Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

Which chart: kube-prometheus-stack

Which version of the chart: 16.14.1

What happened:

The record cluster_quantile:apiserver_request_duration_seconds:histogram_quantile, defined in kube-apiserver.rules.yaml and kube-apiserver-histogram.rules.yaml, contains a lot of NaN values (because of some 0 values in the instant-vector in histogram_quantile). This triggers PrometheusMissingRuleEvaluations alert.

What you expected to happen:

Handle the rule to not trigger alert.

Anything else:

I noticed this commit redefines this record in a second group of rules f501c4ed62c9e77cf96b46e83202f6ea17a13b97 (kube-apiserver-histogram.rules in addition to kube-apiserver.rules).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[antoineozenne]

/no-stale

[antoineozenne]

Still exists in 18.0.0.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[antoineozenne]

/no-stale

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[antoineozenne]

/no-stale

[antoineozenne]

I can provide more information if you need.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[antoineozenne]

/no-stale

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[antoineozenne]

/no-stale

[bryanasdev000]

Can confirm, same here.

Also related to #1283, with a possible workaround.

EDIT: Also related in my case kubernetes-monitoring/kubernetes-mixin#392 and https://docs.microfocus.com/itom/HCMX:2021.05/PrometheusManyToManyMatching .

[antoineozenne]

I think setting .Values.defaultRules.rules.kubeApiserver to false isn't really a workaround as it disables the monitoring. :)

[bryanasdev000]

I think setting .Values.defaultRules.rules.kubeApiserver to false isn't really a workaround as it disables the monitoring. :)

Forgot the quotes at workaround :P

In my specific case, resolving the many to many also resolved out of order result after a pod restart.

In my case it all refers to an old/dirty installation of Prometheus. In your context, does a new cluster have the same problems?

I am running K8S 1.19-1.21 with kube-prometheus-stack-23.3.2.

[antoineozenne]

It is the case for a new installation, yes.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[antoineozenne]

/no-stale

[bryanasdev000]

/no-stale

@antoineozenne take a look at: #1799

Basically, setting .Values.defaultRules.rules.kubeApiserver to false fixes it, as it disables the duplicate. You keep the monitoring for the API Server and its rules.

[antoineozenne]

Thank you @bryanasdev000, will use that.