[MON-2358] add alert PrometheusOperatorRejectedResources

[raptorsun]

This PR add runbook for the alert PrometheusOperatorRejectedResources.
I have different opinion in the mitigation sections than the runbook of Prometheus Operator, which suggests the CR does not conform CRD schema. I find if the CR does not conform the schema, kubelet API server will reject before Prometheus Operator does. The error should come from the value of certain fields instead.

[simonpasquier]

not sure that it's worth mentioning it.

[raptorsun]

I suggest we tell users not to bother with the schema, the problem comes from value, because user may not know that schema is validated before writing the resource to database.

[simonpasquier]

I would write down a list of possible causes and how to remediate

• ServiceMonitor or PodMonitor referencing a file in the filesystem => use secret/configmap key reference
• Missing secret or configmap key reference => verify that the secret/key exists
• Invalid relabeling configuration => fix the configuration
• ...

[raptorsun]

Added a list of possible causes, not exhaustive though.

[simonpasquier]

IIUC namespace is either openshift-monitoring or openshift-user-workload-monitoring. In the first case, there's nothing much to do except filing a support case (e.g. it's a product bug). In the second case, the user deploying the resource should fix something on their end. This can be explicitly mentioned in the runbook.

[raptorsun]

Yup, I will add "If the namespace is named openshift-.*, for example, openshift-monitoring,
this may be a bug in Openshift, please file a support case."

[raptorsun]

Thank you for the review @bburt-rh :) The runbook has been updated accordingly.

[jan--f]

/lgtm
but I'll add a
/hold
for @bburt-rh to give his final approval. Feel free to unhold after.

[raptorsun]

Thank you for the review @bburt-rh :)
The runbook has been updated accordingly.

[simonpasquier]

This paragraph is a bit unclear to me. The namespace label can be either openshift-monitoring or openshift-user-workload-monitoring but the faulty resource might live elsewhere.

[bburt-rh]

@simonpasquier - WDYT of adding the following sentence:

"However, note that the namespace label in the description can be either openshift-monitoring or openshift-user-workload-monitoring, but the faulty resource might still be located elsewhere."

[simonpasquier]

Can we rephrase it to make it more clear about what isn't allowed? E.g. "A ServiceMonitor or PodMonitor object references a file to use as bearer token or TLS file which isn't allowed for user-defined monitoring. Instead it is necessary to create a secret holding the credential data in the same namespace as the ServiceMonitor or PodMonitor object and use a secret key reference in the ServiceMonitor or PodMonitor."

[bburt-rh]

Slightly edited version of @simonpasquier's suggested wording:

"A violation can occur when a ServiceMonitor or PodMonitor object references a file to use as a bearer token or references a TLS file. These configurations are not allowed in user-defined monitoring. Instead, in user-defined monitoring, you must create a secret that contains the credential data in the same namespace as the ServiceMonitor or PodMonitor object and use a secret key reference in the ServiceMonitor or PodMonitor configuration."

WDYT?

[simonpasquier]

I'd list by monitor resources:

• PodMonitor and ServiceMonitor
  • ...
• AlertmanagerConfig
  • ...
• PrometheusRules
  • ...

[raptorsun]

Thanks @bburt-rh @simonpasquier for the review.
Runbook is updated according to latest comments, please have a look :)

[simonpasquier]

the page should always filter by firing alerts -> I'd remove this step

[simonpasquier]

I think that it's confusing because the namespace isn't the namespace where the faulty resource lives.

[simonpasquier]

I'd rather tell to look at the labels directly.

[simonpasquier]

I'd make the difference between openshift-monitoring and openshift-user-workload-monitoring more explicit.

The course of action depends on the value of the `namespace` label:
* When the value is `openshift-monitoring`, this is an issue with the platform monitoring stack. Please submit request to the support.
* When the value is `openshift-user-workload-monitoring`, this is an issue with a user-defined monitoring resource.

[simonpasquier]

Should be moved with the other ServiceMonitor/PodMonitor causes?

[raptorsun]

/unhold

[simonpasquier]

One small nit otherwise lgtm.

[simonpasquier]

Suggested change

	Find the custom resource type and the namespace in the `description` label
	Identify the custom resource type and the namespace from the `resource` and `namespace` labels

[raptorsun]

Thank you for the review, @bburt-rh @simonpasquier :D
The runbook is updated accoridngly.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[jan--f]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan--f, raptorsun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• alerts/cluster-monitoring-operator/OWNERS [jan--f,raptorsun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@raptorsun: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.