Create pod priority class for user workload monitoring

[lilic]

This introduces a new PriorityClass named openshift-user-critical, and applies that policy to all user workload components. We want this because we have seen user workload pods get scheduled in favor of platform monitoring pods. We always want platform monitoring to have higher priority scheduling as it monitors platform critical workloads, user workload monitoring stack is now second priority class before all rest of user workloads.

See background on this openshift/enhancements#369.

• I added CHANGELOG entry for this change.
• No user facing changes, so no entry in CHANGELOG was needed.

[lilic]

cc @openshift/openshift-team-monitoring PTAL, I tested this on a cluster and the class is applied, not sure we need a test for it? If we were to test any disruption to the workloads, that would be testing the priority class k8s feature. Any other test suggestions, welcome!

[s-urbaniak]

My intuition is also that testing if the priority is applied is out of scope here (i.e. disruption).

What we could test is if the actual priority value of system-cluster-critical is higher than openshift-user-critical. That could be done as an e2e tests which reads out the values in a concrete cluster.

[lilic]

@s-urbaniak Sounds good, this test would be there to determine the new priority class is lower priority than the system one, correct?

[lilic]

level=error msg=Cluster operator authentication Degraded is True with IngressStateEndpoints_MissingSubsets::OAuthServerConfigObservation_Error::OAuthServiceCheckEndpointAccessibleController_SyncError::OAuthServiceEndpointsCheckEndpointAccessibleController_SyncError::RouterCerts_NoRouterCertSecret: OAuthServiceCheckEndpointAccessibleControllerDegraded: Get "https://172.30.152.80:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

unrelated it seems
/retest

[lilic]

/retest

[s-urbaniak]

@lilic the test looks spot on (at least to me) 👍 I have just one logical question.

[lilic]

@s-urbaniak PTAL, adjusted as per feedback, thanks!

[simonpasquier]

/lgtm

should we get an official ack from OpenShift architects?

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lilic, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lilic,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[lilic]

@simonpasquier thanks!

should we get an official ack from OpenShift architects?

We got an 👍 from Derek already, before that on forum-arch folks said yes to it, on both the name and this approach and it can be used by other user workloads in the future. Just location was never agreed upon, so for now we create it, and if other components want to consume it we can move it. SGTY?

[simonpasquier]

@lilic this is fine by me then! I wasn't sure reading the enhancement proposal and comments that everything was already agreed :)

[lilic]

@simonpasquier Yep was lots of comments :) openshift/enhancements#369 (comment) and openshift/enhancements#369 (comment) are the main comments, everything else is specific to the priority class being applied to other components or its location. There was also a lot of out of band discussions on slack.