Bug 1705649 : Cluster with halted master did not reschedule operators after 5m of being down

[ravisantoshgudimetla]

As of now, because of infinite tolerations against all the possible taints, we are seeing that operators are not getting evicted from nodes that have NoExecute taint on them. This PR tightens the conditions around which can be pods can be scheduled/evicted. The downside is there is a very good chance that pods would be evicted from nodes that have certain conditions like disk-pressure, memory-pressure, taints added by other controllers(operators) etc. So, please make sure that this change is ok with your operator/operand before merging this PR.

/cc @sjenning @smarterclayton

[mfojtik]

@ravisantoshgudimetla kube apiserver is static pod... I heard from @sjenning that static pods can't be evicted (ever). So I wonder if this PR makes sense for static pods (KAS, KCM and KSM)

[deads2k]

/lgtm

[ravisantoshgudimetla]

kube apiserver is static pod... I heard from @sjenning that static pods can't be evicted (ever). So I wonder if this PR makes sense for static pods (KAS, KCM and KSM)

Static pods can be evicted(by kubelet if they're not critical pods) but we have decided not to apply tolerations to the static. I will remove the tolerations for them soon.

[ravisantoshgudimetla]

/hold

[ravisantoshgudimetla]

/hold cancel

[mfojtik]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, mfojtik, ravisantoshgudimetla

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [deads2k,mfojtik]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ravisantoshgudimetla]

/retest

[ravisantoshgudimetla]

error: could not run steps: step e2e-aws-operator failed: template pod "e2e-aws-operator" failed: the pod ci-op-jick3xv8/e2e-aws-operator failed after 53m39s (failed containers: test): ContainerFailed one or more containers exited

FAIL: github.com/openshift/cluster-kube-apiserver-operator/test/e2e TestNamedCertificates/User_three.test 1m0.25s FAIL: github.com/openshift/cluster-kube-apiserver-operator/test/e2e TestNamedCertificates 3m8.1s

@mfojtik @deads2k, seeing the above error in e2e-operator failures ^

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[ravisantoshgudimetla]

ns/openshift-machine-config-operator pod/etcd-quorum-guard-57cff7b968-jplzg 0/6 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't satisfy existing pods anti-affinity rules, 3 node(s) didn't match node selector. (61 times)

/cc @RobertKrawitz

[eparis]

/retest

[RobertKrawitz]

Ref https://bugzilla.redhat.com/show_bug.cgi?id=1707212#c6

[smarterclayton]

As we get closer to release, please ensure code changes have a bug and the bug is associated in the PR title - follow the conventions described in previous emails about how to associate bugs with PRs. The PR title must be Bug XXXX: <description>.

This PR didn't get correct title because it was Bug XXXXXX : (extra space)