chore: add priorityClassName: system-node-critical to kube-system,…

[jackfrancis]

… master-schedulable components

Reason for Change:

For kube-system components that are declared to tolerate master node taints, we want to add priorityClassName: system-node-critical as well to doubly ensure that they are schedulable, even if the master node taints are customized. Otherwise, master node taint customization is liable to make system-critical components unschedulable.

Included all components that fit the above description for k8s 1.9 and above.

Reference: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/

Issue Fixed:

Fixes #573

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

Merging #555 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #555      +/-   ##
==========================================
+ Coverage   54.75%   54.79%   +0.03%     
==========================================
  Files          97       97              
  Lines       14708    14716       +8     
==========================================
+ Hits         8054     8064      +10     
+ Misses       5980     5979       -1     
+ Partials      674      673       -1

[mboersma]

lgtm

[jackfrancis]

omsagent cluster scenario manually tested

[jackfrancis]

flannel scenario manually tested

[palma21]

I see you used system-cluster-critical and not system-node-critical any reason for that?

Also to fix the full problem scope I believe we still need a wildcard toleration for kube-proxy at least.
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"

[palma21]

Add-on for the reasons:

Adding priorityClass to ensure that kube-system components are schedulable when node taints are customized and added to any node.

Fixes #573

[jackfrancis]

@palma21 does the most recent commit address the desired add'l kube-proxy tolerations?

[palma21]

I believe there's a bug here with the indentation that is making the tests fail

[serbrech]

This is to ensure that we never schedule kube-proxy on master, correct?

[jackfrancis]

It's the opposite: it ensures that kube-proxy will always be able to be scheduled on any node, regardless of taint.

[CecileRobertMichon]

lgtm