Guaranteed Scheduling For Critical Pods

[palma21]

Problem Summary:

Users expect their critical system pods to always be scheduled as needed regardless of any actions they take, like tainting nodes for workload specific scheduling.
Today because AKS and AKS-engine system pods are controlled by the addon manager and the default pod spec enforced by it does not use any tolerations or priority this leads to the pods being evicted after for example a taint and/or not be scheduled in the tainted nodes anymore.

Proposed Solution

Add PriorityClasses and wildcard tolerations to the critical system pods