Revert changes made to eviction and qos

[derekwaynecarr]

What this PR does / why we need it:
This PR reverts changes that were made to qos and eviction designs that extended the meaning of a critical pod annotation.

#38836
#39059
#39114

Which issue this PR fixes
Addresses #40024 for the master branch.

Special notes for your reviewer:
For details on why this is being done, see #38322 (comment)

Release note:

Revert: Admit critical pods in the kubelet
Revert: Don't evict static pods
Revert: assign -998 as the oom_score_adj for critical pods (e.g. kube-proxy)

[k8s-reviewable]

This change is 

[derekwaynecarr]

/cc @dchen1107 @vishh @davidopp @bprashanth @liggitt @smarterclayton @thockin @matchstick

[k8s-cherrypick-bot]

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

[derekwaynecarr]

This needs to be cherry-picked to release-1.4 and release-1.5.

[vishh]

Can we instead make these changes gated behind a flag? If we revert them, kube-proxy will get evicted due to very many reasons including that of eviction bugs.

[dchen1107]

@derekwaynecarr I provided some background context and the rough risk analysis at #38322 (comment)

I am ok with reverting this from main branch which is the initial goal @bprashanth and I discussed. But the production is bleeding if we revert those commits from 1.4 and 1.5 branches unless we have alternative solution.

[derekwaynecarr]

@dchen1107 - i would like to revert master. ideally, i would like to revert 1.4 and 1.5. there are a number of pro/cons in any cluster deployment choice that could impact operators and users in production. the deployment model should not have an impact on the entire community in the manner this PR did. we basically invented a workload priority design really quickly, and rapidly pushed it out to present and prior releases with little to no community discussion or documentation to support one preferred deployment choice. i worry this establishes a precedent that i would not like to see followed in the future.

if reverting in 1.4 and 1.5 is not agreeable, can a member of your team flag-gate the feature behind an experimental flag in 1.4 and 1.5 (default to false) and reduce the scope of the annotation to only be meaningful in the kubelet if the pod is in the kube-system namespace?

as for this pr, i think it should merge as-is in master and the experimental flag should never appear in 1.5.

[k8s-github-robot]

@derekwaynecarr PR needs rebase

[derekwaynecarr]

Typo in previous comment, I think experimental flag should not be needed in 1.6... too many releases in my head

[vishh]

@derekwaynecarr A fix for solving static pod evictions will not land in v1.6 since it involves quite a few changes (moving static pods to Daemon Set, supporting daemon set upgrades, rightsizing kube proxy, switching to Guaranteed QoS, etc). Thankfully most of these missing features/bugs are being addressed in the v1.6 time frame.
For v1.6 (and the previous releases), I'd prefer adding a flag gate (disabled by default) for eviction changes including critical pod annotation & static pod eviction policy.

[vishh]

Do we need this revert to go in as-is before the flag gates are added? I'd prefer adding the flag gates directly.

[thockin]

I spoke to @smarterclayton on slack, too - please let's pursue a flag gate. I did not realize that OpenShift didn't have the same problems (I certainly heard it from several customers not just GKE),a nd I ACK that I didn't give enough thought to multi-tenant scenarios. My brain is bad and I feel bad. We'll accelerate the *real* solution and make sure that you are approver.

…

On Tue, Jan 17, 2017 at 4:46 PM, Vish Kannan ***@***.***> wrote: Do we need this revert to go in as-is before the flag gates are added? I'd prefer adding the flag gates directly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#40025 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVOSn9-xrcddpNqRRQsV6I3FrmU12ks5rTWDzgaJpZM4Ll1JB> .

[liggitt]

please let's pursue a flag gate

even with the flag enabled, the effect of the annotation needs to be limited to static pods in the kube-system namespace (or something equally restrictive). users should never be able to end-run existing qos/quota controls via annotations in the API

[thockin]

Without a real priority scheme in place yet, we need to guarantee that things which are actually critical to the operation of the cluster (e.g. DNS) are always run. For better or for worse, we're behind the curve on priority, and people are pushing clusters harder than we anticipated. When DNS gets de-scheduled, nobody has a good time.

…

On Wed, Jan 18, 2017 at 7:54 AM, Jordan Liggitt ***@***.***> wrote: please let's pursue a flag gate even with the flag enabled, the effect of the annotation needs to be limited to static pods in the kube-system namespace (or something equally restrictive). users should never be able to end-run existing qos/quota controls via annotations in the API — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#40025 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVEWON-H3bBCdsrkQmNHlyAEJWTptks5rTjW7gaJpZM4Ll1JB> .

[liggitt]

then at least limit the power to a privileged namespace (along with the flag gate)

[k8s-github-robot]

[APPROVALNOTIFIER] Needs approval from an approver in each of these OWNERS Files:

• cluster/OWNERS
• pkg/kubelet/OWNERS

We suggest the following people:
cc @mikedanese @derekwaynecarr
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[derekwaynecarr]

closing in favor of #40655