Taint Based Eviction

[davidopp]

Feature Description

• One-line feature description (can be used as a release note):
• Primary contact (assignee): @gmarek
• Responsible SIGs: @kubernetes/sig-scheduling-feature-requests
• KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20200127-taint-based-evictions.md
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y)
  • Beta release target (x.y) 1.13
  • Stable release target (x.y)

[davidopp]

This is finished except for documentation.

NoExecute taint effect is now in Beta (as part of moving taints/tolerations to Beta), and taint-based eviction for node problems is in Alpha.

The PRs involved were:

• Forgiveness api changes kubernetes#39469: Forgiveness (tolerationSeconds) API changes
• Forgiveness library changes kubernetes#39914: Forgiveness (tolerationSeconds) library changes
• add defaultTolerationSeconds admission controller kubernetes#41414 and enable DefaultTolerationSeconds admission controller by default kubernetes#41815: defaultTolerationSeconds admission controller
• Make DaemonSets survive taint-based evictions when nodes turn unreachable/notReady kubernetes#41896: Make DaemonSets survive taints
• TaintController kubernetes#40355: TaintController
• Promote taint helpers from e2e/framework to pkg/controller/controller_utils kubernetes#41014: Refactoring
• NodeController sets NodeTaints instead of deleting Pods kubernetes#41133: NodeController sets node taints instead of deleting pods
• Switch to SVG test-infra#197: Enable e2e test suite in Jenkins

[idvoretskyi]

@davidopp @gmarek @kevin-wangzefeng please, provide us with the release notes and documentation PR (or links) at the features spreadsheet.

[davidopp]

Documentation PR out for review at https://github.com/kubernetes/kubernetes.github.io/pull/2774/files

[davidopp]

ref/ kubernetes/kubernetes#43815 (comment)

[davidopp]

Regarding "taint-based eviction for node problems is in Alpha": nobody is available to move it to beta in 1.7, so it will stay in alpha in 1.7.

[gyliu513]

@davidopp I can help on this and target for 1.7, one question is that I saw kubernetes/kubernetes#40355 is already enabled by default, so can you please explain more for what do you mean by move this to beta in 1.7?

[gmarek]

@gyliu513 - taint controller is enabled by default, but using taints instead of direct evictions in case of node problems isn't. To move it to beta there's at least one thing to be done (except renaming stuff from alpha to bete), which is to rewrite/write new NodeController unit tests, as they currently assume direct evictions. There are some very basic tests for taint-based evictions, but they should be drastically extended (e.g. cover all master-disruption logic).

[davidopp]

I think we can only do this in 1.7 if @gmarek has the bandwidth to do all the reviews and define what we need to do to move it to beta. (Sounds like he's already done most of the second thing above.)

@gmarek do you have time?

[gmarek]

Yes, I can find time for reviews and I can find time to think through what needs to be done, if there's someone willing to work on it.

[gyliu513]

@davidopp @gmarek I will be the volunteer for this, can you please assign this to me? @gmarek will go through this feature and propose sth to you

[davidopp]

assigned

[gmarek]

@gyliu513 OK - let me know if you need some directions.

[idvoretskyi]

@davidopp @gmarek I've updated the feature description to fit the new template. Please, fill the empty fields in the new template (their actual state was unclear).

[davidopp]

Move to Beta is goal for 1.7.

[gyliu513]

@gmarek I want to split the work to two tasks:

1. Rename from alpha to beta.
2. Update unit test for node controller to cover more cases.

Comments?

[palnabarun]

I went ahead and updated the issue comment with the link to the KEP.

[Ye-Tian-Zero]

@damemi Hi, is the progress of moving #1450 from scheduling to node still move on? Will it be done before v1.18 KEP freeze?

[damemi]

@skilxn-go I opened a PR to move the KEP here: #1510

[Ye-Tian-Zero]

Thanks, got it

[palnabarun]

Hi @damemi, just a friendly reminder that the Code Freeze will go into effect on Thursday 5th March.

Can you please link all the k/k PRs or any other PRs which should be tracked for this enhancement?

Thank You :)

[damemi]

Hi @palnabarun, we have an umbrella issue which links to the issues/PRs that are in the works for this: kubernetes/kubernetes#87161

[palnabarun]

Thank you @damemi for updating this. :)

[sethmccombs]

Hey @damemi -

Seth here, Docs shadow on the 1.18 release team.

Does this enhancement work planned for 1.18 require any new docs or modifications to existing docs?

If not, can you please update the 1.18 Enhancement Tracker Sheet (or let me know and I'll do so)

If doc updates are required, reminder that the placeholder PRs against k/website (branch dev-1.18) are due by Friday, Feb 28th.

Let me know if you have any questions!

[ingvagabund]

@sethmccombs IIUC, given I have the doc PR opened against dev-1.18 branch (kubernetes/website#19302), there's no need to update any sheet, right?

[sethmccombs]

@ingvagabund you got it, I'll update the Enhancement tracking sheet!

[palnabarun]

Hi @damemi, this a reminder that we are just two days away from Code Freeze on 5th March.

By the Code Freeze, all the relevant PR's should be merged else you would need to file an exception request.

[damemi]

@palnabarun I've updated the 3 PRs that this is waiting on to see if they will merge by code freeze.

1. cleanup: move the test of TaintBasedEvictions features to sig-node kubernetes#88152
2. Move TaintBasedEvictions feature gates to GA kubernetes#87487
3. Taint based eviction promoted to GA in 1.18 website#19302

[damemi]

@palnabarun actually before those 3 can merge, we need to get the KEP move approved: #1510

[palnabarun]

@damemi I see that the PR's are blocked on approvals at the moment. Do you think they would make it before the deadline?

Today EOD is the Code Freeze

Please file an exception if you think the PR's might slip the deadline.

[damemi]

I think we will need more time to get the approvals, what's the process to file an exception?

[jeremyrickard]

@damemi, you can find the process here https://github.com/kubernetes/sig-release/blob/master/releases/EXCEPTIONS.md

[damemi]

@jeremyrickard thanks, exception filed

[palnabarun]

@damemi The exception request was approved. :)

[palnabarun]

Hi @damemi, since this enhancement graduated to Stable this release 🚀, the status can now be set to be Implemented.

Can you please update the status? After that, we will close this issue.

[damemi]

@palnabarun sure, opened that here: #1625

[palnabarun]

Thank you @damemi :)

[palnabarun]

The corresponding enhancement has graduated to Stable. 🥳

Closing this issue.

/close

[k8s-ci-robot]

@palnabarun: Closing this issue.

In response to this:

The corresponding enhancement has graduated to Stable. 🥳

Closing this issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[palnabarun]

Edit: The issue description here had a 404 link. Pointed it to the correct one.