Add ExtendedResourceToleration admission controller. #55839

rohitagarwal003 · 2017-11-16T03:40:19Z

/kind feature
/sig scheduling
/area hw-accelerators

There's elaborate discussion on this in #55080. In short, we would like to enable cluster operators and/or cloud providers to create dedicated nodes with extended resources (like GPUs, FPGAs etc.) that are reserved for pods requesting such resources. Taints is the kubernetes concept to create dedicated nodes. If the cluster operator or cloud provider wants to create dedicated node pools, they are expected to taint the nodes containing extended resources with the key equal to the name of the resource and effect equal to NoSchedule. If they do that, only pods that have a toleration for such a taint can be scheduled there. To make it easy for the user, this admission controller when enabled, automatically adds a toleration with key example.com/device, operator Exists and effect NoSchedule if an extended resource of name example.com/device is requested.

Release note:

Add ExtendedResourceToleration admission controller. This facilitates creation of dedicated nodes with extended resources. If operators want to create dedicated nodes with extended resources (like GPUs, FPGAs etc.), they are expected to taint the node with extended resource name as the key. This admission controller, if enabled, automatically adds tolerations for such taints to pods requesting extended resources, so users don't have to manually add these tolerations.

rohitagarwal003 · 2017-11-16T03:40:57Z

/assign @vishh @davidopp @bsalamat @jiayingz

rohitagarwal003 · 2017-11-16T03:42:51Z

/assign @derekwaynecarr @liggitt

jiayingz · 2017-11-16T21:28:46Z

/lgtm

jberkus · 2017-11-17T16:27:49Z

Hey, release team reminder. If this PR is approved, please remember to add the 1.9 milestone and approve it for that milestone by Monday, Nov. 20th to make the Code Slush cutoff. Thanks!

vishh · 2017-11-17T23:13:28Z

plugin/pkg/admission/extendedresourcetoleration/admission.go

+	}
+
+	for _, toleration := range pod.Spec.Tolerations {
+		if len(toleration.Key) == 0 && len(toleration.Value) == 0 &&


nit: Can this be a function that is something like func podHasWildcardToleration(*api.pod) bool?

I suspect people don't use wildcard toleration much, and also it might be easier from an understandability perspective (both of the code and someone looking at the pod) if you always add the toleration. So I would suggest to just remove this special case where the wildcard toleration already exists.

vishh · 2017-11-17T23:18:29Z

/lgtm

k8s-github-robot · 2017-11-18T20:24:26Z

[MILESTONENOTIFIER] Milestone Pull Request Current

@bsalamat @davidopp @derekwaynecarr @jiayingz @liggitt @mindprince @vishh

Pull Request Labels

sig/scheduling: Pull Request will be escalated to these SIGs if needed.
priority/important-soon: Escalate to the pull request owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
kind/feature: New functionality.

Help

rohitagarwal003 · 2017-11-19T01:34:04Z

/retest

davidopp · 2017-11-19T01:59:02Z

/lgtm

dims · 2017-11-19T14:24:42Z

/assign @liggitt

caesarxuchao

A nit. Fix in follow-up PR is fine.

caesarxuchao · 2017-11-20T18:46:36Z

plugin/pkg/admission/extendedresourcetoleration/admission.go

+
+// Admit updates the toleration of a pod based on the resources requested by it.
+// If an extended resource of name "example.com/device" is requested, it adds
+// a toleration with key "example.com/device", operator "Exists" and effect "NoSchedule".


The result is that such pods can be scheduled to nodes with the "example.com/device" taint, right? Consider adding that in the comment and please also add this plugin to the doc.

Yes. I will send another PR with the comment fix. And another one to update the public documentation.

are arbitrary resources allowed? does this give pod authors a way to cause arbitrary taints to be added? I expected a specific prefix/namespace to ensure resource-specific taints were tolerated, not arbitrary taints

It only adds tolerations for the resource you requested and that resource cannot be in kubernetes.io namespace.

Now you can request an arbitrary string as the resource (and then this will add that arbitrary string as the toleration) but then your pod will not get scheduled because that arbitrary string doesn't exist as a resource. And if the arbitrary string exists as a resource, we do want this admission controller to add that as a toleration.

Now you can request an arbitrary string as the resource (and then this will add that arbitrary string as the toleration) but then your pod will not get scheduled because that arbitrary string doesn't exist as a resource.

Thinking through this some more:

by default, users can already add arbitrary tolerations to their pod specs

any cluster using taints/tolerations as a security measure to keep user pods off specific nodes already has to use something like PodTolerationRestriction to whitelist the tolerations a pod is allowed to have

that whitelist would give control over allowing custom-resource tolerations while disallowing arbitrary security-related tolerations

So I think this still works as is

caesarxuchao · 2017-11-20T18:50:26Z

/approve no-issue

caesarxuchao · 2017-11-20T18:52:12Z

Release note is user facing. Please add a brief description of what the plugin does and who needs it.

rohitagarwal003 · 2017-11-20T19:02:50Z

Updated the release note with more details. When the final release notes doc will get generated, I would link this release note to the documentation in kubernetes.io

caesarxuchao · 2017-11-20T19:05:40Z

plugin/pkg/admission/extendedresourcetoleration/admission.go

+	resources := sets.String{}
+	for _, container := range pod.Spec.Containers {
+		for resourceName := range container.Resources.Requests {
+			if helper.IsExtendedResourceName(resourceName) {


@mindprince, regarding @liggitt's question, this line ensures that the resourceName (and the toleration key) has a "kubernetes.io/" or a "pod.alpha.kubernetes.io/opaque-int-resource-" prefix. The "pod.alpha.kubernetes.io/opaque-int-resource-" prefix seems strong enough. But the "kubernetes.io/" seems very generic, will it collide with other taint key name?

It ensures the opposite. It ensures that the resource name doesn't have kubernetes.io as the prefix.

Right. I missed the !.

caesarxuchao · 2017-11-20T19:06:03Z

/hold for @liggitt's question

caesarxuchao · 2017-11-20T19:07:50Z

/hold

liggitt · 2017-11-20T19:20:25Z

/hold cancel

bsalamat · 2017-11-20T19:25:42Z

/lgtm

k8s-github-robot · 2017-11-20T19:25:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, caesarxuchao, davidopp, jiayingz, mindprince, vishh

Associated issue: 55080

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~cmd/kube-apiserver/OWNERS~~ [caesarxuchao]
~~plugin/OWNERS~~ [davidopp]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-11-20T19:36:43Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-11-20T20:28:36Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

dixudx · 2017-12-08T03:12:17Z

Update this admission controller in doc kubernetes/website#6618.

k8s-ci-robot assigned bsalamat, davidopp, jiayingz and vishh Nov 16, 2017

k8s-ci-robot assigned derekwaynecarr and liggitt Nov 16, 2017

k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 16, 2017

rohitagarwal003 force-pushed the extended-resource-toleration branch from 138e76c to 6471801 Compare November 17, 2017 09:01

k8s-github-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2017

vishh reviewed Nov 17, 2017

View reviewed changes

vishh added this to the v1.9 milestone Nov 17, 2017

vishh added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Nov 17, 2017

k8s-github-robot added the milestone/needs-approval label Nov 17, 2017

k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2017

davidopp added status/approved-for-milestone and removed milestone/needs-approval labels Nov 18, 2017

rohitagarwal003 force-pushed the extended-resource-toleration branch from 6471801 to 0b9f41f Compare November 19, 2017 00:47

k8s-github-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2017

Add ExtendedResourceToleration admission controller.

3c4c85f

rohitagarwal003 force-pushed the extended-resource-toleration branch from 0b9f41f to 3c4c85f Compare November 19, 2017 00:48

k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2017

caesarxuchao reviewed Nov 20, 2017

View reviewed changes

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 20, 2017

caesarxuchao reviewed Nov 20, 2017

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2017

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2017

k8s-github-robot merged commit b3f7ad7 into kubernetes:master Nov 20, 2017

rohitagarwal003 mentioned this pull request Nov 21, 2017

Built in support for dedicated nodes with extended compute resources (auto toleration) #55080

Closed

rohitagarwal003 mentioned this pull request Nov 30, 2017

Add wildcard tolerations to kube-proxy #56589

Merged

dixudx mentioned this pull request Dec 8, 2017

add ExtendedResourceToleration admission controller kubernetes/website#6618

Merged

Add ExtendedResourceToleration admission controller. #55839

Add ExtendedResourceToleration admission controller. #55839

rohitagarwal003 commented Nov 16, 2017 •

edited

rohitagarwal003 commented Nov 16, 2017

rohitagarwal003 commented Nov 16, 2017

jiayingz commented Nov 16, 2017

jberkus commented Nov 17, 2017

vishh Nov 17, 2017

davidopp Nov 18, 2017

rohitagarwal003 Nov 19, 2017

vishh commented Nov 17, 2017

k8s-github-robot commented Nov 18, 2017

rohitagarwal003 commented Nov 19, 2017

davidopp commented Nov 19, 2017

dims commented Nov 19, 2017

caesarxuchao left a comment

caesarxuchao Nov 20, 2017

rohitagarwal003 Nov 20, 2017

liggitt Nov 20, 2017

rohitagarwal003 Nov 20, 2017

liggitt Nov 20, 2017

caesarxuchao commented Nov 20, 2017

caesarxuchao commented Nov 20, 2017

rohitagarwal003 commented Nov 20, 2017

caesarxuchao Nov 20, 2017

rohitagarwal003 Nov 20, 2017

caesarxuchao Nov 20, 2017

caesarxuchao commented Nov 20, 2017

caesarxuchao commented Nov 20, 2017

liggitt commented Nov 20, 2017

bsalamat commented Nov 20, 2017

k8s-github-robot commented Nov 20, 2017

k8s-github-robot commented Nov 20, 2017

k8s-github-robot commented Nov 20, 2017

dixudx commented Dec 8, 2017

Add ExtendedResourceToleration admission controller. #55839

Add ExtendedResourceToleration admission controller. #55839

Conversation

rohitagarwal003 commented Nov 16, 2017 • edited

rohitagarwal003 commented Nov 16, 2017

rohitagarwal003 commented Nov 16, 2017

jiayingz commented Nov 16, 2017

jberkus commented Nov 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vishh commented Nov 17, 2017

k8s-github-robot commented Nov 18, 2017

rohitagarwal003 commented Nov 19, 2017

davidopp commented Nov 19, 2017

dims commented Nov 19, 2017

caesarxuchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao commented Nov 20, 2017

caesarxuchao commented Nov 20, 2017

rohitagarwal003 commented Nov 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao commented Nov 20, 2017

caesarxuchao commented Nov 20, 2017

liggitt commented Nov 20, 2017

bsalamat commented Nov 20, 2017

k8s-github-robot commented Nov 20, 2017

k8s-github-robot commented Nov 20, 2017

k8s-github-robot commented Nov 20, 2017

dixudx commented Dec 8, 2017

rohitagarwal003 commented Nov 16, 2017 •

edited