Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerConfig #3546

whs · 2017-10-05T19:38:44Z

No description provided.

k8s-ci-robot · 2017-10-05T19:38:52Z

Hi @whs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

chrislovecnm

Thanks for the PR! Couple of questions. Which versions of k8s support the configmap? If we have support at the beginning of a specific version, we will need validation. Does the scheduler start without the configmap in place? Can you add some documentation in the configspec markdown doc under docs?

Always appreciate the help!

chrislovecnm · 2017-10-06T01:08:28Z

pkg/apis/kops/componentconfig.go

@@ -330,6 +330,10 @@ type KubeSchedulerConfig struct {
 	Image string `json:"image,omitempty"`
 	// LeaderElection defines the configuration of leader election client.
 	LeaderElection *LeaderElectionConfiguration `json:"leaderElection,omitempty"`
+	// Name of configmap to use for scheduler policy
+	PolicyConfigMap string `json:"policyConfigMap,omitempty" flag:"policy-configmap"`


Can we re-word the API elements into the typical go syntax for comments? The first word of the comment is the name, in this case PolicyConfigMap.

See https://golang.org/doc/effective_go.html#commentary. Comments documenting declarations should be full sentences, even if that seems a little redundant. This approach makes them format well when extracted into godoc documentation. Comments should begin with the name of the thing being described and end in a period:

chrislovecnm · 2017-10-06T01:10:48Z

/ok-to-test

whs · 2017-10-06T01:26:35Z

This feature was added in 1.7.0 (Scheduler can recieve its policy configuration from a ConfigMap kubernetes#43892). Reading that I'll need to add a note that the scheduler must be restarted manually when configmap is modified
The scheduler will exit if the configmap is not found. I'll add that to the comment, but do we have some way to validate it?
Could you point to where the docs for kube-scheduler is? I only see kubelet options is documented in the cluster spec and the other cluster services are undocumented. If we don't have one already I can add it.

chrislovecnm · 2017-10-06T01:54:30Z

This feature was added in 1.7.0 (kubernetes/kubernetes#43892). Reading that I'll need to add a note that the scheduler must be restarted manually when configmap is modified

So we have a bit of a chicken and egg problem with the cluster starting properly.
@justinsb / @blakebarnett any ideas on how to address this? If the configmap does not exist, the scheduler will not start. Without the scheduler starting, cni will not install. Wonder if we can add a default config map in addons and then have that overriden? Interesting problem to solve.

@whs the chicken and egg problem is that in a usual case we need to cluster to start. The cluster depends on the scheduler to get components like CNI running, which is a base component. Would be optimal if this configmap was reloadable.

We will need to validate that the k8s cluster is over 1.7.x. You can see how we are validating at a component level here: https://github.com/kubernetes/kops/blob/master/pkg/model/components/kubecontrollermanager.go

There is a go file under the same package for the scheduler.

The scheduler will exit if the configmap is not found. I'll add that to the comment, but do we have some way to validate it?

We can validate that the k8s cluster version is compatible with the cli option. The chicken and egg problem with k8s being up and the configmap is kinda bugging me.

Could you point to where the docs for kube-scheduler is? I only see kubelet options is documented in the cluster spec and the other cluster services are undocumented. If we don't have one already I can add it.

K8s or kops?

whs · 2017-10-06T02:26:14Z

K8s or kops?

In this file only kubelet is documented. Are there other place in kops docs that this option should be documented or should I add to that file?

chrislovecnm · 2017-10-06T02:37:32Z

@whs we do not have full documentation in that markdown file, but when new changes come in we try to add them there. Just cut a new section for the scheduler.

whs · 2017-10-06T11:01:27Z

If we throw an error when this key is present during cluster creation, only allowing it with update, would that work?

…

On Oct 6, 2017 9:37 AM, "Chris Love" ***@***.***> wrote: @whs <https://github.com/whs> we do not have full documentation in that markdown file, but when new changes come in we try to add them there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3546 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAVFi_AoAcBqgcfarbWUKaIlcNf9Gumgks5spZJxgaJpZM4Pvl4Q> .

whs · 2017-10-06T17:31:34Z

All pending concerns should be now resolved, except for that the cluster still won't start with this flag on. Either we do something about that or just document it.

justinsb · 2017-10-09T06:51:05Z

So this PR as it stands is good. We can map flags, and you've done that perfectly (you've even matched the casing in pkg/apis/componentconfig/v1alpha1/types.go in k/k, which is great, because eventually we'll want to reconcile there so if we can avoid differences so much the better).

So the question is where should the configuration live, I guess. Is there a default scheduler configuration we could create? (just as we create a default LimitRange, for example: https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/limit-range.addons.k8s.io/v1.5.0.yaml ). This feels like the first step towards componentconfig, which is super exciting.

I'm guessing what we should do is to automatically create a default scheduler configmap (perhaps in 1.9), and point the scheduler to read from that configmap. We'd have to make sure we didn't replace it on kops/kubernetes updates I guess. Does that sound right @whs ?

/lgtm

whs · 2017-10-09T07:07:37Z

The default config for kube-scheduler is hardcoded in scheduler's source. It should be like this if written as JSON.

I'm not sure that providing a default scheduler configmap would be the best solution here as we would have to keep track of scheduler default policy update, and update the file. (for example, when pod affinity it was implemented by adding a predicate and priority) Also, it leads to the question on what should we do if the user updated the file and upgrade the cluster.

To be clear, if the user doesn't provide --policy-configmap the scheduler would just start with the default configuration. The only problem is when this option is present, but the referenced configmap is not present.

Providing default configmap if the user specified this option is one way we could make this work, but it would be complicated. Since this is an option for advanced users, I think noting in the docs and godoc that this option will not work on cluster creation should be fine.

Is this ok with you? And is it possible to detect cluster creation when validating the spec so I can add this check in the code as well?

…nfig

…Kube 1.7.x

k8s-github-robot · 2017-10-10T02:12:12Z

/lgtm cancel //PR changed after LGTM, removing LGTM. @justinsb @whs

…uster creation

whs · 2017-10-10T02:34:40Z

I added the docs. I think we're ready to merge.

justinsb · 2017-10-10T02:47:41Z

/lgtm

Hopefully they'll add reloading once it becomes clearer how reloading should work :-)

k8s-github-robot · 2017-10-10T02:48:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~OWNERS~~ [justinsb]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

chrislovecnm · 2017-10-10T02:56:08Z

@whs something I thought about. We could have kops manage the config map.

whs · 2017-10-10T02:59:30Z

That could work, and was pointed out by @justinsb earlier as well. As this option is optional (if you don't specify anything the scheduler have its hardcoded default), I think adding kops-managed scheduler policy would add more burden to the kops maintainers, as we need to keep track of upstream policy changes.

chrislovecnm · 2017-10-10T03:02:50Z

@whs only create the config map if the user wants it. And set the scheduler flag. We have dynamic addons. Let me know if you are interested in how to do this.

k8s-github-robot · 2017-10-10T03:08:02Z

/test all [submit-queue is verifying that this PR is safe to merge]

whs · 2017-10-10T03:27:13Z

I think I have some idea now. Give me an hour..

@Justindb

per @Justindb

k8s-github-robot · 2017-10-10T03:41:51Z

Automatic merge from submit-queue.

Automatic merge from submit-queue. UsePolicyConfigMap for kube-scheduler Continued from #3546 In this version, a single option `usePolicyConfigMap` is added that will install scheduler.addons.k8s.io, which contains a default configmap.

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 5, 2017

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 5, 2017

chrislovecnm previously requested changes Oct 6, 2017

View reviewed changes

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 6, 2017

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 6, 2017

k8s-ci-robot assigned justinsb Oct 9, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 9, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 9, 2017

whs added 5 commits October 10, 2017 09:10

Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerCo…

091e87e

…nfig

Regenerated API

0f2dde9

Updated PolicyConfigMap comment

78023ba

Added docs on policyConfigMap

be8aa9f

Added error when trying to use kube-scheduler policyConfigMap before …

c00f5ea

…Kube 1.7.x

whs force-pushed the master branch from 40ee4ce to c00f5ea Compare October 10, 2017 02:10

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 10, 2017

Added documentation that PolicyConfigMap should not be used during cl…

a06fbba

…uster creation

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 10, 2017

k8s-github-robot merged commit d7d62b8 into kubernetes:master Oct 10, 2017

whs mentioned this pull request Oct 10, 2017

UsePolicyConfigMap for kube-scheduler #3581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerConfig #3546

Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerConfig #3546

whs commented Oct 5, 2017

k8s-ci-robot commented Oct 5, 2017

chrislovecnm left a comment

chrislovecnm Oct 6, 2017

chrislovecnm commented Oct 6, 2017

whs commented Oct 6, 2017

chrislovecnm commented Oct 6, 2017

whs commented Oct 6, 2017

chrislovecnm commented Oct 6, 2017 •

edited

Loading

whs commented Oct 6, 2017 via email

whs commented Oct 6, 2017

justinsb commented Oct 9, 2017

whs commented Oct 9, 2017

k8s-github-robot commented Oct 10, 2017

whs commented Oct 10, 2017

justinsb commented Oct 10, 2017

k8s-github-robot commented Oct 10, 2017

chrislovecnm commented Oct 10, 2017

whs commented Oct 10, 2017

chrislovecnm commented Oct 10, 2017

k8s-github-robot commented Oct 10, 2017

whs commented Oct 10, 2017

k8s-github-robot commented Oct 10, 2017

Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerConfig #3546

Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerConfig #3546

Conversation

whs commented Oct 5, 2017

k8s-ci-robot commented Oct 5, 2017

chrislovecnm left a comment

Choose a reason for hiding this comment

chrislovecnm Oct 6, 2017

Choose a reason for hiding this comment

chrislovecnm commented Oct 6, 2017

whs commented Oct 6, 2017

chrislovecnm commented Oct 6, 2017

whs commented Oct 6, 2017

chrislovecnm commented Oct 6, 2017 • edited Loading

whs commented Oct 6, 2017 via email

whs commented Oct 6, 2017

justinsb commented Oct 9, 2017

whs commented Oct 9, 2017

k8s-github-robot commented Oct 10, 2017

whs commented Oct 10, 2017

justinsb commented Oct 10, 2017

k8s-github-robot commented Oct 10, 2017

chrislovecnm commented Oct 10, 2017

whs commented Oct 10, 2017

chrislovecnm commented Oct 10, 2017

k8s-github-robot commented Oct 10, 2017

whs commented Oct 10, 2017

k8s-github-robot commented Oct 10, 2017

chrislovecnm commented Oct 6, 2017 •

edited

Loading