[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins #83694

ahg-g · 2019-10-09T19:30:41Z

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind feature

What this PR does / why we need it:

Plumped handlers necessary to instantiate some predicates/priorities. This is necessary during phase 1 of framework migration.

Which issue(s) this PR fixes:
Fixes #83687

NONE

k8s-ci-robot · 2019-10-09T19:31:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-scheduler/OWNERS~~ [ahg-g]
~~pkg/scheduler/OWNERS~~ [ahg-g]
~~test/integration/scheduler/OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hex108 · 2019-10-10T01:46:04Z

pkg/scheduler/framework/plugins/default_registry.go

 )

+// DefaultRegistryArgs arguments needed to create default plugin factories.
+type DefaultRegistryArgs struct {


I prefer to name it to RegistryArgs, because when declaring a DefaultRegistryArgs variable does not really give you a default registry args by default.

This is args for default registry, not default RegistryArgs, but I see how this can be interpreted the other way, I changed it as you suggested.

hex108 · 2019-10-10T01:56:58Z

pkg/scheduler/scheduler.go

+		time.Duration(options.bindTimeoutSeconds)*time.Second)
+
+	if options.frameworkRegistry == nil {
+		options.frameworkRegistry = frameworkplugins.NewDefaultRegistry(&frameworkplugins.DefaultRegistryArgs{


If we create a new function(e.g. NewRegistryArgs), we could call it from cmd/kube-scheduler/app/server.go, then we do not need to disable out-of-tree framework plugins.

that will require us to define global variables for the cache and volume binder, which is a pain to deal with. The framework is changing so fast right now there is no point in actually supporting out-of-tree plugins before beta.

that will require us to define global variables for the cache and volume binder

I could not get your point why we need to define global variables for them, we could still declare them as local variables and use them as following. Is there any problem?

schedulerCache := internalcache.New(30*time.Second, stopCh) volumeBinder := volumebinder.NewVolumeBinder(client, nodeInformer, pvcInformer, pvInformer, storageClassInformer

The framework is changing so fast right now there is no point in actually supporting out-of-tree plugins before beta.

We are trying this feature, not sure whether others are using it now. Even it is alpha, we could get feedback from users. If it is not a must, I think it might be better to keep it.

that will require us to define global variables for the cache and volume binder

I could not get your point why we need to define global variables for them, we could still declare them as local variables and use them as following. Is there any problem?

schedulerCache := internalcache.New(30*time.Second, stopCh) volumeBinder := volumebinder.NewVolumeBinder(client, nodeInformer, pvcInformer, pvInformer, storageClassInformer

schedulerCache and volumeBinder are shared handlers, we can't instantiate them multiple times, they need to be created once.

The framework is changing so fast right now there is no point in actually supporting out-of-tree plugins before beta.

We are trying this feature, not sure whether others are using it now. Even it is alpha, we could get feedback from users. If it is not a must, I think it might be better to keep it.

It is going to be tricky to keep this, and it is very premature to say we support out-of-tree plugins at this stage because we are changing the framework very frequently as we discover its limitation during the migration. If you are testing this feature, I think you can still carry a small patch to enable this back in your dev environment. Is that reasonable?

Huang-Wei · 2019-10-10T17:01:34Z

cmd/kube-scheduler/app/server.go

+	if len(registryOptions) != 0 {
+		return errors.New("out-of-tree framework plugins are not yet supported")
 	}


Instead of totally disabling it, does a warning giving the limitations sound better? If so, does it resolve @hex108 's concern?

Huang-Wei · 2019-10-10T18:37:18Z

pkg/scheduler/scheduler.go

+	volumeBinder := volumebinder.NewVolumeBinder(client, nodeInformer, pvcInformer, pvInformer, storageClassInformer,
+		time.Duration(options.bindTimeoutSeconds)*time.Second)
+
+	if options.frameworkRegistry == nil {


If external users register something into frameworkRegistry, i.e. it's not nil, IMO it's still possible for the internal scheduler to register the frameworkplugins.RegistryArgs below? Are you concerned with some side effects? If so, we can warn users.

I mean we can still give a bare frameworkRegistry to end users to register custom things at cmd/.../app.go, and then inject the necessary fields here (to pass to legacy predicates/priorities).

I think I know how to do this. We can add an option to scheduler.New to pass a "outOfTreeRegistry", which collects out-of-tree plugins that we append to the default registry, I will send a patch to show you how it looks, thanks for brainstorming.

…actories for default plugins

Huang-Wei · 2019-10-10T21:03:53Z

/lgtm

Thanks, @ahg-g !

Huang-Wei · 2019-10-10T21:29:13Z

/retest

hex108 · 2019-10-11T01:28:12Z

It is great!

The release note also needs to be changed. :)

[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins

k8s-ci-robot requested review from draveness and Huang-Wei October 9, 2019 19:32

ahg-g force-pushed the ahg-adv-config branch from faee910 to d6b4061 Compare October 9, 2019 19:37

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 9, 2019

ahg-g force-pushed the ahg-adv-config branch from d6b4061 to 908b273 Compare October 9, 2019 19:38

hex108 reviewed Oct 10, 2019

View reviewed changes

neolit123 mentioned this pull request Oct 10, 2019

The cgroup name can't be nil while systemReserved is designated #83229

Closed

ahg-g changed the title ~~Disabled out-of-tree framework plugins~~ [migration phase 1] Disabled out-of-tree framework plugins Oct 10, 2019

Huang-Wei reviewed Oct 10, 2019

View reviewed changes

ahg-g changed the title ~~[migration phase 1] Disabled out-of-tree framework plugins~~ [migration phase 1] Make schedulercache, volume binder available when registering default plugins Oct 10, 2019

ahg-g changed the title ~~[migration phase 1] Make schedulercache, volume binder available when registering default plugins~~ [migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins Oct 10, 2019

ahg-g force-pushed the ahg-adv-config branch 2 times, most recently from dd314d7 to a8ce4ac Compare October 10, 2019 20:05

made scheduler cache and volume binder available when instantiating f…

456df97

…actories for default plugins

ahg-g force-pushed the ahg-adv-config branch from a8ce4ac to 456df97 Compare October 10, 2019 20:21

k8s-ci-robot assigned Huang-Wei Oct 10, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 10, 2019

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Oct 11, 2019

k8s-ci-robot merged commit a5e6ac0 into kubernetes:master Oct 11, 2019

k8s-ci-robot added this to the v1.17 milestone Oct 11, 2019

ohsewon pushed a commit to ohsewon/kubernetes that referenced this pull request Oct 16, 2019

Merge pull request kubernetes#83694 from ahg-g/ahg-adv-config

95dc239

[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins

ahg-g deleted the ahg-adv-config branch January 10, 2020 15:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins #83694

[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins #83694

ahg-g commented Oct 9, 2019 •

edited

Loading

k8s-ci-robot commented Oct 9, 2019

hex108 Oct 10, 2019

ahg-g Oct 10, 2019

hex108 Oct 10, 2019

ahg-g Oct 10, 2019

hex108 Oct 10, 2019

ahg-g Oct 10, 2019

Huang-Wei Oct 10, 2019

Huang-Wei Oct 10, 2019

Huang-Wei Oct 10, 2019 •

edited

Loading

ahg-g Oct 10, 2019

Huang-Wei commented Oct 10, 2019

Huang-Wei commented Oct 10, 2019

hex108 commented Oct 11, 2019

[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins #83694

[migration phase 1] Make scheduler cache, volume binder and listers available when registering default plugins #83694

Conversation

ahg-g commented Oct 9, 2019 • edited Loading

k8s-ci-robot commented Oct 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei Oct 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei commented Oct 10, 2019

Huang-Wei commented Oct 10, 2019

hex108 commented Oct 11, 2019

ahg-g commented Oct 9, 2019 •

edited

Loading

Huang-Wei Oct 10, 2019 •

edited

Loading