Add support for multiple scheduling profiles #88285

alculquicondor · 2020-02-18T21:42:54Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

kube-scheduler can run with more than one profile. Given a pod, the profile is selected by using .spec.SchedulerName.
Profiles should have different scheduler names. They should have the same queue sort plugin configuration.

Which issue(s) this PR fixes:

Part of #85737, kubernetes/enhancements#1451

Special notes for your reviewer:

This PR builds on top of #88087, so you can review from the commit titled: Support multiple scheduling profiles in a single scheduler

For convenience, I split this PR in 2 commits, which will be squashed after the review.

implementation and tests for new package.
fixes for existing tests.

A follow up PR will add unit and integration tests to exercise the multiple profiles.

Does this PR introduce a user-facing change?:

kube-scheduler can run more than one scheduling profile. Given a pod, the profile is selected by using its `.spec.SchedulerName`.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20200114-multi-scheduling-profiles.md

fejta-bot · 2020-02-19T00:23:03Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

pkg/scheduler/apis/config/testing/compatibility_test.go

pkg/scheduler/factory_test.go

pkg/scheduler/scheduler_test.go

ahg-g · 2020-02-19T01:58:11Z

pkg/scheduler/scheduler_test.go

@@ -591,7 +598,7 @@ func TestSchedulerFailedSchedulingReasons(t *testing.T) {

 // queuedPodStore: pods queued before processing.
 // scache: scheduler cache that might contain assumed pods.
-func setupTestScheduler(queuedPodStore *clientcache.FIFO, scache internalcache.Cache, informerFactory informers.SharedInformerFactory, recorder events.EventRecorder, fakeVolumeBinder *volumebinder.VolumeBinder, fns ...st.RegisterPluginFunc) (*Scheduler, chan *v1.Binding, chan error) {
+func setupTestScheduler(queuedPodStore *clientcache.FIFO, scache internalcache.Cache, informerFactory informers.SharedInformerFactory, broadcaster events.EventBroadcaster, fakeVolumeBinder *volumebinder.VolumeBinder, fns ...st.RegisterPluginFunc) (*Scheduler, chan *v1.Binding, chan error) {


Instead of passing it as an argument, I would create the EventBroadcaster inside the function and return it (since a couple of tests require it).

It seems that there are the same amount of tests that use it than tests that don't.
I prefer to keep it as is for now given that the PR is quite big already.

pkg/scheduler/factory.go

pkg/scheduler/profile/profile.go

alculquicondor · 2020-02-19T18:20:24Z

/retest

alculquicondor · 2020-02-19T20:41:40Z

/assign @Huang-Wei for a second pass

It looks like the api-review is almost done

Huang-Wei

Some comments.

pkg/scheduler/core/generic_scheduler.go

cmd/kube-scheduler/app/server.go

Huang-Wei · 2020-02-20T08:16:19Z

pkg/scheduler/factory.go

+		prof := &c.profiles[i]
+		plugins := &schedulerapi.Plugins{}
+		plugins.Append(defaultPlugins)
+		plugins.Apply(prof.Plugins)
+		prof.Plugins = plugins


nit: these 5 lines can be extracted into a function to avoid duplication (with L304~L308)

I wouldn't mind too much. They are slightly different and createFromConfig should go away soon when we remove policy support.

pkg/scheduler/core/generic_scheduler.go

pkg/scheduler/scheduler.go

alculquicondor · 2020-02-21T11:20:51Z

/priority important-soon

/hold for #88087

alculquicondor · 2020-02-25T15:19:06Z

/hold cancel

alculquicondor · 2020-02-25T15:21:55Z

/assign @msau42
for test

k8s-ci-robot · 2020-02-25T15:21:57Z

@alculquicondor: GitHub didn't allow me to assign the following users: for, test.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @msau42 for test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ahg-g

looks good to me.

ahg-g · 2020-02-25T16:00:55Z

pkg/scheduler/scheduler.go

@@ -215,7 +197,9 @@ func WithPodMaxBackoffSeconds(podMaxBackoffSeconds int64) Option {
 }

 var defaultSchedulerOptions = schedulerOptions{
-	schedulerName: v1.DefaultSchedulerName,
+	profiles: []schedulerapi.KubeSchedulerProfile{
+		{SchedulerName: v1.DefaultSchedulerName},


Add a comment clarifying that the default provider plugins is used by default

Signed-off-by: Aldo Culquicondor <acondor@google.com>

ahg-g · 2020-02-25T18:56:40Z

/lgtm

alculquicondor · 2020-02-25T18:58:38Z

/retest

msau42 · 2020-02-25T19:49:41Z

/approve
for test

k8s-ci-robot · 2020-02-25T19:50:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-scheduler/OWNERS~~ [alculquicondor]
~~pkg/scheduler/OWNERS~~ [alculquicondor]
~~test/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alculquicondor · 2020-02-25T20:46:40Z

/retest

alculquicondor · 2020-02-25T21:08:26Z

/retest

k8s-ci-robot requested review from ahg-g and draveness February 18, 2020 21:43

alculquicondor force-pushed the multiprofiles-runtime branch from 59ca8c3 to 9329416 Compare February 18, 2020 22:07

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 18, 2020

alculquicondor mentioned this pull request Feb 18, 2020

Run multiple Scheduling Profiles kubernetes/enhancements#1451

Closed

alculquicondor force-pushed the multiprofiles-runtime branch 2 times, most recently from 5f08ebd to f7464c4 Compare February 18, 2020 22:58

ahg-g reviewed Feb 19, 2020

View reviewed changes

alculquicondor force-pushed the multiprofiles-runtime branch from f7464c4 to 0b55fe5 Compare February 19, 2020 16:02

k8s-ci-robot added the sig/apps Categorizes an issue or PR as relevant to SIG Apps. label Feb 19, 2020

alculquicondor mentioned this pull request Feb 19, 2020

Add Schedulings Profiles to kubescheduler.config.k8s.io/v1alpha2 #88087

Merged

Huang-Wei reviewed Feb 20, 2020

View reviewed changes

alculquicondor mentioned this pull request Feb 21, 2020

Add compatibility tests for framework plugin configuration #88191

Merged

alculquicondor force-pushed the multiprofiles-runtime branch from 8b58bdb to 1bd94be Compare February 21, 2020 11:19

alculquicondor mentioned this pull request Feb 24, 2020

Scheduler: Add compatibility tests for framework configuration #88027

Closed

2 tasks

alculquicondor force-pushed the multiprofiles-runtime branch from 1bd94be to de4522c Compare February 25, 2020 15:00

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 25, 2020

k8s-ci-robot assigned msau42 Feb 25, 2020

alculquicondor mentioned this pull request Feb 25, 2020

Add unit and integration tests for running multiple scheduling profiles #88526

Merged

ahg-g reviewed Feb 25, 2020

View reviewed changes

Support multiple scheduling profiles in a single scheduler

c048858

Signed-off-by: Aldo Culquicondor <acondor@google.com>

alculquicondor force-pushed the multiprofiles-runtime branch from de4522c to c048858 Compare February 25, 2020 17:40

k8s-ci-robot assigned ahg-g Feb 25, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 25, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 25, 2020

k8s-ci-robot merged commit f7c37d3 into kubernetes:master Feb 26, 2020

k8s-ci-robot added this to the v1.18 milestone Feb 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for multiple scheduling profiles #88285

Add support for multiple scheduling profiles #88285

alculquicondor commented Feb 18, 2020 •

edited

Loading

fejta-bot commented Feb 19, 2020

ahg-g Feb 19, 2020

alculquicondor Feb 19, 2020

alculquicondor commented Feb 19, 2020

alculquicondor commented Feb 19, 2020

Huang-Wei left a comment

Huang-Wei Feb 20, 2020

alculquicondor Feb 20, 2020

alculquicondor commented Feb 21, 2020

alculquicondor commented Feb 25, 2020

alculquicondor commented Feb 25, 2020 •

edited

Loading

k8s-ci-robot commented Feb 25, 2020

ahg-g left a comment

ahg-g Feb 25, 2020

alculquicondor Feb 25, 2020

ahg-g commented Feb 25, 2020

alculquicondor commented Feb 25, 2020

msau42 commented Feb 25, 2020

k8s-ci-robot commented Feb 25, 2020

alculquicondor commented Feb 25, 2020

alculquicondor commented Feb 25, 2020

Add support for multiple scheduling profiles #88285

Add support for multiple scheduling profiles #88285

Conversation

alculquicondor commented Feb 18, 2020 • edited Loading

fejta-bot commented Feb 19, 2020

ahg-g Feb 19, 2020

Choose a reason for hiding this comment

alculquicondor Feb 19, 2020

Choose a reason for hiding this comment

alculquicondor commented Feb 19, 2020

alculquicondor commented Feb 19, 2020

Huang-Wei left a comment

Choose a reason for hiding this comment

Huang-Wei Feb 20, 2020

Choose a reason for hiding this comment

alculquicondor Feb 20, 2020

Choose a reason for hiding this comment

alculquicondor commented Feb 21, 2020

alculquicondor commented Feb 25, 2020

alculquicondor commented Feb 25, 2020 • edited Loading

k8s-ci-robot commented Feb 25, 2020

ahg-g left a comment

Choose a reason for hiding this comment

ahg-g Feb 25, 2020

Choose a reason for hiding this comment

alculquicondor Feb 25, 2020

Choose a reason for hiding this comment

ahg-g commented Feb 25, 2020

alculquicondor commented Feb 25, 2020

msau42 commented Feb 25, 2020

k8s-ci-robot commented Feb 25, 2020

alculquicondor commented Feb 25, 2020

alculquicondor commented Feb 25, 2020

alculquicondor commented Feb 18, 2020 •

edited

Loading

alculquicondor commented Feb 25, 2020 •

edited

Loading