Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Schedulings Profiles to kubescheduler.config.k8s.io/v1alpha2 #88087

Merged
merged 1 commit into from Feb 24, 2020

Conversation

@alculquicondor
Copy link
Member

alculquicondor commented Feb 12, 2020

What type of PR is this?

/kind api-change

What this PR does / why we need it:

Cluster operators can use profiles to offer different scheduling behaviors to pods. Each profile is associated to a scheduler name which pods can select through .spec.schedulerName.
If not defined, a profile is added with the name default-scheduler.

This PR only implements API changes, along with defaults, validation and conversion from v1alpha1. kube-scheduler will only use the first profile and preserve its behavior.

A follow up PR will add runtime support for the multiple profiles.

Which issue(s) this PR fixes:

Part of #87617, #85737 and kubernetes/enhancements#1451

Does this PR introduce a user-facing change?:

Add Scheduling Profiles to kubescheduler.config.k8s.io/v1alpha2

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20200114-multi-scheduling-profiles.md
@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Feb 12, 2020

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch 2 times, most recently from 7dba86c to 71a5c2f Feb 12, 2020
@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 12, 2020

/assign @ahg-g @damemi

@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 12, 2020

/hold for the parent PR

@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch from 71a5c2f to 0dccc0b Feb 12, 2020
cmd/kube-scheduler/app/options/options_test.go Outdated Show resolved Hide resolved
pkg/scheduler/apis/config/types.go Show resolved Hide resolved
// SchedulerName is name of the scheduler, used to select which pods
// will be processed by this profile, based on pod's "spec.SchedulerName".
Comment on lines 106 to 107

This comment has been minimized.

Copy link
@ahg-g

ahg-g Feb 13, 2020

Member
Name is name of the profile ...
pkg/scheduler/apis/config/v1alpha2/defaults.go Outdated Show resolved Hide resolved
pkg/scheduler/apis/config/v1alpha2/defaults.go Outdated Show resolved Hide resolved
pkg/scheduler/apis/config/validation/validation.go Outdated Show resolved Hide resolved
pkg/scheduler/apis/config/types.go Outdated Show resolved Hide resolved
func validateCommonQueueSort(path *field.Path, profiles []config.KubeSchedulerProfile) field.ErrorList {
allErrs := field.ErrorList{}
var canon *config.PluginSet
if profiles[0].Plugins != nil {

This comment has been minimized.

Copy link
@ahg-g

ahg-g Feb 13, 2020

Member

How about the PluginConfig? It should be the same for all profiles as well, but I guess this is harder to validate since we don't really know what exact QueueSort Plugin names we will end up with at this point, so perhaps we can delay this to Framework instantiation.

This comment has been minimized.

Copy link
@alculquicondor

alculquicondor Feb 13, 2020

Author Member

That is my thinking as well. I left a TODO here anyways.

@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch 2 times, most recently from d93ade2 to 60237ed Feb 13, 2020
@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch from 60237ed to 8cf0d2f Feb 14, 2020
@k8s-ci-robot k8s-ci-robot added size/XL and removed size/XXL labels Feb 14, 2020
Copy link
Member

ahg-g left a comment

looks good to me

For api review
/cc @liggitt

pkg/scheduler/apis/config/types.go Outdated Show resolved Hide resolved
@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch from 838a475 to 2afc7ab Feb 18, 2020
@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 18, 2020

@liggitt Friendly reminder

@@ -239,7 +242,8 @@ func (o *Options) Config() (*schedulerappconfig.Config, error) {
}

coreBroadcaster := record.NewBroadcaster()
coreRecorder := coreBroadcaster.NewRecorder(scheme.Scheme, corev1.EventSource{Component: c.ComponentConfig.SchedulerName})
// TODO(#85737): Support more than one profile.

This comment has been minimized.

Copy link
@liggitt

liggitt Feb 18, 2020

Member

in the meantime, fail if more than one is specified so we don't leave master in a should-not-ship state?

This comment has been minimized.

Copy link
@alculquicondor

alculquicondor Feb 19, 2020

Author Member

Actually, since this recorder is used for leader election, we can only use one name. The first profile's name makes the most sense.

I added the error in server.go

This comment has been minimized.

Copy link
@liggitt

liggitt Feb 21, 2020

Member

that definitely needs to be documented in the user guide for this... that is not intuitive

This comment has been minimized.

Copy link
@alculquicondor

alculquicondor Feb 24, 2020

Author Member

Will do.

pkg/scheduler/apis/config/types.go Show resolved Hide resolved
} else {
profile.Plugins = nil
}
profile.PluginConfig = *(*[]config.PluginConfig)(unsafe.Pointer(&in.PluginConfig))

This comment has been minimized.

Copy link
@liggitt

liggitt Feb 18, 2020

Member

Prefer allocating an array and calling Convert_v1alpha1_PluginConfig_To_config_PluginConfig. In general, leave the unsafe casting to the generated conversion code (which does generate-time checks that the types are actually equivalent)

profile := &obj.Profiles[i]
if profile.SchedulerName == nil {
// Validation ensures that only one profile uses DefaultSchedulerName.
profile.SchedulerName = pointer.StringPtr(api.DefaultSchedulerName)

This comment has been minimized.

Copy link
@liggitt

liggitt Feb 18, 2020

Member

would it make more sense to only apply a default name if there's a single profile, and leave it undefaulted (and make validation fail with a "required" error) if there's more than a single profile?

This comment has been minimized.

Copy link
@alculquicondor

alculquicondor Feb 19, 2020

Author Member

Yes, this gives a more comprehensible error :)
Done

@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch from 2afc7ab to 22071ad Feb 19, 2020
@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 19, 2020

/hold for squash

@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 19, 2020

Anything else to add? Note that the implementation PR is already up for review #88285

@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 21, 2020

/priority important-soon

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Feb 21, 2020

/approve

API change lgtm. @ahg-g can lgtm.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 21, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ahg-g

This comment has been minimized.

Copy link
Member

ahg-g commented Feb 24, 2020

@alculquicondor please squash

Signed-off-by: Aldo Culquicondor <acondor@google.com>
@alculquicondor alculquicondor force-pushed the alculquicondor:mutiprofiles-api branch from 22071ad to 9e71741 Feb 24, 2020
@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 24, 2020

Rebased and squashed

@ahg-g

This comment has been minimized.

Copy link
Member

ahg-g commented Feb 24, 2020

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Feb 24, 2020
@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 24, 2020

/retest

@Huang-Wei

This comment has been minimized.

Copy link
Member

Huang-Wei commented Feb 24, 2020

/unhold

@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 24, 2020

/hold cancel

@alculquicondor

This comment has been minimized.

Copy link
Member Author

alculquicondor commented Feb 24, 2020

/retest

@k8s-ci-robot k8s-ci-robot merged commit f6525db into kubernetes:master Feb 24, 2020
16 checks passed
16 checks passed
cla/linuxfoundation alculquicondor authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kind Job succeeded.
Details
pull-kubernetes-e2e-kind-ipv6 Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@k8s-ci-robot k8s-ci-robot added this to the v1.18 milestone Feb 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
You can’t perform that action at this time.