Show how to use scheduling profiles on k8s.io/docs/concepts/scheduling-eviction/kube-scheduler/

[JoelColledge]

This is a Feature Request

What would you like to be added

The Predicates/Priorities (also called filtering/scoring) approach to configuring kube-scheduler is being deprecated in favor of the Plugins approach. Currently the Plugins approach is sparsely documented. An example of how to configure it would be fantastic.

Why is this needed

It is currently hard to find out how to configure the scheduler plugins.

Page to Update:
https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/

Or a different page in this section.

Related issues:

#21051 covers marking the old approach as deprecated in the reference docs.

Outline solution:

Here is an outline of the steps I took to configure the scheduling plugins. I hope the scheduler developers or someone else can describe a simpler approach. This is for a cluster installed with kubeadm.

Add the file /etc/kubernetes/scheduler-config.conf to the master node(s), with content

apiVersion: kubescheduler.config.k8s.io/v1alpha2
bindTimeoutSeconds: 600
clientConnection:
  acceptContentTypes: ""
  burst: 100
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/scheduler.conf
  qps: 50
disablePreemption: false
enableContentionProfiling: true
enableProfiling: true
extenders: null
healthzBindAddress: 0.0.0.0:10251
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
  leaseDuration: 15s
  renewDeadline: 10s
  resourceLock: endpointsleases
  resourceName: kube-scheduler
  resourceNamespace: kube-system
  retryPeriod: 2s
metricsBindAddress: 0.0.0.0:10251
percentageOfNodesToScore: 0
podInitialBackoffSeconds: 1
podMaxBackoffSeconds: 10
profiles:
- schedulerName: default-scheduler
  plugins:
    score:
      disabled:
        - name: NodeResourcesLeastAllocated
      enabled:
        - name: NodeResourcesLeastAllocated
          weight: 10000

Replace the file /etc/kubernetes/manifests/kube-scheduler.yaml with content

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --config=/etc/kubernetes/scheduler-config.conf
    image: k8s.gcr.io/kube-scheduler:v1.18.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
    - mountPath: /etc/kubernetes/scheduler-config.conf
      name: config
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
  - hostPath:
      path: /etc/kubernetes/scheduler-config.conf
      type: FileOrCreate
    name: config
status: {}

The change to the second file will cause the scheduler to restart and the new configuration should be in use.

Notes on /etc/kubernetes/scheduler-config.conf:

• This was generated using

  docker run --name tmp -v /etc/kubernetes/scheduler.conf:/etc/kubernetes/scheduler.conf:ro --entrypoint kube-scheduler k8s.gcr.io/kube-scheduler:v1.18.2 --write-config-to /tmp/scheduler-config.conf --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
  docker cp tmp:/tmp/scheduler-config.conf .
  

  And then the plugin configuration at the bottom was modified.
• I was not able to get all plugins to be disabled with "*" as described. It is necessary to disable a plugin if you want to replace it with a different weight.

Notes on /etc/kubernetes/manifests/kube-scheduler.yaml:

• This is largely the same as the original produced by kubeadm, with just the addition of the --config flag and the extra hostPath mount.
• Contrary to the CLI reference flags do not seem to override values in the config file.

Additional notes:

• Adding the flag --v=10 to command causes some of scheduling values to be logged, which was sufficient for me to verify that my changes really had an effect.
• This verbose log file is much easier to read once you disable leader election (in the config file)

[sftim]

If you're documenting a “how to” guide for configuring the scheduler and its plugins, I recommend adding that to the Tasks section.

[sftim]

/priority backlog
/sig scheduling
/language en

[alculquicondor]

note https://kubernetes.io/docs/reference/scheduling/profiles/

[alculquicondor]

Please add me in the reviews

[electrocucaracha]

Just for the record, If you want to configure your scheduler thru KinD tool you can use the following instructions:

cat << EOF > scheduler-config.conf
apiVersion: kubescheduler.config.k8s.io/v1beta1
clientConnection:
  acceptContentTypes: ""
  burst: 100
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/scheduler.conf
  qps: 50
disablePreemption: false
enableContentionProfiling: true
enableProfiling: true
extenders: null
healthzBindAddress: 0.0.0.0:10251
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
  leaseDuration: 15s
  renewDeadline: 10s
  resourceLock: endpointsleases
  resourceName: kube-scheduler
  resourceNamespace: kube-system
  retryPeriod: 2s
metricsBindAddress: 0.0.0.0:10251
percentageOfNodesToScore: 0
podInitialBackoffSeconds: 1
podMaxBackoffSeconds: 10
profiles:
- schedulerName: default-scheduler
  plugins:
    score:
      disabled:
        - name: DefaultTopologySpread
EOF

cat << EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        scheduler:
          extraArgs:
            config: /etc/kubernetes/scheduler-config.conf
          extraVolumes:
            - name: configuration
              hostPath: /etc/kubernetes/scheduler-config.conf
              mountPath: /etc/kubernetes/scheduler-config.conf
              readOnly: true
              pathType: File
  - role: worker
  - role: worker
EOF
fi

docker cp scheduler-config.conf kind-control-plane:/etc/kubernetes/scheduler-config.conf

[alculquicondor]

Feel free to submit pull requests

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[irvifa]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[Frans-Lukas]

I've struggled with configuring and working with scheduler plugins for a day or two now because of poor and misleading documentation about scheduler configuration. A link to this github issue from the scheduler documentation would be highly appreciated. Also, I've been trying to configure plugins with minikube but haven't fully understood how to do that, which lead to me trying to work with kubeadm or kind instead.

[alculquicondor]

/help

unfortunately, we don't have bandwidth to improve the docs at the moment, but we welcome PRs

[k8s-ci-robot]

@alculquicondor:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

unfortunately, we don't have bandwidth to improve the docs at the moment, but we welcome PRs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

cc @Huang-Wei

[damemi]

Hi @Frans-Lukas, I'm sorry for your frustration. Being that the scheduler Profiles config is still relatively new (and many are still transitioning to it) we are open to feedback and there is ongoing work to make the documentation better. Could you share more specifically what in the current documentation (mostly https://kubernetes.io/docs/reference/scheduling/config/) was misleading or not helpful, and what parts of this issue were most helpful that you think should be documented for others?

At its core, the plugins configuration is not much different than the old Policy configuration (in terms of how you use it), and in general applies across most k8s installs:

1. create a file (or configmap from a file) of the kubescheduler.config.k8s.io/v1beta1 config object, with your plugins config
2. mount the configmap and pass the file to --config in the scheduler's static pod manifest
3. restart the scheduler container to pick up the changes

There are many ways to do this in different environments, such as in #21128 (comment), and it may be worth us documenting those different paths.

Like @alculquicondor mentioned though, any help in improving this documentation is welcome

[Frans-Lukas]

Thank you both for your quick replies. Some of the biggest issues were in understanding where the config files are located, how to "mount the configmap" and if the config file should be edited inside the master node or in the hosting machine. I.e. what is a config object and how do I mount it? It felt like some of the documentation was contradictory. I'll make a more detailed write up of how we interpreted the documentation and what we think is missing tomorrow.

[damemi]

@Frans-Lukas setting up pods to read from configmap data is an essential feature of Kubernetes (not just related to the scheduler, it can be used with your own application pods as well). The options for that are well documented, please see https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/.

Or, if you wish to read the file directly from the host, you can mount local directories and files into pods using Volumes https://kubernetes.io/docs/concepts/storage/volumes/

With an understanding of how one can use configmaps and volumes to pass data to your pods, setting the configuration for the scheduler is essentially no different (though it is often run as a static pod, meaning changes will not be automatically picked up by the apiserver). Much of the scheduler documentation does assume that users will already be familiar with these concepts, however it may be helpful to reference them more clearly.

[Frans-Lukas]

@damemi Alright, we got it working! Your reply definitely helped, thank you.

We originally followed this tutorial: https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md as it was the only one with clear instructions as to how to modify the scheduler plugins, but we got strange errors about kubescheduler.config.k8s.io/v1alpha2 not being correct (and if we changed it to v1beta1 we got other pod errors). What I think was most unclear is that on https://kubernetes.io/docs/reference/scheduling/config/ it says to run kube-scheduler --config <filename> but I could not find any documentation on what kube-scheduler is. I also came across https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#plugin-configuration multiple times, and all it says is:

You can enable or disable plugins in the scheduler configuration. If you are using Kubernetes v1.18 or later, most scheduling plugins are in use and enabled by default.

Without any instructions or links on how to enable or disable plugins. It does link to the github repository linked above with instructions that seems to be deprecated.

Anyway, for anyone else having similar issues, what we did was that we entered the master node of the cluster, edited kubernetes/manifests/kube-scheduler.yaml by adding a config, a mountPath and a hostPath, I.e:
(+ = added lines)

spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
+    - --config=/etc/kubernetes/scheduler-config.conf
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    - --port=0

.......
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
+    - mountPath: /etc/kubernetes/scheduler-config.conf
+      name: config
+      readOnly: true
.......
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
+  - hostPath:
+      path: /etc/kubernetes/scheduler-config.conf
+      type: FileOrCreate
+    name: config

With the contents of /etc/kubernetes/scheduler-config.conf being:

apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
  - plugins:
      queueSort:
        disabled:
         - name: "*"

This makes the scheduler pod fail as there must be a queueSort plugin, however, if the scheduler fails because of this, we know that the config works.

[damemi]

@Frans-Lukas that's great, I'm glad it is working for you :)

we got strange errors about kubescheduler.config.k8s.io/v1alpha2 not being correct (and if we changed it to v1beta1 we got other pod errors)

Yeah, that's a typo on that page. It should be v1beta1 now (pr to change that kubernetes-sigs/scheduler-plugins#147), but if you still have them, what were your other errors when you changed it to v1beta1? This would just help us fix the docs

it says to run kube-scheduler --config but I could not find any documentation on what kube-scheduler is

kube-scheduler is the scheduler binary, what is running in your pod under spec.containers.command

Without any instructions or links on how to enable or disable plugins.

There are a few example configurations on the page linked to from that section, such as multiple profiles which links to another in extension points, the latter specifically showing how to enable and disable by extension point. We could make these examples more prominent.

Thank you for the detailed feedback! Knowing exactly where we can improve will make it much easier for future users.

[Frans-Lukas]

@damemi Great! I'm glad I'm able to help.

The errors we are getting from trying to use the config in https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md after changing to v1beta1 is:

I0219 14:47:27.047390 1 registry.go:173] Registering SelectorSpread plugin
I0219 14:47:27.047592 1 registry.go:173] Registering SelectorSpread plugin
I0219 14:47:27.411318 1 serving.go:331] Generated self-signed cert in-memory
I0219 14:47:27.564203 1 registry.go:173] Registering SelectorSpread plugin
I0219 14:47:27.564219 1 registry.go:173] Registering SelectorSpread plugin
W0219 14:47:27.565635 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
W0219 14:47:27.565811 1 client_config.go:613] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
couldn't create scheduler using provider "DefaultProvider": initializing profiles: creating profile for scheduler name default-scheduler: error initializing plugin "Coscheduling": failed to init rest.Config: invalid configuration: no configuration has been provided, 

[damemi]

@Frans-Lukas ah, the relevant message there is

couldn't create scheduler using provider "DefaultProvider": initializing profiles: creating profile for scheduler name default-scheduler: error initializing plugin "Coscheduling": failed to init rest.Config: invalid configuration: no configuration has been provided,

This is because the sample config in scheduler-plugins is meant to be used with a custom-built scheduler with the Coscheduling 3rd-party plugin compiled (you can actually compile that repo into a kube-scheduler binary and use an image with that binary in your pod manifest).

Since you are using it with the default scheduler, it fails to start because it does not have the Coscheduling plugin. In other words, you would get a similar message trying to enable any nonexistent plugin or typo in a plugin name. So that is a valid error, and it indicates you have properly passed the config to the scheduler.

[alculquicondor]

Anyway, for anyone else having similar issues, what we did was that we entered the master node of the cluster, edited kubernetes/manifests/kube-scheduler.yaml by adding a config, a mountPath and a hostPath, I.e:

Note that we cannot say something like that in https://kubernetes.io/docs/reference/scheduling/config/ as users can setup the scheduler in many different ways, using such file or not.

What I think was most unclear is that on https://kubernetes.io/docs/reference/scheduling/config/ it says to run kube-scheduler --config but I could not find any documentation on what kube-scheduler is.

Note that the doc links to https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/

It seems that you are not trying to develop new scheduling plugins, you just want to configure the k8s scheduler, so most of https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md doesn't apply to you.

Again, I encourage you to send a PR to improve https://kubernetes.io/docs/reference/scheduling/config/. Note that we have a long-standing related issue that we are still looking for help with #23889. This would allow to have the code documentation for the config file in the website. For now, to learn the details of this file, you have to look at the code https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-scheduler/config/v1beta1/types.go

[Frans-Lukas]

@damemi I compiled the scheduler-plugins image (according to these instructions), and set the image in my /etc/kubernetes/manifests/kube-scheduler.yaml file. I.e:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --config=/etc/kubernetes/coscheduling-config.yaml
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    - --port=0
    image: localhost:5000/scheduler-plugins/kube-scheduler:latest   <---- The compiled custom scheduler
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
 .....

Which I thought would create the scheduler pod with the custom scheduler image but I still get the same error message.

W0219 17:10:50.073294       1 client_config.go:613] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
couldn't create scheduler using provider "DefaultProvider": initializing profiles: creating profile for scheduler name default-scheduler: error initializing plugin "Coscheduling": failed to init rest.Config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

@alculquicondor We are trying to develop a plugin, just starting out with trying to enable the default ones. I'd love to send a PR as soon as I feel like I understand the process. I am just starting to get comfortable working with configuring the scheduler, but I feel like there are still some missing pieces.

Again, thanks for the help.

[alculquicondor]

If you are looking into creating your own plugin, consider opening an issue in https://github.com/kubernetes-sigs/scheduler-plugins/ instead.
This repo is not the right audience for that. I'm sure @Huang-Wei can provide extra guidance there.

[Huang-Wei]

@Frans-Lukas Let's move the discussion to https://github.com/kubernetes-sigs/scheduler-plugins/ as here the topic is more about configuring default scheduler with multiple profiles.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.