Add KubeSchedulerConfiguration for k8s 1.19 and up

[maciejaszek]

What type of PR is this?
/kind feature

What this PR does / why we need it:
With release of version 1.19.0 of kubernetes KubeSchedulerConfiguration was graduated to beta. Many flags that configure the same settings as KubeSchedulerConfiguration does are deprecated.

Which issue(s) this PR fixes:
Fixes #7350

Does this PR introduce a user-facing change?:

New vars for configuring kube-scheduler were introduced (including extenders and profiles). Default vaules can be found at roles/kubernetes/control-plane/defaults/main/kube-scheduler.yml

[k8s-ci-robot]

Hi @maciejaszek. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[floryut]

/cc @oomichi @EppO

[EppO]

defaults look good! Thanks for your PR

/lgtm

[champtar]

How de we keep the defaults in sync with kubeadm ? Should we have a if defined for most vars ?

[maciejaszek]

@champtar Thanks for review. Could you elaborate? which vars are you talking about?
I've moved scheduler related defaults from main.yml to kube-scheduler.yml. The new scheduler configuration file is passed as a scheduler --config flag in scheduler/extraArgs section of kubeadm-config.yaml. The only scheduler options that are set both by configuration file and flags are kube_scheduler_leader_elect_lease_duration and kube_scheduler_leader_elect_renew_deadline. Those are already guarded by if.

[champtar]

Hi @maciaszczykm, Here are some variables that we now set explicitly instead of just taking the defaults

acceptContentTypes: "{{ kube_scheduler_client_accept_content_types }}"
burst: {{ kube_scheduler_client_burst }}
contentType: "{{ kube_scheduler_client_content_type }}"
qps: {{ kube_scheduler_client_qps }}
retryPeriod: "{{ kube_scheduler_leader_elect_retry_period }}"
resourceLock: "{{ kube_scheduler_leader_elect_resource_lock }}"
resourceName: "{{ kube_scheduler_leader_elect_resource_name }}"
resourceNamespace: "{{ kube_scheduler_leader_elect_resource_namespace }}"
percentageOfNodesToScore: {{ kube_scheduler_percentage_of_nodes_to_score }}
podInitialBackoffSeconds: {{ kube_scheduler_pod_initial_backoff_seconds }}
podMaxBackoffSeconds: {{ kube_scheduler_pod_max_backoff_seconds }}

If upstream finds out that some default need to be improved we will be stuck with the old default for a long time.
I would prefer to set only what we MUST or what the user override, but not have a copy/paste of current default as of 1.20
Tell me if I'm still not clear

[maciejaszek]

@champtar Thanks for clarification, understood. I think that besides defaults that were already present, we might have following ones:

kube_scheduler_extenders: []
kube_scheduler_percentage_of_nodes_to_score: 0
kube_scheduler_pod_initial_backoff_seconds: null
kube_scheduler_pod_max_backoff_seconds: null
kube_scheduler_profiles: []

The kube_scheduler_percentage_of_nodes_to_score, kube_scheduler_pod_initial_backoff_seconds and kube_scheduler_pod_max_backoff_seconds are highly unlikely to change as null/0 makes no sense and are used to indicate that default k8s values should be used.

I'd also add kube_scheduler_client_conn, kube_scheduler_leader_election and kube_scheduler_config_extra_opts dicts, so that user may add settings that kubespray won't provide.

If that's OK, I'll amend and force-push my patches.

[champtar]

@maciejaszek please go ahead and ping us when done

[champtar]

/ok-to-test

[maciejaszek]

@champtar @EppO @floryut new patchset pushed.

[champtar]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floryut, maciejaszek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floryut]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment