Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

volumebinding: scheduler queueing hints - PersistentVolume #124960

Closed
wants to merge 4 commits into from
Closed

volumebinding: scheduler queueing hints - PersistentVolume #124960

wants to merge 4 commits into from
+107 −1

Conversation

bells17
Copy link
Contributor

@bells17 bells17 commented May 20, 2024 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

kube-scheduler implements scheduling hints for the VolumeBinding plugin.
The scheduling hints allow the scheduler to determine whether to retry or skip scheduling a Pod based on the changes made to the PersistentVolume resource referenced by the plugin.

Which issue(s) this PR fixes:

Part of #118893
KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4247-queueinghint/README.md
Base PR: #124939

Special notes for your reviewer:

Fields Impacting QueueingHintFn

PersistentVolume (PV) is not included in this table because it can undergo extensive changes when a conversion is performed by csi-translation-lib.

resource field Referenced in PreFilter+Filter? Admission Overwrite Prevention Config Need to Check Changes in QHint?
PersistentVolume(PV) .metadata.labels o x o
PersistentVolume(PV) .metadata.annotations o x o
PersistentVolume(PV) .spec.capacity o x o
PersistentVolume(PV) .spec.(PersistentVolumeSource) o o x
PersistentVolume(PV) .spec.accessModes o x o
PersistentVolume(PV) .spec.claimRef o x o
PersistentVolume(PV) .spec.persistentVolumeReclaimPolicy o x o
PersistentVolume(PV) .spec.storageClassName o x o
PersistentVolume(PV) .spec.mountOptions o x o
PersistentVolume(PV) .spec.volumeMode o o x
PersistentVolume(PV) .spec.nodeAffinity o o o
PersistentVolume(PV) .spec.volumeAttributesClassName o o x
PersistentVolume(PV) .status.phase o x o
PersistentVolume(PV) .status.message o x o
PersistentVolume(PV) .status.reason o x o
PersistentVolume(PV) .status.lastPhaseTransitionTime o x o

ref(ja): https://zenn.dev/bells17/scraps/65bd6891012bdc

Does this PR introduce a user-facing change?

kube-scheduler implements scheduling hints for the VolumeBinding plugin.
The scheduling hints allow the scheduler to retry scheduling a Pod that was previously rejected by the VolumeBinding plugin only if a new resource referenced by the plugin was created or an existing resource referenced by the plugin was updated.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 20, 2024
@k8s-ci-robot k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label May 20, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bells17
Once this PR has been reviewed and has the lgtm label, please assign gnufied for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels May 20, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 20, 2024
@bells17
Copy link
Contributor Author

bells17 commented May 20, 2024

/cc @sanposhiho @utam0k

@bells17 bells17 mentioned this pull request May 20, 2024
Open
12 tasks
@sanposhiho
Copy link
Member

/kind feature

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels May 20, 2024
@bells17 bells17 requested a review from AxeZhan May 20, 2024 16:03
@bells17
Copy link
Contributor Author

bells17 commented May 21, 2024

/retest

@bells17
Copy link
Contributor Author

bells17 commented May 21, 2024

@AxeZhan I've addressed the points you mentioned in your feedback. I would appreciate it if you could take another look at the changes.
9d537e4

@AxeZhan
Copy link
Member

AxeZhan commented May 21, 2024

/cc @carlory
I'm not familiar with storage😅. I'll let folks from sig-storage take a look first.

@k8s-ci-robot k8s-ci-robot requested a review from carlory May 21, 2024 15:08
@carlory
Copy link
Member

carlory commented May 22, 2024

I'm quite busy today, I'll find time to take a look this week

@carlory
Copy link
Member

carlory commented May 22, 2024

/assign

@bells17 bells17 changed the title WIP: volumebinding: scheduler queueing hints - PersistentVolume volumebinding: scheduler queueing hints - PersistentVolume May 22, 2024
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 22, 2024
@bells17
Copy link
Contributor Author

bells17 commented May 22, 2024

/cc @jsafrane @xing-yang

sanposhiho
sanposhiho reviewed May 26, 2024
View reviewed changes
pkg/scheduler/framework/plugins/volumebinding/volume_binding.go Outdated

for _, vol := range pod.Spec.Volumes {
if vol.Name == newPV.Name {
logger.V(4).Info("PersistentVolume referenced by the Pod was created or updated")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, we're using level:5 or 6 in QHint.

pkg/scheduler/framework/plugins/volumebinding/volume_binding.go Outdated
)

for _, vol := range pod.Spec.Volumes {
if vol.Name == newPV.Name {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is pod.Spec.Volumes[*].Name related to PV's name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, that was my mistake, so I've fixed it.

@bells17 bells17 force-pushed the qhint-volume-binding-pv branch 2 times, most recently from 41979de to b0c6e46 Compare May 26, 2024 13:36
@bells17 bells17 requested a review from sanposhiho May 26, 2024 13:48
@bells17
Copy link
Contributor Author

bells17 commented May 26, 2024

@sanposhiho Thank you for your review. I have made the necessary changes, so please take another look.

carlory
carlory reviewed May 27, 2024
View reviewed changes
pkg/scheduler/framework/plugins/volumebinding/volume_binding.go
@@ -99,7 +100,7 @@ func (pl *VolumeBinding) EventsToRegister() []framework.ClusterEventWithHint {
{Event: framework.ClusterEvent{Resource: framework.StorageClass, ActionType: framework.Add | framework.Update}},
// We bind PVCs with PVs, so any changes may make the pods schedulable.
{Event: framework.ClusterEvent{Resource: framework.PersistentVolumeClaim, ActionType: framework.Add | framework.Update}},
{Event: framework.ClusterEvent{Resource: framework.PersistentVolume, ActionType: framework.Add | framework.Update}},
{Event: framework.ClusterEvent{Resource: framework.PersistentVolume, ActionType: framework.Add | framework.Update}, QueueingHintFn: pl.isSchedulableAfterPersistentVolumeChange},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why this plugin registers PV events. IMO, The pv events can not directly influence a pod's schedulability. During the process of binding a PV to a PVC, the PV are always updated first, then the PVC is updated. So the later PVC event may make the Pod become schedulable. so registering PV events maybe not necessary but it may accelerate the pod scheduling process (it doesn't wait for the pvc to be bound and put the pod into backoffQ).

It was added by this PR: #100003

cc @jsafrane @sanposhiho @cofyc @Huang-Wei @saad-ali

@bells17
Copy link
Contributor Author

bells17 commented May 31, 2024

/close
#124961 (comment)

@k8s-ci-robot
Copy link
Contributor

@bells17: Closed this PR.

In response to this:

/close
#124961 (comment)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@carlory carlory carlory left review comments

@mattcary mattcary Awaiting requested review from mattcary

@saikat-royc saikat-royc Awaiting requested review from saikat-royc

@utam0k utam0k Awaiting requested review from utam0k

@AxeZhan AxeZhan Awaiting requested review from AxeZhan

@jsafrane jsafrane Awaiting requested review from jsafrane

@xing-yang xing-yang Awaiting requested review from xing-yang

@sanposhiho sanposhiho Awaiting requested review from sanposhiho

Assignees

@carlory carlory

Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@bells17 @k8s-ci-robot @sanposhiho @AxeZhan @carlory