Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add separate cache for CSINode #80084

Merged
merged 2 commits into from Aug 2, 2019
Merged

Add separate cache for CSINode #80084

merged 2 commits into from Aug 2, 2019

Conversation

bertinatto
Copy link
Member

@bertinatto bertinatto commented Jul 12, 2019 •
edited

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Sparkled from the discussion in #79416 (comment), this PR introduces a new structure to store cached CSINode objects in the scheduler.

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 12, 2019
@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 12, 2019
draveness
draveness reviewed Jul 12, 2019
View reviewed changes
Copy link
Contributor

@draveness draveness left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jul 12, 2019
@bertinatto
Copy link
Member Author

bertinatto commented Jul 12, 2019 •
edited

@bsalamat, @msau42, @tedyu:

This is work-in-progress, but could you provide early feedback on the mechanism used here to prevent the issues we discussed in #79416?

In summary, this PR decouples the lifecycle of CSINode objects from the nodes cache. As a result, we no longer need to worry about creating/deleting nodeInfoListItem.

I believe the main change we should look at at this point is in the file cache.go:

https://github.com/kubernetes/kubernetes/pull/80084/files#diff-f4a894ca5e905aa5f613269fc967fe2c

bertinatto
bertinatto commented Jul 12, 2019
View reviewed changes
pkg/scheduler/factory/factory.go
@@ -574,6 +574,7 @@ func (c *configFactory) getPluginArgs() (*PluginFactoryArgs, error) {
NodeLister: &nodeLister{c.nodeLister},
PDBLister: c.pdbLister,
NodeInfo: c.schedulerCache,
CSINodeInfo: c.schedulerCache,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I re-used the NodeInfo interface above by adding a new GetCSINodeInfo() method to it. However, the patch got fairly bigger because the NodeInfo interface is used in many places in k8s.

@bertinatto
Copy link
Member Author

/test pull-kubernetes-verify

@bertinatto
Copy link
Member Author

/assign @msau42, @bsalamat

@bertinatto
Copy link
Member Author

/test pull-kubernetes-bazel-test

msau42
msau42 reviewed Jul 16, 2019
View reviewed changes
pkg/scheduler/algorithm/predicates/csi_volume_predicate.go Outdated Show resolved Hide resolved
pkg/scheduler/algorithm/predicates/csi_volume_predicate.go Outdated Show resolved Hide resolved
pkg/scheduler/algorithm/predicates/csi_volume_predicate.go Outdated

csiNode, err := c.csiNodeInfo.GetCSINodeInfo(node.Name)
if err != nil {
klog.V(5).Infof("Could not get a CSINode object for the node: %v", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is possible for older nodes?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, nodes where for some reason there's no CSINode. On second thought, I think we want to return the error and prevent scheduling the pod (recording the error in the events).

@bertinatto bertinatto force-pushed the cache_csinode branch 6 times, most recently from 9049a56 to 0ee6e2b Compare July 17, 2019 07:57
@bertinatto bertinatto changed the title [WIP] Add separate cache for CSINode Add separate cache for CSINode Jul 17, 2019
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 17, 2019
@bertinatto
Copy link
Member Author

/test pull-kubernetes-node-e2e-containerd

@bertinatto bertinatto force-pushed the cache_csinode branch 3 times, most recently from 2da5623 to c0648bd Compare July 30, 2019 08:20
@bertinatto
Copy link
Member Author

/test pull-kubernetes-kubemark-e2e-gce-big

@bertinatto
Copy link
Member Author

@ahg-g, @bsalamat, @msau42:

All comments should be addressed. Could you please take another look? Thanks!

@msau42
Copy link
Member

msau42 commented Jul 31, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 31, 2019
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 1, 2019
@bertinatto
Copy link
Member Author

/test pull-kubernetes-e2e-gce-device-plugin-gpu

@bertinatto
Copy link
Member Author

bertinatto commented Aug 1, 2019 •
edited

@msau42, lost your lgtm because this PR needed rebase.

@msau42
Copy link
Member

msau42 commented Aug 1, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 1, 2019
bsalamat
bsalamat reviewed Aug 1, 2019
View reviewed changes
Copy link
Member

@bsalamat bsalamat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bertinatto, bsalamat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 1, 2019
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@bertinatto
Copy link
Member Author

/retest

@bertinatto
Copy link
Member Author

/test pull-kubernetes-integration

TestRescheduleProvisioning timed out. Ran it locally in a loop for 25 minutes and no errors.

@bertinatto
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit 73b1bcb into kubernetes:master Aug 2, 2019
@k8s-ci-robot k8s-ci-robot added this to the v1.16 milestone Aug 2, 2019
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Aug 2, 2019 •
edited

@bertinatto: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-integration c5d9af2 link /test pull-kubernetes-integration

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tedyu
Copy link
Contributor

tedyu commented Aug 2, 2019

With this PR committed, is schedulerCache#csiNodes still needed ?

@bertinatto bertinatto mentioned this pull request Nov 4, 2019
Closed
6 tasks
@cofyc cofyc mentioned this pull request Dec 1, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bsalamat bsalamat bsalamat left review comments

@msau42 msau42 msau42 left review comments

@Jerry-Ge Jerry-Ge Jerry-Ge left review comments

@draveness draveness draveness requested changes

@k82cn k82cn Awaiting requested review from k82cn

@wgliang wgliang Awaiting requested review from wgliang

Assignees

@bsalamat bsalamat

@msau42 msau42

@ahg-g ahg-g

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.16
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@bertinatto @msau42 @k8s-ci-robot @fejta-bot @tedyu @draveness @bsalamat @Jerry-Ge @ahg-g