Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: concurrent map writes error in VolumeBinding plugin during Filter #96777

Merged
merged 1 commit into from Nov 23, 2020
Merged

fix: concurrent map writes error in VolumeBinding plugin during Filter #96777

merged 1 commit into from Nov 23, 2020

Conversation

lianghao208
Copy link
Member

@lianghao208 lianghao208 commented Nov 21, 2020 •
edited by liggitt

What type of PR is this?

/kind bug
/sig scheduling

What this PR does / why we need it:
this PR fix the issue #96772 mentioning concurrent map writes error in VolumeBinding plugin during Filter. The reason is framework.CycleState will be cloned e.g. when invoking addNominatedPods()

kubernetes/pkg/scheduler/core/generic_scheduler.go

Line 388 in 8095565

stateOut := state.Clone()

when CycleState is cloned during filtering, new mutex is created, but stateData is not cloned, we should set a mutex lock in struct stateData to make sure using the same mutex when updating stateData.podVolumesByNode

Which issue(s) this PR fixes:

Fixes #96772

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fixes a regression in 1.19 where concurrent map write can be encountered in the scheduler in the VolumeBinding plugin filter

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
no

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 21, 2020
@k8s-ci-robot
Copy link
Contributor

Hi @lianghao208. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 21, 2020
@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Nov 21, 2020
@Huang-Wei Huang-Wei mentioned this pull request Nov 21, 2020
Closed
@Huang-Wei
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 21, 2020
@lianghao208
Copy link
Member Author

/retest

2 similar comments
@lianghao208
Copy link
Member Author

/retest

@lianghao208
Copy link
Member Author

/retest

cofyc
cofyc reviewed Nov 21, 2020
View reviewed changes
Copy link
Member

@cofyc cofyc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to add a test case for "concurrent map write" case?

pkg/scheduler/framework/plugins/volumebinding/volume_binding.go Outdated Show resolved Hide resolved
pkg/scheduler/framework/plugins/volumebinding/volume_binding.go Outdated Show resolved Hide resolved
pkg/scheduler/framework/plugins/volumebinding/volume_binding.go Outdated Show resolved Hide resolved
@Huang-Wei
Copy link
Member

is it possible to add a test case for "concurrent map write" case?

@lianghao208 You can try to run related UT by providing -race to see if it's reproducible.

@lianghao208
Copy link
Member Author

lianghao208 commented Nov 21, 2020 •
edited

@lianghao208 You can try to run related UT by providing -race to see if it's reproducible.

@Huang-Wei I don't think providing -race in UT volume_binding_test.go can reproduce it...

@lianghao208
Copy link
Member Author

/retest

@cofyc
Copy link
Member

cofyc commented Nov 22, 2020

Given that VolumeBinding's clone() returns the pointer, so the embedded mutex can protect its data no matter Filter() is called on a nominated node or regular node. I think this PR is good to go. @cofyc WDYT, are you ok with the pure fix, or strongly prefer to add a test to produce the race?

I'm ok with this pure fix.

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, cofyc, lianghao208

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cofyc
Copy link
Member

cofyc commented Nov 22, 2020

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 22, 2020
@Huang-Wei
Copy link
Member

@ahg-g I guess it's too late to catch the 1.20 train. Let's wait for it to be merged in master(1.21)，and cherry pick to 1.20.1?

@lianghao208
Copy link
Member Author

@Huang-Wei If this patch needs backport v.19 and v.20, do I need to submit PR to cherry pick?

@ahg-g
Copy link
Member

ahg-g commented Nov 23, 2020

sg, lets patch back to v1.9 and v1.20

@ahg-g
Copy link
Member

ahg-g commented Nov 23, 2020

/triage accepted
/priority critical-urgent

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 23, 2020
@ahg-g
Copy link
Member

ahg-g commented Nov 23, 2020

actually we can set the milestone to get it merged since this is a critical bug

/milestone v1.20
/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 23, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.20 milestone Nov 23, 2020
@lianghao208
Copy link
Member Author

/retest

@ahg-g
Copy link
Member

ahg-g commented Nov 23, 2020

@lianghao208 once this is merged, please create a patch against the 1.19 branch, the follow command should automatically create one when this is merged:

hack/cherry_pick_pull.sh upstream/release-1.19 96777

@lianghao208
Copy link
Member Author

once this is merged, please create a patch against the 1.19 branch

@ahg-g will do.

@k8s-ci-robot k8s-ci-robot merged commit 7335824 into kubernetes:master Nov 23, 2020
This was referenced Nov 23, 2020
Merged
Closed
@Huang-Wei
Copy link
Member

@Huang-Wei If this patch needs backport v.19 and v.20, do I need to submit PR to cherry pick?

Yes, you can leverage hack/cherry_pick_pull.sh to achieve that.

@ahg-g ahg-g mentioned this pull request Nov 24, 2020
Merged
@Huang-Wei Huang-Wei mentioned this pull request Nov 24, 2020
Closed
k8s-ci-robot added a commit that referenced this pull request Nov 27, 2020
@k8s-ci-robot
Merge pull request #96809 from lianghao208/automated-cherry-pick-of-#…
0714c69 
…96777-upstream-release-1.19

Automated cherry pick of #96777: fix: concurrent map writes error in VolumeBinding plugin during Filter
@Huang-Wei Huang-Wei mentioned this pull request Apr 28, 2021
Merged
@liggitt liggitt added the kind/regression Categorizes issue or PR as related to a regression from a prior release. label Apr 26, 2022
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Sep 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cofyc cofyc cofyc left review comments

@Huang-Wei Huang-Wei Huang-Wei left review comments

@humblec humblec Awaiting requested review from humblec

@msau42 msau42 Awaiting requested review from msau42

Assignees

@cofyc cofyc

@ahg-g ahg-g

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/regression Categorizes issue or PR as related to a regression from a prior release. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.

concurrent map writes in VolumeBinding plugin during Filter
6 participants
@lianghao208 @k8s-ci-robot @Huang-Wei @ahg-g @cofyc @liggitt