Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In SelectorSpreadPriority, consider all pods when scoring for zone #73711

Closed

Conversation

Ramyak
Copy link
Contributor

@Ramyak Ramyak commented Feb 4, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:
Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per zone, resulting in imbalanced cluster.

If one zone is more loaded than other zones, this makes it worse by getting more and more pods scheduled in the same zone.

Fix: Consider all pods matching all the selectors when spreading across zones.

Which issue(s) this PR fixes:
Fixes #72916

Special notes for your reviewer:

Follow up to
PR: #72801
Issue: #71327

Does this PR introduce a user-facing change?:

In SelectorSpreadPriority, count all pods when spreading for zone.

/sig scheduling

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 4, 2019
@k8s-ci-robot
Copy link
Contributor

Hi @Ramyak. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@Ramyak
Copy link
Contributor Author

Ramyak commented Feb 4, 2019

/assign @k82cn
/assign @bsalamat

@k8s-ci-robot k8s-ci-robot added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Feb 5, 2019
Copy link
Member

@neolit123 neolit123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the PR!
please add a release note instead of NONE in the PR description, that explains the change.

/ok-to-test
/priority important-longterm

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 5, 2019
@Ramyak
Copy link
Contributor Author

Ramyak commented Feb 6, 2019

Ping @bsalamat / @k82cn

@@ -53,6 +53,23 @@ func (f FakePodLister) List(s labels.Selector) (selected []*v1.Pod, err error) {
return selected, nil
}

// Search returns []*v1.Pod matching all selectors
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks just returns selected []*v1.Pod is ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated comment. Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I mean Search only needs to return []*v1.Pod, error seems redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/kubernetes/kubernetes/pull/73711/files#diff-6f5753a252698c11328724f8ed0b007cR51

error here is not optional. It is part of the interface definition.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all right. thanks

return true
}

return cache.FilteredList(matchAllSelctors, selectors[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the length of selectors is always bigger than 0?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is checked in CalculateSpreadPriorityReduce which calls Search as far as I read.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review. Added a len(selectors) == 0 check here too.

Copy link
Member

@szuecs szuecs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good to me, but I didn’t check surroundings.

maxCountByNodeName = result[i].Score
if len(selectors) > 0 {
pods, _ := s.podLister.Search(selectors)
if pods != nil && len(pods) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can drop the nil check, because Go internally handles it in len() and will return 0 length.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks.

@Ramyak Ramyak force-pushed the ramya/deleted-pods-selector-spread branch from ad1cae3 to fbbb3e1 Compare February 13, 2019 01:15
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 13, 2019
@Ramyak Ramyak force-pushed the ramya/deleted-pods-selector-spread branch from fbbb3e1 to 43f88a7 Compare February 13, 2019 01:19
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 15, 2019
@szuecs
Copy link
Member

szuecs commented May 15, 2019

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 15, 2019
@szuecs
Copy link
Member

szuecs commented May 15, 2019

@Ramyak can you please add a release note entry, just follow https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md

@k8s-ci-robot k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label May 16, 2019
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label May 16, 2019
@Ramyak Ramyak force-pushed the ramya/deleted-pods-selector-spread branch from 43f88a7 to 154aaea Compare May 16, 2019 15:04
@Ramyak
Copy link
Contributor Author

Ramyak commented May 16, 2019

/test pull-kubernetes-e2e-gce-100-performance

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 14, 2019
@szuecs
Copy link
Member

szuecs commented Aug 28, 2019

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 28, 2019
@szuecs
Copy link
Member

szuecs commented Aug 28, 2019

@Ramyak would you rebase and resolve the conflicts, please?

@tedyu
Copy link
Contributor

tedyu commented Aug 28, 2019

this makes it worse by getting more and more pods scheduled in the same zone.

Is it possible to reproduce the above in a unit test so that we know the effectiveness of the fix ?

@Ramyak Ramyak force-pushed the ramya/deleted-pods-selector-spread branch from 154aaea to cdd072f Compare September 4, 2019 15:39
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Ramyak
To complete the pull request process, please assign bsalamat
You can assign the PR to them by writing /assign @bsalamat in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Ramyak Ramyak force-pushed the ramya/deleted-pods-selector-spread branch from cdd072f to 43510b7 Compare October 9, 2019 23:45
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jan 8, 2020
@k8s-ci-robot
Copy link
Contributor

@Ramyak: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 7, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SelectorSpreadPriority does not spread pods across zones for pods dropped by predicates
9 participants