Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-1.20] Balance nodes in scheduling e2e #98811

Merged
merged 2 commits into from Mar 12, 2021
Merged

[release-1.20] Balance nodes in scheduling e2e #98811

merged 2 commits into from Mar 12, 2021

Conversation

damemi
Copy link
Contributor

@damemi damemi commented Feb 5, 2021 •
edited

This adds a call to createBalancedPods during the ubernetes_lite scheduling e2es,
which are prone to improper score balancing due to unbalanced utilization.

What type of PR is this?
/kind flake

What this PR does / why we need it:
Backport of #98699 (and #94684)

In clusters that are unevenly running more components than a vanilla k8s install, resource request variance can be amplified which will influence scheduler scoring decisions. Occasionally this resource-balancing score influence is more than the desired spreading of the test, leading to flakes (of pods being unevenly spread). An example of this variance was observed here: openshift#547 (comment)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig scheduling

@k8s-ci-robot k8s-ci-robot added this to the v1.20 milestone Feb 5, 2021
@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. kind/flake Categorizes issue or PR as related to a flaky test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 5, 2021
@k8s-ci-robot
Copy link
Contributor

@damemi: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Feb 5, 2021
@k8s-ci-robot k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Feb 5, 2021
damemi
damemi commented Feb 5, 2021
View reviewed changes
Copy link
Contributor Author

@damemi damemi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @alculquicondor
another backport

@alculquicondor
Copy link
Member

it looks like it needs some manual work

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 5, 2021
@damemi
Copy link
Contributor Author

damemi commented Feb 5, 2021

it looks like it needs some manual work

Yeah, it was missing the changes from #94684 (which added the cleanUp() function to ensure all of the pods get cleaned up)

@alculquicondor
Copy link
Member

nit: reorder the commits

ingvagabund and others added 2 commits February 5, 2021 15:02
@ingvagabund @damemi
e2e: Pod should avoid nodes that have avoidPod annotation: clean rema…
1598f8b 
…ining pods

The test is not cleaning all pods it created.
Memory balancing pods are deleted once the test namespace is.
Thus, leaving the pods running or in terminating state when a new test is run.
In case the next test is "[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run",
the test can fail.
@damemi
Balance nodes in scheduling e2e
2927f04 
This adds a call to createBalancedPods during the ubernetes_lite scheduling e2es,
which are prone to improper score balancing due to unbalanced utilization.
@damemi damemi force-pushed the 1.20-balance-nodes-ubernetes branch from c18eeb5 to 2927f04 Compare February 5, 2021 20:02
@alculquicondor
Copy link
Member

FYI @ingvagabund

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 5, 2021
@alculquicondor
Copy link
Member

/hold

add to the description a link to the other PR that is being backported

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 5, 2021
@damemi
Copy link
Contributor Author

damemi commented Feb 5, 2021

add to the description a link to the other PR that is being backported

Done! thanks

@alculquicondor
Copy link
Member

/hold cancel

@kubernetes/release-managers

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 5, 2021
hasheddan
hasheddan approved these changes Feb 5, 2021
View reviewed changes
Copy link
Contributor

@hasheddan hasheddan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/retest

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, damemi, hasheddan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@adtac
Copy link
Member

adtac commented Feb 8, 2021

/test pull-kubernetes-node-e2e
(test failed after 22s)

@hasheddan
Copy link
Contributor

hasheddan commented Feb 12, 2021 •
edited

@damemi is this also applicable to 1.19 and 1.18? I am less concerned about this being backported to those branches since it is a test-only change, but we generally prefer to merge CPs to all applicable branches at the same time so they are included in relevant round of patches.

I see the 1.19 backport is already open #98810. Any word on 1.18?

@feiskyer feiskyer added cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. and removed do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. labels Mar 11, 2021
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

3 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit feae2a5 into kubernetes:release-1.20 Mar 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hasheddan hasheddan hasheddan approved these changes

@alculquicondor alculquicondor Awaiting requested review from alculquicondor

@resouer resouer Awaiting requested review from resouer

Assignees

@alculquicondor alculquicondor

@hasheddan hasheddan

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@damemi @k8s-ci-robot @alculquicondor @adtac @hasheddan @fejta-bot @feiskyer @ingvagabund