Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-1.19] Balance nodes in scheduling e2e #98810

Merged
merged 2 commits into from Mar 11, 2021
Merged

[release-1.19] Balance nodes in scheduling e2e #98810

merged 2 commits into from Mar 11, 2021

Conversation

damemi
Copy link
Contributor

@damemi damemi commented Feb 5, 2021 •
edited

What type of PR is this?
/kind flake

What this PR does / why we need it:
Backporting #98699 (and 2318992) to ensure nodes are balanced in scheduling e2e

In clusters that are unevenly running more components than a vanilla k8s install, resource request variance can be amplified which will influence scheduler scoring decisions. Occasionally this resource-balancing score influence is more than the desired spreading of the test, leading to flakes (of pods being unevenly spread). An example of this variance was observed here: openshift#547 (comment)

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig scheduling

@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Feb 5, 2021
@k8s-ci-robot k8s-ci-robot added do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. release-note-none Denotes a PR that doesn't merit a release note. kind/flake Categorizes issue or PR as related to a flaky test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 5, 2021
@k8s-ci-robot
Copy link
Contributor

@damemi: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Feb 5, 2021
@k8s-ci-robot k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Feb 5, 2021
damemi
damemi commented Feb 5, 2021
View reviewed changes
Copy link
Contributor Author

@damemi damemi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @alculquicondor
this would be good to backport, should be a clean change but please take a look

@damemi damemi changed the title Balance nodes in scheduling e2e [release-1.19] Balance nodes in scheduling e2e Feb 5, 2021
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 5, 2021
ingvagabund and others added 2 commits February 5, 2021 15:01
@ingvagabund @damemi
e2e: Pod should avoid nodes that have avoidPod annotation: clean rema…
36544ac 
…ining pods

The test is not cleaning all pods it created.
Memory balancing pods are deleted once the test namespace is.
Thus, leaving the pods running or in terminating state when a new test is run.
In case the next test is "[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run",
the test can fail.
@damemi
Balance nodes in scheduling e2e
6e297f9 
This adds a call to createBalancedPods during the ubernetes_lite scheduling e2es,
which are prone to improper score balancing due to unbalanced utilization.
@damemi damemi force-pushed the 1.19-balance-nodes-ubernetes branch from 5b0a7d5 to 6e297f9 Compare February 5, 2021 20:02
@alculquicondor
Copy link
Member

FYI @ingvagabund

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, damemi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Feb 5, 2021
@alculquicondor
Copy link
Member

Ref #94684

/retest

@hasheddan hasheddan mentioned this pull request Feb 12, 2021
Merged
@feiskyer feiskyer removed the do-not-merge/cherry-pick-not-approved Indicates that a PR is not yet approved to merge into a release branch. label Mar 11, 2021
@feiskyer feiskyer added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 11, 2021
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

2 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit de3ee48 into kubernetes:release-1.19 Mar 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@draveness draveness Awaiting requested review from draveness

@k82cn k82cn Awaiting requested review from k82cn

@alculquicondor alculquicondor Awaiting requested review from alculquicondor

Assignees

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.19
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@damemi @k8s-ci-robot @alculquicondor @fejta-bot @feiskyer @ingvagabund