Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

eviction: Deflake TestStart #108010

Merged
merged 1 commit into from
Mar 17, 2022
Merged

eviction: Deflake TestStart #108010

merged 1 commit into from
Mar 17, 2022

Conversation

endocrimes
Copy link
Member

@endocrimes endocrimes commented Feb 8, 2022 •
edited

What type of PR is this?

/kind flake
/sig node

What this PR does / why we need it:

TestStart was previously flaky. In approx 100_000 local runs, it failed
70% of the time, and has been mentioned as a flaky unit test several times in
the past.

This flake was due to a race condition with the logic as written and the
go scheduler. UpdateThreshold calls notifier.Start(events) in a new Go
Routine, which is not guarunteed to be called immediately.

This meant that if m.Start() was called before notifier.Start(), the
test would fail, as the notifier would not have been started before the
4 events were processed and lock released.

Here, we update the test to more closely match the intended application
behaviour, and have events passed to the channel when Start is called
on the notifier.

This ensures that -Start gets called and additionally validates
that the correct channel is provided to the notifier.

Stop was never called previously, as it only gets called on a subsequent
call to UpdateThreshold. AnyTimes() hid that this did not occur.

Which issue(s) this PR fixes:

Related to: #107708

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/flake Categorizes issue or PR as related to a flaky test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. sig/node Categorizes an issue or PR as relevant to SIG Node. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 8, 2022
@endocrimes
eviction: Deflake TestStart
3630328 
TestStart was previously flaky. In approx 100_000 local runs, it failed
about 70% of the time, and has been mentioned as a flaky unit test in
the past.

This flake was due to a race condition with the logic as written and the
go scheduler. UpdateThreshold calls `notifier.Start(events)` in a new Go
Routine, which is not guarunteed to be called immediately.

This meant that if `m.Start()` was called before `notifier.Start()`, the
test would fail, as the notifier would not have been started before the
4 events were processed and lock released.

Here, we update the test to more closely match the intended application
behaviour, and have events passed to the channel when `Start` is called
on the notifier.

This ensures that -Start gets called and additionally validates
that the correct channel is provided to the notifier.

Stop was never called previously, as it only gets called on a subsequent
call to UpdateThreshold. `AnyTimes()` hid that this did not occur.
@endocrimes
Copy link
Member Author

/test pull-kubernetes-unit

lol other flakes on a flake fix

@MadhavJivrajani
Copy link
Contributor

/priority important-soon
/triage accepted

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 9, 2022
@ehashman ehashman moved this from Triage to PRs - Needs Reviewer in SIG Node CI/Test Board Feb 9, 2022
pacoxu
pacoxu approved these changes Feb 10, 2022
View reviewed changes
Copy link
Member

@pacoxu pacoxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
tested

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 10, 2022
@pacoxu
Copy link
Member

pacoxu commented Feb 10, 2022

/assign mrunalp

@ehashman ehashman moved this from PRs - Needs Reviewer to PRs - Needs Approver in SIG Node CI/Test Board Feb 11, 2022
@endocrimes
Copy link
Member Author

/assign @klueska

adisky
adisky reviewed Mar 2, 2022
View reviewed changes
Copy link
Contributor

@adisky adisky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@adisky
Copy link
Contributor

adisky commented Mar 2, 2022

/assign @klueska @dchen1107 @derekwaynecarr
(for approval)

@eddiezane eddiezane mentioned this pull request Mar 3, 2022
Merged
@endocrimes
Copy link
Member Author

ping

ehashman
ehashman reviewed Mar 15, 2022
View reviewed changes
Copy link
Member

@ehashman ehashman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@pacoxu
Copy link
Member

pacoxu commented Mar 16, 2022

27 test failures (1 today) look like linksearch githubfile bug

It is still one of the top flakes in http://storage.googleapis.com/k8s-metrics/flakes-latest.json.

@dims
Copy link
Member

dims commented Mar 17, 2022

/approve

thanks @endocrimes

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, endocrimes, pacoxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2022
@k8s-ci-robot k8s-ci-robot merged commit 56062f7 into kubernetes:master Mar 17, 2022
SIG Node CI/Test Board automation moved this from PRs - Needs Approver to Done Mar 17, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.24 milestone Mar 17, 2022
@endocrimes endocrimes deleted the dani/eviction-flake branch March 17, 2022 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ehashman ehashman ehashman left review comments

@adisky adisky adisky left review comments

@pacoxu pacoxu pacoxu approved these changes

@sjenning sjenning Awaiting requested review from sjenning

Assignees

@mrunalp mrunalp

@klueska klueska

@pacoxu pacoxu

@ehashman ehashman

@derekwaynecarr derekwaynecarr

@adisky adisky

@dchen1107 dchen1107

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node CI/Test Board
Done
SIG Node CI/Test Board
Archived in project
Milestone
v1.24
Development

Successfully merging this pull request may close these issues.

None yet