Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deflake TestExpectationsOnRecreate #93617

Merged
merged 1 commit into from Aug 6, 2020

Conversation

liggitt
Copy link
Member

@liggitt liggitt commented Jul 31, 2020

What type of PR is this?
/kind flake

What this PR does / why we need it:
Fixes a flake in TestExpectationsOnRecreate due to the test not considering a requeue caused by the update of the DS during the processing of the initial create

Which issue(s) this PR fixes:
xref #93605
fixes #93604

Does this PR introduce a user-facing change?:

NONE

/sig apps
/cc @tnozicka @janetkuo

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/flake Categorizes issue or PR as related to a flaky test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. sig/apps Categorizes an issue or PR as relevant to SIG Apps. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 31, 2020
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2020
@dims
Copy link
Member

dims commented Aug 1, 2020

/retest

1 similar comment
@liggitt
Copy link
Member Author

liggitt commented Aug 2, 2020

/retest

@liggitt liggitt added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Aug 2, 2020
@k8s-ci-robot k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Aug 2, 2020
@liggitt liggitt added this to the v1.19 milestone Aug 2, 2020
@liggitt
Copy link
Member Author

liggitt commented Aug 2, 2020

/retest

@liggitt
Copy link
Member Author

liggitt commented Aug 2, 2020

/skip

@liggitt
Copy link
Member Author

liggitt commented Aug 4, 2020

cc @kubernetes/sig-apps-pr-reviews

Copy link
Contributor

@tnozicka tnozicka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt thanks for looking into the test and finding out the cause!

I've sketched an alternative in #93725 while I was debugging the test with handling the addNode case (https://github.com/kubernetes/kubernetes/pull/93725/files#diff-54638f25c4f816a4ba21c67ef036c363R504)

I am also fine going with this PR and picking it or addressing that as a follow up.

/approve

if actual := dsc.queue.Len(); actual != expected {
t.Fatalf("expected queue len to remain at %d, got %d", expected, actual)
}
time.Sleep(10 * time.Millisecond)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if we need the wait, it gets some chance at catching potential error (but we are only oscillating between 0 and 1) but would't the same be seen with go test -count=100 -parallel=100? Although the cost isn't high.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm mostly trying to avoid hotlooping the CPU

t.Fatal("Unexpected item in the queue")
// process updates DS, update adds to queue
waitForQueueLength(1, "updated DS")
ok = dsc.processNextWorkItem()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be better just to

dsc.queue.Get()
dsc.queue.Done(key)

to avoid possible side effects of sync, although those should be constant on second run

Copy link
Member Author

@liggitt liggitt Aug 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to avoid possible side effects of sync, although those should be constant on second run

I actually wanted to exercise that and verify a requeue doesn't occur, since a second sync is really what happens

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose I prefer a separate test case but it's fine

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, tnozicka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Contributor

@tnozicka tnozicka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2020
@liggitt
Copy link
Member Author

liggitt commented Aug 6, 2020

/retest

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Aug 6, 2020

@liggitt: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-kind-ipv6 8e6a866 link /test pull-kubernetes-e2e-kind-ipv6

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot merged commit f5334fa into kubernetes:master Aug 6, 2020
@liggitt liggitt deleted the deflake-ds-test branch August 6, 2020 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Daemonset TestExpectationsOnRecreate fails consistently
4 participants