Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix flake in detach tests #107924

Merged
merged 1 commit into from
Feb 4, 2022
Merged

fix flake in detach tests #107924

merged 1 commit into from
Feb 4, 2022

Conversation

gnufied
Copy link
Member

@gnufied gnufied commented Feb 2, 2022

Fixes #99877

Before - flakes

After:

~> ~/goal/bin/stress ./reconciler.test -test.run=Test_Run_Positive_VolumeAttachMountUnmountDetach                                                                                                                                                                       
5s: 454 runs so far, 0 failures                                                                                                                                                                                                                                                           
10s: 930 runs so far, 0 failures                                                                                                                                                                                                                                                          
15s: 1415 runs so far, 0 failures                                                                                                                                                                                                                                                         
20s: 1900 runs so far, 0 failures                                                                                                                                                                                                                                                         
25s: 2394 runs so far, 0 failures                                                                                                                                                                                                                                                         
30s: 2888 runs so far, 0 failures                                                                                                                                                                                                                                                         
^C

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Feb 2, 2022
@k8s-ci-robot
Copy link
Contributor

@gnufied: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 2, 2022
@gnufied
Copy link
Member Author

gnufied commented Feb 2, 2022

/release-note-none

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Feb 2, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 2, 2022
@gnufied
Copy link
Member Author

gnufied commented Feb 2, 2022

/kind flake

@k8s-ci-robot k8s-ci-robot added kind/flake Categorizes issue or PR as related to a flaky test. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Feb 2, 2022
@gnufied
Copy link
Member Author

gnufied commented Feb 2, 2022

cc @BenTheElder

@gnufied
Copy link
Member Author

gnufied commented Feb 3, 2022

/retest

aojea
aojea reviewed Feb 3, 2022
View reviewed changes
pkg/volume/testing/testing.go

node := string(nodeName)
volumeNodes, exist := fv.VolumesAttached[volumeName]
if !exist || !volumeNodes.Has(node) {
return fmt.Errorf("trying to detach volume %q that is not attached to the node %q", volumeName, node)
}

fv.DetachCallCount++
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without understanding the mechanics of the tests, this is not counting all calls to detach, the volume fails to be attached or doesn't exist we don't consider that call

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep that sounds right. The test is somewhat racy and detach can get called more than once for same volume, while we wait for detach in tests. Not counting calls of already detached volumes fixes the flake.

@gnufied
Copy link
Member Author

gnufied commented Feb 3, 2022

/assign @jingxu97 @mattcary

@BenTheElder
Copy link
Member

tricky!
thanks for digging into this!
/lgtm
/hold
(feel free to remove, hold in case you want another pair of eyes on this, I also think this is right :-))

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 4, 2022
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 4, 2022
@gnufied
Copy link
Member Author

gnufied commented Feb 4, 2022

nah, it should be fine. This is simple enough change.

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 4, 2022
@mattcary
Copy link
Contributor

mattcary commented Feb 4, 2022

/lgtm

nice

@k8s-ci-robot k8s-ci-robot merged commit 8f5a12d into kubernetes:master Feb 4, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.24 milestone Feb 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aojea aojea aojea left review comments

@Jiawei0227 Jiawei0227 Awaiting requested review from Jiawei0227

@msau42 msau42 Awaiting requested review from msau42

Assignees

@BenTheElder BenTheElder

@jingxu97 jingxu97

@mattcary mattcary

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.24
Development

Successfully merging this pull request may close these issues.

[flaky test] Test_Run_Positive_VolumeAttachMountUnmountDetach
6 participants
@gnufied @k8s-ci-robot @BenTheElder @mattcary @aojea @jingxu97