Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[test/kubelet]: Fix deadlock in watch manager test #92920

Merged
merged 1 commit into from Aug 27, 2020
Merged

[test/kubelet]: Fix deadlock in watch manager test #92920

merged 1 commit into from Aug 27, 2020

Conversation

gaurav1086
Copy link
Contributor

Signed-off-by: Gaurav Singh gaurav1086@gmail.com

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:

/kind api-change
/kind bug
/kind cleanup
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
Fixes a deadlock condition if all the goroutines (10) will fail and try to write to the errCh and block instead of finishing the execution, while the main thread will also block through wait() for all the goroutines to finish before it tries to read from the errCh in the select operation, thus causing a deadlock. A simple illustration: https://play.golang.com/p/x0WpLjw_f1i

The solution is to create a buffered channel upto the size of the total number of the goroutines so they can all finish execution asynchronously.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
https://play.golang.com/p/x0WpLjw_f1i
Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 8, 2020
@k8s-ci-robot
Copy link
Contributor

Hi @gaurav1086. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 8, 2020
@gaurav1086
Copy link
Contributor Author

/assign @msau42

neolit123
neolit123 reviewed Jul 8, 2020
View reviewed changes
Copy link
Member

@neolit123 neolit123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/sig node

@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Jul 8, 2020
@neolit123
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 8, 2020
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 8, 2020
@neolit123
Copy link
Member

might be worth to include this PR in the 1.19 release?
(/milestone v1.19)

@gaurav1086
Copy link
Contributor Author

/retest

@SergeyKanzhelev
Copy link
Member

/retest

@msau42
Copy link
Member

msau42 commented Jul 9, 2020

/approve

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 9, 2020
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@gaurav1086 gaurav1086 force-pushed the TestWatchBasedManager_fix_deadlock branch from 64a63d3 to 5d74555 Compare July 12, 2020 14:45
@gaurav1086
[test/kubelet]: Fix deadlock in watch manager test
22dc474 
Signed-off-by: Gaurav Singh <gaurav1086@gmail.com>
@gaurav1086 gaurav1086 force-pushed the TestWatchBasedManager_fix_deadlock branch from 5d74555 to 22dc474 Compare July 12, 2020 14:56
@gaurav1086
Copy link
Contributor Author

/retest

neolit123
neolit123 reviewed Jul 20, 2020
View reviewed changes
Copy link
Member

@neolit123 neolit123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve
/priority important-longterm

@k8s-ci-robot k8s-ci-robot added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 20, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaurav1086, msau42, neolit123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@neolit123
Copy link
Member

/milestone v1.19
this seems appropriate to include in 1.19?

@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Jul 20, 2020
@SergeyKanzhelev
Copy link
Member

/retest

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@liggitt
Copy link
Member

liggitt commented Jul 21, 2020

this is a good catch, but only really causes problems when the test would have already failed

moving to 1.20 since we're approaching code thaw and any 1.19 fixes will need cherry picks shortly

/milestone v1.20

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.19, v1.20 Jul 21, 2020
@liggitt
Copy link
Member

liggitt commented Jul 21, 2020

/retest

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

2 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@liggitt
Copy link
Member

liggitt commented Jul 21, 2020

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/92920/pull-kubernetes-node-e2e/1285251532081598464 flake is #93210

@liggitt liggitt removed this from the v1.20 milestone Aug 26, 2020
@liggitt liggitt added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Aug 26, 2020
@liggitt liggitt added this to the v1.20 milestone Aug 26, 2020
@justaugustus
Copy link
Member

//test pull-kubernetes-e2e-kind-ipv6

@justaugustus
Copy link
Member

Ooop.
/test pull-kubernetes-e2e-kind-ipv6

@justaugustus
Copy link
Member

/test pull-kubernetes-conformance-kind-ga-only-parallel
/test pull-kubernetes-integration

@k8s-ci-robot k8s-ci-robot merged commit 343596b into kubernetes:master Aug 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@neolit123 neolit123 neolit123 left review comments

@SataQiu SataQiu Awaiting requested review from SataQiu

@SergeyKanzhelev SergeyKanzhelev Awaiting requested review from SergeyKanzhelev

Assignees

@neolit123 neolit123

@msau42 msau42

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@gaurav1086 @k8s-ci-robot @neolit123 @SergeyKanzhelev @msau42 @fejta-bot @liggitt @justaugustus