Bug 1845202: Fix deadlock, fix service lag. #178

squeed · 2020-06-08T14:36:47Z

Backport of two upstream PRs:

ovn-org/ovn-kubernetes#1378 - long service creation time.
ovn-org/ovn-kubernetes#1384 - informer deadlock

This fixes two issues I've seen during debugging tests.

The only reason to lock the events array was to ensure that informer shutdown worked correctly (mostly for unit tests), and that when a nil event queue was found to stop distributing events to it. Instead wait for all processEvent() goroutines for each event queue to terminate before returning from Shutdown(), which handles stopping event distribution. Just leave the event queues open as there is no way to safely close now, because the shared informer may still be delivering events to the factories handler before it notices that the stop channel has been closed. This isn't a problem though, since we only ever stop the informer on exit anyway. Signed-off-by: Dan Williams <dcbw@redhat.com>

Save on some nbctl calls. Signed-off-by: Casey Callendrello <cdc@redhat.com>

openshift-ci-robot · 2020-06-08T14:36:50Z

@squeed: This pull request references Bugzilla bug 1841828, which is invalid:

expected Bugzilla bug 1841828 to depend on a bug targeting a release in 4.6.0, 4.6.z and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

bug 1841828: OVN fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-08T14:37:26Z

@squeed: This pull request references Bugzilla bug 1841828, which is invalid:

expected Bugzilla bug 1841828 to depend on a bug targeting a release in 4.6.0, 4.6.z and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

bug 1841828: OVN fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed · 2020-06-08T14:43:44Z

/test e2e-gcp-ovn

openshift-ci-robot · 2020-06-08T14:43:59Z

@squeed: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test e2e-metal-ipi
/test images

Use /test all to run all jobs.

In response to this:

/test e2e-gcp-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-08T16:02:58Z

@squeed: This pull request references Bugzilla bug 1845202, which is invalid:

expected Bugzilla bug 1845202 to depend on a bug targeting a release in 4.6.0, 4.6.z and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1845202: Fix deadlock, fix service lag.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed · 2020-06-08T16:03:06Z

/hold
until CI is fixed.

squeed · 2020-06-08T16:45:51Z

/bugzilla refresh

openshift-ci-robot · 2020-06-08T16:45:53Z

@squeed: This pull request references Bugzilla bug 1845202, which is invalid:

expected Bugzilla bug 1845202 to depend on a bug targeting a release in 4.6.0, 4.6.z and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed · 2020-06-08T16:46:34Z

/bugzilla refresh

openshift-ci-robot · 2020-06-08T16:46:43Z

@squeed: This pull request references Bugzilla bug 1845202, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1841828 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Bugzilla bug 1841828 targets the "4.6.0" release, which is one of the valid target releases: 4.6.0, 4.6.z
bug has dependents

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-09T07:37:28Z

@squeed: This pull request references Bugzilla bug 1845202. The bug has been updated to no longer refer to the pull request using the external bug tracker.

In response to this:

Bug 1845202: Fix deadlock, fix service lag.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-09T07:37:34Z

@squeed: This pull request references Bugzilla bug 1845202, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1841828 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Bugzilla bug 1841828 targets the "4.6.0" release, which is one of the valid target releases: 4.6.0, 4.6.z
bug has dependents

In response to this:

Bug 1845202: Fix deadlock, fix service lag.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed · 2020-06-09T08:51:45Z

/retest

squeed · 2020-06-09T12:00:22Z

aws terraform failure
/test e2e-aws-ovn

gcp passed.

pecameron

/lgtm

openshift-ci-robot · 2020-06-09T12:31:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pecameron, squeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [pecameron,squeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

squeed · 2020-06-09T12:39:20Z

/hold cancel
now that CI is fixed

openshift-bot · 2020-06-09T12:43:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

squeed · 2020-06-09T14:01:05Z

e2e-aws and e2e-gcp passed! Awesome.

squeed · 2020-06-09T14:33:47Z

Does e2e-metal-ipi even use ovn-kubernetes? Also, I see that openshift/release#9514 was just merged, so it won't ever go green on this PR.

squeed · 2020-06-09T14:38:53Z

/override ci/prow/e2e-metal-ipi

openshift-ci-robot · 2020-06-09T14:39:17Z

@squeed: Overrode contexts on behalf of squeed: ci/prow/e2e-metal-ipi

In response to this:

/override ci/prow/e2e-metal-ipi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed · 2020-06-09T14:39:23Z

Yeah, confirmed, metal-ipi doesn't even install ovn-kubernetes. It's safe to override.

openshift-ci-robot · 2020-06-09T14:40:27Z

@squeed: All pull requests linked via external trackers have merged: openshift/ovn-kubernetes#178. Bugzilla bug 1845202 has been moved to the MODIFIED state.

In response to this:

Bug 1845202: Fix deadlock, fix service lag.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2020-06-09T19:39:10Z

@abhat: cannot fork openshift/ovn-kubernetes: fork of openshift/ cannot show up on GitHub: timed out waiting for openshift-cherrypick-robot/ to appear on GitHub: status code 404 not one of [200], body: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abhat · 2020-06-09T19:39:26Z

/cherry-pick release-4.4

openshift-cherrypick-robot · 2020-06-09T19:45:10Z

@abhat: cannot fork openshift/ovn-kubernetes: fork of openshift/ cannot show up on GitHub: timed out waiting for openshift-cherrypick-robot/ to appear on GitHub: status code 404 not one of [200], body: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}

In response to this:

/cherrypick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2020-06-09T19:51:10Z

@abhat: cannot fork openshift/ovn-kubernetes: fork of openshift/ cannot show up on GitHub: timed out waiting for openshift-cherrypick-robot/ to appear on GitHub: status code 404 not one of [200], body: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

stevekuznetsov · 2020-06-09T20:20:04Z

/cherry-pick release-4.4

openshift-cherrypick-robot · 2020-06-09T20:20:10Z

@stevekuznetsov: #178 failed to apply on top of branch "release-4.4":

Using index info to reconstruct a base tree...
M	go-controller/cmd/ovnkube/ovnkube.go
M	go-controller/hybrid-overlay/cmd/hybrid-overlay-node/hybrid-overlay-node.go
M	go-controller/hybrid-overlay/pkg/controller/master_test.go
M	go-controller/hybrid-overlay/pkg/controller/node_linux_test.go
M	go-controller/pkg/factory/factory.go
M	go-controller/pkg/node/gateway_init_linux_test.go
M	go-controller/pkg/ovn/master_test.go
M	go-controller/pkg/ovn/ovn_test.go
Falling back to patching base and 3-way merge...
Auto-merging go-controller/pkg/ovn/ovn_test.go
Auto-merging go-controller/pkg/ovn/master_test.go
Auto-merging go-controller/pkg/node/gateway_init_linux_test.go
Auto-merging go-controller/pkg/factory/factory.go
CONFLICT (content): Merge conflict in go-controller/pkg/factory/factory.go
Auto-merging go-controller/hybrid-overlay/pkg/controller/node_linux_test.go
Auto-merging go-controller/hybrid-overlay/pkg/controller/master_test.go
Auto-merging go-controller/hybrid-overlay/cmd/hybrid-overlay-node/hybrid-overlay-node.go
CONFLICT (content): Merge conflict in go-controller/hybrid-overlay/cmd/hybrid-overlay-node/hybrid-overlay-node.go
Auto-merging go-controller/cmd/ovnkube/ovnkube.go
CONFLICT (content): Merge conflict in go-controller/cmd/ovnkube/ovnkube.go
error: Failed to merge in the changes.
Patch failed at 0001 factory: remove unecessary locking for enqueueEevent

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dcbw and others added 2 commits June 8, 2020 16:19

master: batch loadbalancer acl updates

c481cd4

Save on some nbctl calls. Signed-off-by: Casey Callendrello <cdc@redhat.com>

openshift-ci-robot added the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Jun 8, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 8, 2020

openshift-ci-robot requested review from dcbw and pecameron June 8, 2020 14:37

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 8, 2020

squeed changed the title ~~bug 1841828: OVN fixes~~ Bug 1845202: Fix deadlock, fix service lag. Jun 8, 2020

openshift-ci-robot removed the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Jun 8, 2020

openshift-ci-robot added the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Jun 8, 2020

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 8, 2020

squeed closed this Jun 9, 2020

squeed reopened this Jun 9, 2020

openshift-ci-robot added bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. and removed bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. labels Jun 9, 2020

pecameron reviewed Jun 9, 2020

View reviewed changes

openshift-ci-robot assigned pecameron Jun 9, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 9, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 9, 2020

knobunc added the staff-eng-approved Indicates a release branch PR has been approved by a staff engineer (formerly group/pillar lead). label Jun 9, 2020

openshift-merge-robot merged commit b9a8857 into openshift:release-4.5 Jun 9, 2020

abhat mentioned this pull request Jun 9, 2020

Bug 1845724: Backport deadlock fixes and dependent commits #185

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1845202: Fix deadlock, fix service lag. #178

Bug 1845202: Fix deadlock, fix service lag. #178

squeed commented Jun 8, 2020 •

edited

Loading

openshift-ci-robot commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

squeed commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

squeed commented Jun 8, 2020

squeed commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

squeed commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

openshift-ci-robot commented Jun 9, 2020

openshift-ci-robot commented Jun 9, 2020

squeed commented Jun 9, 2020

squeed commented Jun 9, 2020

pecameron left a comment

openshift-ci-robot commented Jun 9, 2020

squeed commented Jun 9, 2020

openshift-bot commented Jun 9, 2020

squeed commented Jun 9, 2020

squeed commented Jun 9, 2020

squeed commented Jun 9, 2020

openshift-ci-robot commented Jun 9, 2020

squeed commented Jun 9, 2020

openshift-ci-robot commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

abhat commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

stevekuznetsov commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

Bug 1845202: Fix deadlock, fix service lag. #178

Bug 1845202: Fix deadlock, fix service lag. #178

Conversation

squeed commented Jun 8, 2020 • edited Loading

openshift-ci-robot commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

squeed commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

squeed commented Jun 8, 2020

squeed commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

squeed commented Jun 8, 2020

openshift-ci-robot commented Jun 8, 2020

openshift-ci-robot commented Jun 9, 2020

openshift-ci-robot commented Jun 9, 2020

squeed commented Jun 9, 2020

squeed commented Jun 9, 2020

pecameron left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Jun 9, 2020

squeed commented Jun 9, 2020

openshift-bot commented Jun 9, 2020

squeed commented Jun 9, 2020

squeed commented Jun 9, 2020

squeed commented Jun 9, 2020

openshift-ci-robot commented Jun 9, 2020

squeed commented Jun 9, 2020

openshift-ci-robot commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

abhat commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

stevekuznetsov commented Jun 9, 2020

openshift-cherrypick-robot commented Jun 9, 2020

squeed commented Jun 8, 2020 •

edited

Loading