[release 3.11]Bug 1824243: Fix egressVXLANMonitor and egressIPTracker deadlock #25027
Conversation
The egressIPTracker has methods that lock eit.mutex and that call evm functions that lock evm.mutex. The problem with this is that evm.mutex has to write to the evm.updates channel which isn't buffered and becomes blocked until eit.setNodeOffline, which also locks eit.mutex, is running. This causes a deadlock. Initially I tried doing a horrible hack by making the updates channel huge, however it wasn't enough. Instead this fix adds a shared nodes list between eit and evm and the evm uses the updates channel just to notify the eit that there are updates in the shared list.
openshift-ci-robot
added
bugzilla/severity-urgent
|
@juanluisvaladas: This pull request references Bugzilla bug 1824243, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/hold |
openshift-ci-robot
added
the
do-not-merge/hold
juanluisvaladas
changed the title
|
/retest |
|
/test e2e-gcp |
|
/approve |
knobunc
added
cherry-pick-approved
openshift-ci-robot
added
the
approved
|
/hold cancel |
openshift-ci-robot
removed
the
do-not-merge/hold
|
/test e2e-gcp |
|
e2e-gcp seems to be haven been down for a few days so I'm not retrying it for a while |
|
/test e2e-gcp |
|
/retest |
1 similar comment
|
/retest |
|
@knobunc could you please skip e2e-gcp tests ? they are broken. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: juanluisvaladas, knobunc The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@knobunc: Overrode contexts on behalf of knobunc: ci/prow/e2e-gcp In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
24 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/override ci/openshift-jenkins/extended_conformance_install This is failing due to a mirror problem. We had a clean test run prior to that. I'll work on working out the test problem separately. |
|
@knobunc: Overrode contexts on behalf of knobunc: ci/openshift-jenkins/extended_conformance_install In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@juanluisvaladas: All pull requests linked via external trackers have merged: openshift/origin#25027. Bugzilla bug 1824243 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
The egressIPTracker has methods that lock eit.mutex and that call
evm functions that lock evm.mutex.
The problem with this is that evm.mutex has to write to the evm.updates
channel which isn't buffered and becomes blocked until
eit.setNodeOffline, which also locks eit.mutex, is running.
This causes a deadlock. Initially I tried doing a horrible hack by
making the updates channel huge, however it wasn't enough.
Instead this fix adds a shared nodes list between eit and evm and the
evm uses the updates channel just to notify the eit that there are
updates in the shared list.