Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release-4.9] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races #955

Merged

Conversation

kyrtapz
Copy link
Contributor

@kyrtapz kyrtapz commented Feb 14, 2022

deleteEgressNode takes care of reassigning egress IPs that are hosted on that node when it gets deleted, becomes unreachable, the label k8s.ovn.org/egress-assignable gets removed.

It is possible that when deleteEgressNode takes a long time to reassign the egress ips another one is called(e.g node becomes unreachable and then NotReady).
Best case scenario the egressIP is reassigned twice, worst case the status doesn't get updated properly or the egress IP is not correctly assigned.

I do not think that this change brings any deadlock potential as this approach is already used in addEgressNode, nevertheless it would be good to focus on that while reviewing the change.

I was able to reproduce the bug and verify the fix by creating 400 pods that match the egress IP assigned to node and then run:

# make the node offline for egressip
oc debug node/$node -- chroot host iptables -A INPUT -i ovn-k8s-mp0 -p tcp --destination-port 9 -j DROP
#wait for egress IP to start reassigning
sleep 5s
#remove the node from egress IP pool
oc label nodes $node k8s.ovn.org/egress-assignable-

Signed-off-by: Patryk Diak pdiak@redhat.com

@openshift-ci openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. labels Feb 14, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 14, 2022

@kyrtapz: This pull request references Bugzilla bug 2053611, which is invalid:

  • expected the bug to target the "4.9.z" release, but it targets "4.11.0" instead
  • expected Bugzilla bug 2053611 to depend on a bug targeting a release in 4.10.0 and in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

[WIP] Bug 2053611: Lock the reassignment procedure during node deletion to avoid races

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 14, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 14, 2022

@kyrtapz: This pull request references Bugzilla bug 2053611, which is invalid:

  • expected the bug to target the "4.9.z" release, but it targets "4.11.0" instead
  • expected Bugzilla bug 2053611 to depend on a bug targeting a release in 4.10.0 and in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

[WIP] Bug 2053611: Lock the reassignment procedure during node deletion to avoid races

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@flavio-fernandes
Copy link
Contributor

/retitle [WIP] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races

@openshift-ci openshift-ci bot changed the title [WIP] Bug 2053611: Lock the reassignment procedure during node deletion to avoid races [WIP] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races Feb 14, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 14, 2022

@kyrtapz: This pull request references Bugzilla bug 2054299, which is invalid:

  • expected dependent Bugzilla bug 2053611 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is NEW instead
  • expected dependent Bugzilla bug 2053611 to target a release in 4.10.0, but it targets "4.8.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

[WIP] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@flavio-fernandes
Copy link
Contributor

/lgtm
Thanks @kyrtapz !!!

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 14, 2022
@flavio-fernandes
Copy link
Contributor

/bugzilla refresh

@openshift-ci openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Feb 14, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 14, 2022

@flavio-fernandes: This pull request references Bugzilla bug 2054299, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2054304 is in the state CLOSED (CURRENTRELEASE), which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2054304 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 14, 2022

@kyrtapz: This pull request references Bugzilla bug 2054299, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2054304 is in the state CLOSED (CURRENTRELEASE), which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2054304 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

[WIP] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

1 similar comment
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 14, 2022

@kyrtapz: This pull request references Bugzilla bug 2054299, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2054304 is in the state CLOSED (CURRENTRELEASE), which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2054304 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

[WIP] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kyrtapz
Copy link
Contributor Author

kyrtapz commented Feb 14, 2022

/retitle Bug 2054299: Lock the reassignment procedure during node deletion to avoid races

@openshift-ci openshift-ci bot changed the title [WIP] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races Bug 2054299: Lock the reassignment procedure during node deletion to avoid races Feb 14, 2022
@openshift-ci openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 14, 2022
@kyrtapz
Copy link
Contributor Author

kyrtapz commented Feb 14, 2022

/assign @trozet

@flavio-fernandes
Copy link
Contributor

/unhold

@flavio-fernandes
Copy link
Contributor

/remove-work-in-progress

@flavio-fernandes
Copy link
Contributor

/shrug

@openshift-ci openshift-ci bot added the ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯ label Feb 14, 2022
@tssurya
Copy link
Contributor

tssurya commented Feb 14, 2022

/shrug cancel

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

12 similar comments
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@huiran0826
Copy link

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Feb 24, 2022
@huiran0826
Copy link

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Feb 24, 2022
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

6 similar comments
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 24, 2022

@kyrtapz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-windows 3c4b792 link false /test e2e-vsphere-windows
ci/prow/okd-e2e-gcp-ovn 3c4b792 link false /test okd-e2e-gcp-ovn
ci/prow/e2e-metal-ipi-ovn-ipv6 3c4b792 link false /test e2e-metal-ipi-ovn-ipv6
ci/prow/4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade 3c4b792 link false /test 4.9-upgrade-from-stable-4.8-e2e-aws-ovn-upgrade
ci/prow/e2e-openstack-ovn 3c4b792 link false /test e2e-openstack-ovn
ci/prow/e2e-vsphere-ovn 3c4b792 link false /test e2e-vsphere-ovn
ci/prow/e2e-ovn-hybrid-step-registry 3c4b792 link false /test e2e-ovn-hybrid-step-registry
ci/prow/e2e-azure-ovn 3c4b792 link false /test e2e-azure-ovn
ci/prow/e2e-aws-ovn-shared-to-local-gateway-mode-migration 3c4b792 link false /test e2e-aws-ovn-shared-to-local-gateway-mode-migration

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 8f71453 into openshift:release-4.9 Feb 24, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 24, 2022

@kyrtapz: All pull requests linked via external trackers have merged:

Bugzilla bug 2054299 has been moved to the MODIFIED state.

In response to this:

[release-4.9] Bug 2054299: Lock the reassignment procedure during node deletion to avoid races

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kyrtapz
Copy link
Contributor Author

kyrtapz commented Feb 24, 2022

/cherry-pick release-4.8

@openshift-cherrypick-robot

@kyrtapz: new pull request created: #970

In response to this:

/cherry-pick release-4.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants