Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OCPBUGS-5077: [release-4.10] Fix load balancer health check locking and update #1463

Merged
merged 2 commits into from
Jan 18, 2023
Merged

OCPBUGS-5077: [release-4.10] Fix load balancer health check locking and update #1463

merged 2 commits into from
Jan 18, 2023

Conversation

trozet
Copy link
Contributor

@trozet trozet commented Dec 21, 2022

Clean backport of fixes that fix locking around healthchecks and handle updates correctly.

trozet and others added 2 commits December 21, 2022 10:08
@trozet
Fixes thread safety with LB healthcheck
02188f9 
Maps for endpoints and services may be accessed at the same time because
the services and endpoints handlers are attached to two different
informer goroutines. This can cause:

fatal error: concurrent map read and map write

goroutine 322 [running]:
 runtime.throw(0x1b2bac6, 0x21)
 /usr/lib/golang/src/runtime/panic.go:1117 +0x72 fp=0xc000811b60 sp=0xc000811b30 pc=0x43e612
 runtime.mapaccess2(0x18e4000, 0xc0018c5e90, 0xc000811bf0, 0x2921980, 0x0)
 /usr/lib/golang/src/runtime/map.go:469 +0x255 fp=0xc000811ba0 sp=0xc000811b60 pc=0x415c55

 github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node.(*loadBalancerHealthChecker).AddEndpoints(0xc0018c5ef0, 0xc0004d9180)
 /go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/node/healthcheck.go:
  github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node.(*gateway).AddEndpoints(...)
 /go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/node/gateway.go:107

Signed-off-by: Tim Rozet <trozet@redhat.com>
(cherry picked from commit 915c779)
@tssurya @trozet
Fix UpdateService healthport checks
3f2849a 
We were not creating healthchecks for service updates
when the etp changed from local to cluster or back. This
PR fixes this. It also fixes some endpoint slices code
that earlier assumed svc:endpointslice was 1:1 mapping.

Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>
(cherry picked from commit 4c7add6)
(cherry picked from commit 5464286)
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 21, 2022

@trozet: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

OCPBUGS-5077: [release-4.10] Fix load balancer health check locking and update

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added jira/severity-low Referenced Jira bug's severity is low for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Dec 21, 2022
@openshift-ci-robot
Copy link
Contributor

@trozet: This pull request references Jira Issue OCPBUGS-5077, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.10.z) matches configured target version for branch (4.10.z)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
  • dependent bug Jira Issue OCPBUGS-2554 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE))
  • dependent Jira Issue OCPBUGS-2554 targets the "4.11.z" version, which is one of the valid target versions: 4.11.0, 4.11.z
  • bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Clean backport of fixes that fix locking around healthchecks and handle updates correctly.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@trozet
Copy link
Contributor Author

trozet commented Dec 21, 2022

/assign @tssurya

@openshift-ci openshift-ci bot requested review from abhat and tssurya December 21, 2022 15:26
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 21, 2022
@abhat
Copy link
Contributor

abhat commented Dec 21, 2022

/override ci/prow/unit
tracked by OCPBUGS-5040.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 21, 2022

@abhat: Overrode contexts on behalf of abhat: ci/prow/unit

In response to this:

/override ci/prow/unit
tracked by OCPBUGS-5040.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dcbw
Copy link
Contributor

dcbw commented Dec 23, 2022

/lgtm

@dcbw dcbw added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Dec 23, 2022
@dcbw
Copy link
Contributor

dcbw commented Dec 23, 2022

/retest-required

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 23, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 23, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@anuragthehatter
Copy link

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jan 17, 2023
@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD 46b044c and 2 for PR HEAD 3f2849a in total

@npinaeva npinaeva mentioned this pull request Jan 17, 2023
Merged
@npinaeva
Copy link
Member

/retest-required

@npinaeva
Copy link
Member

depends on #1479 for unit test fix

@trozet
Copy link
Contributor Author

trozet commented Jan 18, 2023

/retest-required

@trozet
Copy link
Contributor Author

trozet commented Jan 18, 2023

/override ci/prow/unit

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 18, 2023

@trozet: Overrode contexts on behalf of trozet: ci/prow/unit

In response to this:

/override ci/prow/unit

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 18, 2023

@trozet: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vsphere-windows 3f2849a link false /test e2e-vsphere-windows
ci/prow/unit 3f2849a link true /test unit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 8828e5e into openshift:release-4.10 Jan 18, 2023
@openshift-ci-robot
Copy link
Contributor

@trozet: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-5077 has been moved to the MODIFIED state.

In response to this:

Clean backport of fixes that fix locking around healthchecks and handle updates correctly.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anuragthehatter anuragthehatter Awaiting requested review from anuragthehatter

@abhat abhat Awaiting requested review from abhat

@tssurya tssurya Awaiting requested review from tssurya

Assignees

@mffiedler mffiedler

@rbbratta rbbratta

@tssurya tssurya

@weliang1 weliang1

@anuragthehatter anuragthehatter

@huiran0826 huiran0826

@asood-rh asood-rh

@jechen0648 jechen0648

@yingwang-0320 yingwang-0320

@qiowang721 qiowang721

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/severity-low Referenced Jira bug's severity is low for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.