Bug 2094039: egressIP: node retrieval failure is not respected, causes panic

[tssurya]

Fixes panic seen during checkEgressNodesReachability
E0606 11:57:34.063567 1 egressip.go:2096] Node:
huirwang-0606c-m5w24-worker-2-dkjmc reachability changed, but could not
retrieve node from cache, err: node
"huirwang-0606c-m5w24-worker-2-dkjmc" not found
W0606 11:57:34.063658 1 egressip.go:1657] Unable to remove GARP
configuration on external logical switch port for egress node:
huirwang-0606c-m5w24-worker-2-dkjmc, err: object not found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x176626d]

goroutine 487 [running]:
github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).checkEgressNodesReachability(0xc000171c00)
/home/surya/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/egressip.go:2099
+0x3ed
created by
github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).initClusterEgressPolicies
/home/surya/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/egressip.go:1772
+0xdb

The problem here is that the node retrieval fails since the node has
already been deleted, instead of respecting that failure we just
log the error saying node object not found and actually continue
to use that object further in the code which causes this panic.

Signed-off-by: Surya Seetharaman suryaseetharaman.9@gmail.com
(cherry picked from commit 36176e5)

- What this PR does and why is it needed

- Special notes for reviewers

- How to verify it

- Description for the changelog

[openshift-ci]

@tssurya: This pull request references Bugzilla bug 2094039, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.11.0) matches configured target release for branch (4.11.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @huiran0826

In response to this:

Bug 2094039: egressIP: node retrieval failure is not respected, causes panic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kyrtapz]

/lgtm

[jcaamano]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcaamano, kyrtapz, tssurya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jcaamano]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 2 against base HEAD ab62636 and 8 for PR HEAD f436d25 in total

[openshift-ci-robot]

/retest-required

Remaining retests: 1 against base HEAD ab62636 and 7 for PR HEAD f436d25 in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD ab62636 and 6 for PR HEAD f436d25 in total

[openshift-ci]

@tssurya: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/4.11-upgrade-from-stable-4.10-local-gateway-e2e-aws-ovn-upgrade	f436d25	link	false	/test 4.11-upgrade-from-stable-4.10-local-gateway-e2e-aws-ovn-upgrade
ci/prow/okd-e2e-gcp-ovn	f436d25	link	false	/test okd-e2e-gcp-ovn
ci/prow/e2e-vsphere-ovn	f436d25	link	false	/test e2e-vsphere-ovn
ci/prow/e2e-azure-ovn	f436d25	link	false	/test e2e-azure-ovn
ci/prow/e2e-aws-ovn-windows	f436d25	link	true	/test e2e-aws-ovn-windows
ci/prow/e2e-openstack-ovn	f436d25	link	false	/test e2e-openstack-ovn
ci/prow/e2e-hypershift	f436d25	link	false	/test e2e-hypershift

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci]

@tssurya: All pull requests linked via external trackers have merged:

• openshift/ovn-kubernetes#1130

Bugzilla bug 2094039 has been moved to the MODIFIED state.

In response to this:

Bug 2094039: egressIP: node retrieval failure is not respected, causes panic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tssurya]

/cherry-pick release-4.10

[openshift-cherrypick-robot]

@tssurya: #1130 failed to apply on top of branch "release-4.10":

Applying: egressIP: node retrieval failure is not respected, causes panic
Using index info to reconstruct a base tree...
M	go-controller/pkg/ovn/egressip.go
A	go-controller/pkg/ovn/obj_retry.go
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): go-controller/pkg/ovn/obj_retry.go deleted in HEAD and modified in egressIP: node retrieval failure is not respected, causes panic. Version egressIP: node retrieval failure is not respected, causes panic of go-controller/pkg/ovn/obj_retry.go left in tree.
Auto-merging go-controller/pkg/ovn/egressip.go
CONFLICT (content): Merge conflict in go-controller/pkg/ovn/egressip.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 egressIP: node retrieval failure is not respected, causes panic
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.