[release-4.16] OCPBUGS-51017: Delete CNI configuration file when SDN pod is stopped

[pperiyasamy]

The openshift sdn CNI configuration file /etc/cni/net.d/80-openshift-network.conf is written on the node upon first time of sdn pod deployment and remains to be there on the node during its lifetime.
This makes multus to assume sdn is in ready state though sdn pod is coming up during reboot scenarios. Because of this, for a pod delete request, CNI DELETE request fails with connection refused error when sdn cni server socket is being created, this may end up in entries for the pod may not get cleaned up (example: IP address allocated for the pod by host-local IPAM plugin).
Previously (4.13) this wasn't a problem because multus propagates the error for cni delete to crio/kubelet so that retries for delete happens for a failure, but this is not a case in 4.14 with this multus change: https://github.com/openshift/multus-cni/blob/release-4.14/pkg/server/api/shim.go#L68-L69.
This PR cleans up the cni config file when sdn pod is stopped. So when multus received cni delete request, it waits upto 45s (https://github.com/openshift/multus-cni/blob/release-4.14/pkg/multus/multus.go#L841-L845) until sdn plugin is actually ready and then delegate the request to sdn plugin.

[openshift-ci-robot]

@pperiyasamy: This pull request references Jira Issue OCPBUGS-51017, which is invalid:

• expected the bug to target the "4.16.z" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected Jira Issue OCPBUGS-51017 to depend on a bug targeting a version in 4.17.0, 4.17.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

The openshift sdn CNI configuration file /etc/cni/net.d/80-openshift-network.conf is written on the node upon first time of sdn pod deployment and remains to be there on the node during its lifetime. This makes multus to assume sdn is in ready state though sdn pod is coming up during reboot scenarios. Because of this, for a pod delete request, CNI DELETE request fails with connection refused error when sdn cni server socket is being created, this may end up in entries for the pod may not get cleaned up (example: IP address allocated for the pod by host-local IPAM plugin). Hence this commit cleans up the cni config file when sdn pod is stopped, it makes multus to wait until sdn plugin is actually ready and then invoke CNI requests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[pperiyasamy]

/assign @martinkennelly @dougbtv

[openshift-ci]

@pperiyasamy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/security	2fedca3	link	false	/test security

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[pperiyasamy]

/jira refresh

[openshift-ci-robot]

@pperiyasamy: This pull request references Jira Issue OCPBUGS-51017, which is invalid:

• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-51257 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is Closed (Won't Do) instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[pperiyasamy]

/jira refresh

[openshift-ci-robot]

@pperiyasamy: This pull request references Jira Issue OCPBUGS-51017, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.z) matches configured target version for branch (4.16.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-51257 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-51257 targets the "4.17.z" version, which is one of the valid target versions: 4.17.0, 4.17.z
• bug has dependents

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[pperiyasamy]

/label backport-risk-assessed

[openshift-ci]

@pperiyasamy: Can not set label backport-risk-assessed: Must be member in one of these teams: [openshift-staff-engineers]

In response to this:

/label backport-risk-assessed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@pperiyasamy: This pull request references Jira Issue OCPBUGS-51017, which is valid.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.z) matches configured target version for branch (4.16.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-51257 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-51257 targets the "4.17.z" version, which is one of the valid target versions: 4.17.0, 4.17.z
• bug has dependents

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

The openshift sdn CNI configuration file /etc/cni/net.d/80-openshift-network.conf is written on the node upon first time of sdn pod deployment and remains to be there on the node during its lifetime.
This makes multus to assume sdn is in ready state though sdn pod is coming up during reboot scenarios. Because of this, for a pod delete request, CNI DELETE request fails with connection refused error when sdn cni server socket is being created, this may end up in entries for the pod may not get cleaned up (example: IP address allocated for the pod by host-local IPAM plugin).
Previously (4.13) this wasn't a problem because multus propagates the error for cni delete to crio/kubelet so that retries for delete happens for a failure, but this is not a case in 4.14 with this multus change: https://github.com/openshift/multus-cni/blob/release-4.14/pkg/server/api/shim.go#L68-L69.
This PR cleans up the cni config file when sdn pod is stopped. So when multus received cni delete request, it waits upto 45s (https://github.com/openshift/multus-cni/blob/release-4.14/pkg/multus/multus.go#L841-L845) until sdn plugin is actually ready and then delegate the request to sdn plugin.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[dougbtv]

Really nice approach Peri!

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dougbtv, pperiyasamy
Once this PR has been reviewed and has the lgtm label, please assign danwinship for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment