Force to invoke all CNI plugin's delete at pods' tearing down

[s1061123]

In case of multiple interfaces in pod, when the pod is deleted,
forEachnetwork() is called with multiple network attachments.
If forEachnetwork() causes the error at the middle of processing networks,
then forEachnetwork() just returns and following network is not processed.
From CNI runtime point of view, all CNI plugin should be invoked to delete
interfaces.

This change introduce 'force' option in forEachnetwork() and try to
continue to process (i.e. delete network) even though forEachnetwork()
causes the error.

[openshift-ci-robot]

@s1061123: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: s1061123
To complete the pull request process, please assign rajatchopra after the PR has been reviewed.
You can assign the PR to them by writing /assign @rajatchopra in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[s1061123]

/assign @rajatchopra

[dcbw]

@s1061123 what's the error we're getting from forEachNetwork()?

[s1061123]

@dcbw fmt.Errorf("network %q requested interface name %q already assigned", req.Name, req.Ifname)
https://github.com/cri-o/ocicni/pull/86/files#diff-bc84663df95b74e63e2857796dcbf5c05eb68f3ee39de405f636cf42923b9e86L459-L460

[s1061123]

@dcbw https://github.com/cri-o/ocicni/blob/master/pkg/ocicni/ocicni.go#L455-L464 here, forEachNetwork() looks all network and interface name, then check allIfNames[<ifname>] = true. If network's interface is dupped then allIfNames[<ifname>] is true.

Then current code just returns, as return fmt.Errorf("network %q requested interface name %q already assigned", req.Name, req.Ifname), even though remained interface is waiting for deletion. That is the issue.

[dcbw]

@s1061123 what I"m trying to think of is why there would be dupes? There should only be a single cache file per network for the pod, each with a different interface name...

[dcbw]

Is it the case that the multus network (that CRIO calls directly) has the same ifname as the default CNI network that is called by multus? And during teardown, ocicni finds both files even though it was only ever called for the Multus network?

If so, that's the detail I wasn't grasping this morning on the call.

[s1061123]

@dcbw right. So the case I met is following process:

1. ocicni invokes multus-cni -> generate a cache (from multus-cni results, multus-net-name + eth0)
2. multus-cni invokes delegate plugin (e.g. flannel) -> generate a cache (from flannel results, flannel-net-name + eth0)
3. multus-cni invokes net-attach-def plugin (e.g. macvlan) -> generate a cache (from macvlan results, net-attach-def-name + net1)

[dcbw]

@s1061123 I do think the "cleanest" solution to this is to have multus give ocicni a different cache dir so that we can cleanly separate the kubelet -> Multus calls from the Mutlus -> delegate calls.

However, we could just ignore the interface conflicts on DEL since DEL is supposed to be permissive anyway. Would that work for you?

[s1061123]

@dcbw For multus case, I've already taken care of it in k8snetworkplumbingwg/multus-cni#638 So ocicni with multus does not have such issue so far.

This PR is for non multus-cni case. Currently, as far as I know, ocicni seems to support multiple CNI invocation (as type cniNetworkPlugin's networks are map[string]*cniNetwork). So ocicni may generates one or more CNI caches without multus cni.

CNI runtime should invoke CNI plugin's DEL command even if some error is happen, to prevent resource leak. However, ocicni may not invoke plugin's DEL command and may causes resource leak, if

• cniNetworkPlugin has multiple CNI networks and
• forEachNetwork() returned with error.

forEachNetwork() may be returned with error if one of following cases:

• CNI cache file has inconsistent

  ocicni/pkg/ocicni/ocicni.go

  Line 460 in ba4cf0c

  return fmt.Errorf("network %q requested interface name %q already assigned", req.Name, req.Ifname)

• one of the CNI plugin causes the error at DEL command

  ocicni/pkg/ocicni/ocicni.go

  Line 515 in ba4cf0c

  if err := actionFn(cniNet, podNetwork, rt); err != nil {

That's why I filed this PR.

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.