Explicitly fail when we are asked to add a duplicate ip to the ipset

[bleggett]

A user is (somehow) able to produce a scenario where, without istio-cni restarting, a pod IP goes missing from the host node ipset, causing an active pod to begin failing host node healthchecks.

Since we destructively mutate the ipset contents in exactly 3 spots:

1. Flush on istio-cni startup
2. Flush/destroy on istio-cni shutdown
3. Removing pod IPs on K8S pod unlabel/delete events

and no istio-cni restart happens as per logs, that leaves (3) as the only way this could manifest, barring outside interference with the contents of the ipset.

The only scenario where I can imagine this happening is if

• two pods reuse the same IP relatively close to each other on the same node.
• K8S pod add and pod remove events for those two pods both overlap, and come in to us out-of-order - we add the new pod with the reused IP, and then somehow later get a remove event for the old pod with the original use of the IP, leading to the set being out of sync.

This PR changes the addPodIPToIpset logic, so that if we try to add a pod IP that is already there, we log/return an error - previously, we would silently overwrite the existing entry.

This will at least help us see if the above happens and cause the pod with the overlapping IP to fail to start - otherwise we should never legitimately have an overwrite.

(We might have missed this in testing months back because we were using ip:port maps, and the odds of two pods using the same IP AND the same ports is even tinier)

[hzxuzhonghu]

Can the node level interception be deprecated now

[bleggett]

Can the node level interception be deprecated now

What do you mean, exactly?

We don't do node level interception at all, but we still do need to do node level SNAT for kubelet healthchecks.

This doesn't change that, this just works around a bug that could exist there.

[hzxuzhonghu]

OK, NVM, I thought is was intercetion on node

[hzxuzhonghu]

we still do need to do node level SNAT for kubelet healthchecks.

Why still need this, now if ambient behaves like sidecar interception, we can eliminate host ipset completely

[bleggett]

we still do need to do node level SNAT for kubelet healthchecks.

Why still need this, now if ambient behaves like sidecar interception, we can eliminate host ipset completely

Even for sidecars we must special-case host kubelet probes - with sidecars we just do it with webhook rewrites for the pod manifest.

Ambient doesn't have mutating webhooks, so we do it this way instead.

Sidecar mutates the pod manifest to redirect the health probes, ambient mutates the source IP to redirect the health probes. It's tricky.

[hzxuzhonghu]

Yes, we know the src addrss of kubelete probe, so can do that better than sidecar via bypass this kind of traffic

[bleggett]

Yes, we know the src addrss of kubelete probe, so can do that better than sidecar via bypass this kind of traffic

We don't know the source address of the kubelet probe. The current approach in ambient doesn't need to know it.

Mechanisms to reliably derive the kubelet IP from the perspective of the pod are very CNI dependent.

We don't bother to derive the pod-facing kubelet IP with ambient, we just use iptables to tell us if the packet originated locally on the node, from the host node netns.

Also, not all packets coming from the kubelet IP originate from the local kubelet process - the kubelet source IP is just not a reliable check. Which is why we don't use it.

It's tricky. Asserting policy based on "kubelet src IP" is just an unavoidably unsound approach, no matter how you slice it.

[howardjohn]

So if we hit the error, we fail to the CNI layer which retries and now the ordering should be correct. Right? It will recover?

[bleggett]

Right. Inelegant, but it will work itself out for now (if the hypothesis about the cause is in fact correct)

[istio-testing]

In response to a cherrypick label: #51081 failed to apply on top of branch "release-1.22":

Applying: Explicitly fail when we are asked to add a duplicate ip to the ipset
Using index info to reconstruct a base tree...
M	cni/pkg/nodeagent/net.go
Falling back to patching base and 3-way merge...
Auto-merging cni/pkg/nodeagent/net.go
CONFLICT (content): Merge conflict in cni/pkg/nodeagent/net.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Explicitly fail when we are asked to add a duplicate ip to the ipset
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

[istio-testing]

In response to a cherrypick label: new issue created for failed cherrypick: #51131