NPM does not remove succeeded pod ips from firewall ipsets

[finlob]

Is this a request for help?:

Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):

ISSUE

Which release version?:

1.0.28

Which component (CNI/IPAM/CNM/CNS):

NPM

Which Operating System (Linux/Windows):

Linux

For Linux: Include Distro and kernel version using "uname -a"

Linux aks-agentpool-34239724-1 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Which Orchestrator and version (e.g. Kubernetes, Docker)

Kubernetes

What happened:

Pods that go to a completed state before being deleted are not removed from the relevant ipset.

This eventually causes incorrect network policy/firewall rules to be applied when an unrelated pod happens to re-use the ipaddress of the completed pod.

What you expected to happen:

A pod that goes to a completed state should be removed from all of the ipsets that it belongs to, similar to when it is deleted.

How to reproduce it (as minimally and precisely as possible):

Create a Job and delete some time after completion. e.g.

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-world
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-e", "print 'hello world...'; sleep 30; print 'goodbye world';"]
      restartPolicy: Never

Observe the logs of one of the azure-npm pods and note that the pod creation is logged but no deletion event is ever recorded. As you might expect, this is also reflected in the underlying ipsets used by the firewall rules.

Anything else we need to know:

I think the problem is in npm/pod.go where the UpdatePod function should handle the pod going into a Succeeded or Failed state. Currently any pod in those states is ignored.

[jaer-tsun]

This should've addressed the issue: #435
The change is already in v1.0.29

[finlob]

I've looked at the code change and suspect that there remains a smaller issue.

If you use the reproduction steps above I suspect that you will be able to see that the ip address remains in the relevant ipsets while the pod is succesful and complete but not yet deleted. During this period other pods can re-use the same ip address and be affected by network policies that were for the completed pod.

It will probably be a tricky bug to track down after this change is in place as it is likely that problems will be temporary and have a tendency to resolve themselves depending on the policies in place for deleting completed pods.

Is it possible to confirm that the pod ip is removed from the relevant ipsets on transition to complete?
It should be possible to see the transition in the azure-npm logs.

[jaer-tsun]

The pod is removed from ipsets on delete from v1.0.29+

Can you re-clarify the issue regarding complete state?

[finlob]

The issue regarding complete state is that there is a state before the pod is deleted where they should be removed from the ipset.

To see the issue run the hello-world job but do not delete the pod or job.

You should see the pod go through the normal lifecycle and at the end if you look at the list of pods (kubectl get pods) you should see something like:
hello-world-dc5mp 0/1 Completed 0 4m54s

While the pod is in this state I do not believe that it will have been removed from the ipset. However other pods can run and re-use the same ipaddress as the completed pod. This means that there is a scenario where pods may sometimes run with incorrect firewall rules.

It is possible to work around the problem by ensuring that pods are deleted after completion but you need to know to arrange for that to occur.

[jaer-tsun]

So we currently check if the pod IP exists before deleting. In this case it seems like the IP is still allocated to the pod.

[finlob]

It may seem that the IP is still allocated but it is not - the IP can be used by new pods.

This behaviour can be confirmed by running lots of separate jobs and leaving them in a completed state. e.g. I ran 500 hello-world pods and then sorted by IP

[steffen-geissinger-by]

Is there any update on this issues? We just faced an AKS cluster with "randomly filled" ipsets. At the end we could reproduce connection issues from one Pod (A) to core-dns with a NetPol which targeted a totally different Pod (B). When looking at the ipsets they looked very strange, but we also don't know how they got into this shape, since the cluster was running for several days and we created/deleted K8s Objects all the time. It seems there's some issue with properly maintaining the ipsets, which leads to "random" network issues in the cluster.

We are unsure if this issue was the root cause or not, but we have Jobs running in our cluster so it could have had an influence. We are currently running NPM v1.0.33

[jaer-tsun]

@steffen-geissinger-jdas , I've just created a PR for this.

If you're constantly cleaning up your pods then this issue shouldn't affect you.

[steffen-geissinger-by]

Thanks for fixing it. We cleaned up all the Pods before looking into the iptable rules and rulesets. Probably we are facing another issue, since the issue persisted after cleaning up. We still have the cluster, but removed all the netpols for the moment. Can we provide any additional information to track down the issue?