Removing MicroK8s sometimes leaves iptables rules behind

[knkski]

I had a MicroK8s instance that was running some services on port 80. After running sudo snap remove microk8s --purge to remove it, I was unable to have anything else running on port 80. As an example, I would run this:

$ sudo python -m http.server 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...

Which would result in this:

curl -v http://localhost:80/
*   Trying 127.0.0.1:80...
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to localhost port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 80: Connection refused

After poking around a bit, I found that there were some iptables rules left behind, even though MicroK8s had been uninstalled:

$ sudo iptables-legacy-save | grep k8s
    -A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"k8s-pod-network\" id: \"ea1eb612eac9d554d5d4bd81aa92bc7dfe3f2bd5cb71a27c54b480e7906f4caa\"" -m multiport --dports 80,443 -j CNI-DN-75cbb066bb1e01246e284
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"k8s-pod-network\" id: \"39b7985bb76e28765f8c57569a6c487cca0c50edf8dfd4ec228d50f7c27a4ec6\"" -m multiport --dports 80,443 -j CNI-DN-c55474bc55530749bbd04
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"k8s-pod-network\" id: \"df4ba89a451649be1cd20224decf041f58ce80c0886af5e70440c3ee9be24df5\"" -m multiport --dports 80,443 -j CNI-DN-01bb7e67bd4e1e4f1f3b0

I then ran these commands to remove the rules:

$ sudo iptables-legacy -t nat -D CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"k8s-pod-network\" id: \"df4ba89a451649be1cd20224decf041f58ce80c0886af5e70440c3ee9be24df5\"" -m multiport --dports 80,443 -j CNI-DN-01bb7e67bd4e1e4f1f3b0
$ sudo iptables-legacy -t nat -D CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"k8s-pod-network\" id: \"39b7985bb76e28765f8c57569a6c487cca0c50edf8dfd4ec228d50f7c27a4ec6\"" -m multiport --dports 80,443 -j CNI-DN-c55474bc55530749bbd04
$ sudo iptables-legacy -t nat -D CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"k8s-pod-network\" id: \"ea1eb612eac9d554d5d4bd81aa92bc7dfe3f2bd5cb71a27c54b480e7906f4caa\"" -m multiport --dports 80,443 -j CNI-DN-75cbb066bb1e01246e284

And running sudo python -m http.server 80 started working again. This only happens sometimes, and I haven't found a cause for it. I did find this bit of documentation that says

There is a top-level chain, CNI-HOSTPORT-DNAT which is always created and never deleted.

Maybe this chain should be explicitly deleted on snap removal?

[ktsakalozos]

Thank you for reporting this @knkski and for diving into this issue. We could try to update the remove [1] hook to include your recommendation. I hope I will have news for you soon.

[1] https://github.com/canonical/microk8s/blob/master/snap/hooks/remove

[ktsakalozos]

Hi @knkski, we just merged a fix on our removal hook. It should be available on latest/edge within the day. If you have time it would be great if you told us if it works for you. Thanks.

[csegarragonz]

Hi, this is still a problem for me in 1.24/stable. Deploying an ingress resource, removing it, removing microk8s, installing it again, and deploying the same resource does not work.

Grepping the ip tables shows plenty rules being left over still after removing the snap.

[strazto]

This is a problem for me with microk8s 1.28/stable,

Post remove, if I diff

sudo iptables -S > iptables_microk8s_pre_remove

sudo snap remove microk8s --purge

sudo iptables -S > iptables_microk8s_post_remove

diff  iptables_microk8s*

I get no diff

[strazto]

actually after stopping and purging, the iptables config file itself was intact, however the rules weren't applied, a reboot fixed it.

For context, I got here from

Apparently I had iptables rules blocking loopback; fixing those fixed this issue.

Originally posted by @HMPerson1 in microsoft/vscode-remote-release#1870 (comment)

I believe microk8s iptables rules were breaking vscode-server