Old network policy iptables rules are not cleared on upgrade

[ShylajaDevadiga]

Environmental Info:
K3s Version:
k3s version v1.25.8+k3s1 (6c5ac02)

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 20.04

Cluster Configuration:
Single node

Describe the bug:
After upgrade we have a mix of old and new iptables that addresses KUBE-ROUTER

Steps To Reproduce:

1. Install k3s with default config
   curl -fL https://get.k3s.io| INSTALL_K3S_VERSION=v1.25.8+k3s1 sh -s - server
2. Upgrade to commit id 027cc18

Expected behavior:
On v1.25.8+k3s1:

$ k3s -v
k3s version v1.25.8+k3s1 (6c5ac022)

$ sudo iptables-save |grep network |grep ROUTER
-A KUBE-ROUTER-FORWARD -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-INPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-OUTPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN

After upgrade

$ sudo iptables-save |grep network |grep ROUTER
-A INPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A FORWARD -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A OUTPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT

Actual behavior:
On v1.25.8+k3s1:

$ k3s -v
k3s version v1.25.8+k3s1 (6c5ac022)

$ sudo iptables-save |grep network |grep ROUTER
-A KUBE-ROUTER-FORWARD -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-INPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-OUTPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN

After upgrade we have both old and new rules

$ sudo iptables-save |grep network |grep ROUTER
-A INPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A FORWARD -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A OUTPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A KUBE-ROUTER-FORWARD -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-INPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-OUTPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN

Workaround:
Run k3s-killall.sh after disabling the network policy controller or during upgrade

Related issue:
#7203

[brandond]

This is similar to #7244 - it looks like the K3s NPC setup needs to do some cleanup of old rules regardless of whether its enabled or not.

[est-suse]

Validated on 59e573d

k3s version v1.25.8+k3s-59e573d1 (59e573d1)
go version go1.19.7

NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Steps To Reproduce:

Install k3s with default config
curl -fL https://get.k3s.io| INSTALL_K3S_VERSION=v1.25.8+k3s1 sh -s - server
Upgrade to commit id 027cc18

Before the upgrade:

sudo iptables-save |grep network |grep ROUTER
-A KUBE-ROUTER-FORWARD -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-INPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN
-A KUBE-ROUTER-OUTPUT -m comment --comment "rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j RETURN

After the upgrade:

ubuntu@ip-172-31-40-237:~$ kubectl get nodes
NAME               STATUS   ROLES                  AGE     VERSION
ip-172-31-40-237   Ready    control-plane,master   6m14s   v1.25.8+k3s-59e573d1
ubuntu@ip-172-31-40-237:~$ sudo iptables-save |grep network |grep ROUTER
-A INPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A FORWARD -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A OUTPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT

Validated on 1.26.3 d9f40d4

-A INPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A FORWARD -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
-A OUTPUT -m comment --comment "KUBE-ROUTER rule to explicitly ACCEPT traffic that comply to network policies" -m mark --mark 0x20000/0x20000 -j ACCEPT
ubuntu@ip-172-31-40-237:~$ kubectl get nodes
NAME               STATUS   ROLES                  AGE     VERSION
ip-172-31-40-237   Ready    control-plane,master   7m12s   v1.26.3+k3s-d9f40d4f