Ingress not working outside of host on cluster upgrade

[kinarashah]

Environmental Info:
K3s Version: Upgraded from v1.25.9+k3s1 to v1.26.4+k3s1

k3s version v1.26.4+k3s1 (8d0255af)
go version go1.19.8

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-7-19 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24 01:58:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 1 node

Describe the bug:

Steps To Reproduce:

config.yaml

write-kubeconfig-mode: 644
cluster-init: true
node-external-ip: xx.xxx.xx.xxx 

Steps:

1. Install k3s: curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.25.9+k3s1 sh -
2. Deploy ingress: kubectl apply -f ing.yaml (Attached ing.yaml below)
3. query ingress FROM OUTSIDE OF THE VM: curl -H 'Host: [test2.com](http://test2.com/)' http://xx.xxx.xx.xxx/name.html . Expected result like ingresstest-deploy-abcde
4. Upgrade k3s: curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.26.4+k3s1 sh -
5. Repeat step 3. Expect same result, but actually get Failed to connect: port 80 after 46 ms: Couldn't connect to server

Expected behavior:
curl ingress should continue to work outside of host after upgrade

Actual behavior:
curl fails with Failed to connect xx.xx.xx.xx port 80 after 46 ms: Couldn't connect to server

Additional context / logs:
logs.zip

[rancher-max]

Couple of additional notes:

1. This appears to happen on all upgrades. I confirmed it happens on upgrade from v1.26.4 to v1.27.1, from v1.27.1 to v1.27.2-rc1, from v1.25.6 to v1.25.7, and from v1.24.8 to v1.24.9+k3s2.
2. This does NOT happen on upgrade from v1.24.7 to v1.24.8, so I believe this broke sometime in the January 2023 release timeframe.
3. The workaround is to restart the svclb-traefik pod
4. Setting disable-network-policy does not fix this
5. Setting different values for egress-selector-mode does not fix this
6. This seems to only happen when node-external-ip is set

[brandond]

I'm curious if reverting #7210 fixes this.

[rbrtbnfgl]

Could it be related to #7561
On that issue the ingress controller is not traefik but it could be somehow related.

[brandond]

Unlike rules managed by the kubelet/kube-router/flannel, CNI plugin rules are only created when the pod starts. If we remove them in the install script, networking to existing pods (such portmap rules for nodePort pods) will be broken until the pods are deleted and recreated. We should not clean CNI rules in the install script.

Root cause on this is the install script change in #7274

[rancher-max]

This is working following the same steps and upgrading to the same versions. This is also working with system-upgrade-controller upgrades (as a regression test).