[Release-1.27] - LoadBalancers crashing after updating from 1.27.3 to 1.29.3

[brandond]

Backport fix for LoadBalancers crashing after updating from 1.27.3 to 1.29.3

• LoadBalancers crashing after updating from 1.27.3 to 1.29.3 #9949

[fmoral2]

Validated on Version:

-$  k3s version v1.27.14+k3s-5f656d12 (5f656d12)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu
AMD

Cluster Configuration:
-1 nodes

Steps to validate the fix

1. start k3s with --node-ip=1pv4,1pv6
2. Patch service traefik to -p '{"spec": {"externalTrafficPolicy": "Local"}}'
3. Validate nodes, pods,serviceLB

Reproduction Issue:

 k3s version v1.30.1+k3s-5cf4d757 (5cf4d757)

 kubectl patch service traefik  -n kube-system -p '{"spec": {"externalTrafficPolicy": "Local"}}'
 service/traefik patched

 kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik

NAME                           READY   STATUS             RESTARTS      AGE
svclb-traefik-a436cc37-mwskr   0/2     CrashLoopBackOff   2 (13s ago)   15s



~$ kubectl  get pods -A
NAMESPACE     NAME                                      READY   STATUS             RESTARTS       AGE
kube-system   local-path-provisioner-75bb9ff978-bf7fw   1/1     Running            0              4m59s
kube-system   coredns-576bfc4dc7-h8nlj                  1/1     Running            0              4m59s
kube-system   helm-install-traefik-crd-x84bt            0/1     Completed          0              4m59s
kube-system   helm-install-traefik-4bbh4                0/1     Completed          1              4m59s
kube-system   metrics-server-557ff575fb-sfwzq           1/1     Running            0              4m59s
kube-system   traefik-5fb479b77-8nmm8                   1/1     Running            0              4m32s
kube-system   svclb-traefik-a436cc37-mwskr              0/2     CrashLoopBackOff   10 (12s ago)   2m56s

Validation Results:

 
 $ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   coredns-77ccd57875-zt2qm                 1/1     Running   0          15s
kube-system   helm-install-traefik-crd-th465           1/1     Running   0          16s
kube-system   helm-install-traefik-hv82q               1/1     Running   0          16s
kube-system   local-path-provisioner-5c6c7f7cd-lmpsd   1/1     Running   0          15s
kube-system   metrics-server-c44988498-2rhcr           0/1     Running   0          15s



ubuntu@ip-:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
No resources found in kube-system namespace.


ubuntu@ip-:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS              RESTARTS   AGE
svclb-traefik-cabd2baa-cflzt   0/2     ContainerCreating   0          2s


ubuntu@ip-:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS    RESTARTS   AGE
svclb-traefik-cabd2baa-cflzt   2/2     Running   0          6s


ubuntu@ip-:~$ kubectl patch service traefik  -n kube-system -p '{"spec": {"externalTrafficPolicy": "Local"}}'
service/traefik patched


ubuntu@ip-:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS        RESTARTS   AGE
svclb-traefik-cabd2baa-cflzt   2/2     Terminating   0          15s


ubuntu@ip-:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS    RESTARTS   AGE
svclb-traefik-cabd2baa-s54mv   2/2     Running   0          2s


ubuntu@ip-:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS    RESTARTS   AGE
svclb-traefik-cabd2baa-s54mv   2/2     Running   0          4s


ubuntu@ip-:~$ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-77ccd57875-zt2qm                 1/1     Running     0          53s
kube-system   helm-install-traefik-crd-th465           0/1     Completed   0          54s
kube-system   helm-install-traefik-hv82q               0/1     Completed   1          54s
kube-system   local-path-provisioner-5c6c7f7cd-lmpsd   1/1     Running     0          53s
kube-system   metrics-server-c44988498-2rhcr           1/1     Running     0          53s
kube-system   svclb-traefik-cabd2baa-s54mv             2/2     Running     0          13s
kube-system   traefik-7d5c94d587-wlcld                 1/1     Running     0          29s