LoadBalancers crashing after updating from 1.27.3 to 1.29.3 #9949

Ndr6 · 2024-04-15T19:47:01Z

Environmental Info:
K3s Version: k3s version v1.29.3+k3s1 (8aecc26)
go version go1.21.8

Node(s) CPU architecture, OS, and Version: Linux astra 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

Cluster Configuration: Single node server

Describe the bug:
I recently upgraded my cluster from 1.27.3 to 1.29.3, and I encountered a problem with klipper-lb that I managed to solve, but I'm not sure if it's a misconfiguration somewhere on my side or a bug.
I use LoadBalancer services to expose the services on my node ports, and after the upgrade, some (3 out of 5) of the klipper-lb daemon sets were in a crash loop. The klipper pod was in a crash loop because the ipv6 forwarding sysctl was not enabled in the container, and the DEST_IPS variable contained both the ipv4 and ipv6 of the node (the daemon sets still working only had the node ipv4 in this variable), leading to an "exit 1" in the container entry script. As a temporary fix, I manually edited the daemonset config to add the missing sysctl. After that the daemonsets were working again.

Steps To Reproduce:

Installed K3s 1.27.3, and replaced traefik with nginx-ingress
Deployed multiple helm charts in separate namespaces
Used multiple (5) LoadBalancer services to route node ports to various services (service yaml provided below)
Upgraded to K3s 1.27.12, to 1.28.8, then to 1.29.3
After the last upgrade, some svclb DaemonSets are crashing

Expected behavior:
After the upgrade, the LoadBalancers should keep working as before

Actual behavior:
LoadBalancers are in a crash loop after the upgrade

Additional context / logs:
Service yaml file

apiVersion: v1
kind: Service
metadata:
  name: adguard-remote-udp
  namespace: adguard
spec:
  selector:
    app.kubernetes.io/instance: adguard-home
  externalTrafficPolicy: Local
  ports:
    - name: dns-udp
      protocol: UDP
      port: 53
      targetPort: 53
    - name: dns-crypt-udp
      protocol: UDP
      port: 5443
      targetPort: 5443
#    - name: dns-quic-udp2
#      protocol: UDP
#      port: 853
#      targetPort: 853
  type: LoadBalancer

Example of a fixed DaemonSet (added lines start with a star):

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: svclb-adguard-remote-udp-0c5cf893
  namespace: kube-system
  uid: 23884e2a-ab3b-485d-93b3-f7ceb7ec827e
  resourceVersion: '24223327'
  generation: 2
  creationTimestamp: '2024-04-13T14:55:00Z'
  labels:
    objectset.rio.cattle.io/hash: 9680de4f75badb67c1c2a69b6f3cf907879e7716
    svccontroller.k3s.cattle.io/nodeselector: 'false'
    svccontroller.k3s.cattle.io/svcname: adguard-remote-udp
    svccontroller.k3s.cattle.io/svcnamespace: adguard
  annotations:
    deprecated.daemonset.template.generation: '2'
    objectset.rio.cattle.io/applied: >-
      H4sIAAAAAAAA/9RVTW/jNhD9K8WcZcVeO3YsoIdgkxZBu45gO70sjGBEjmLWFEmQI22MQP+9oPKlIN6sUaALFDnEJGceOe+N3jwAOvUX+aCsgQzQuXDSjCCBnTISMrhAqqxZEUMCFTFKZITsAdAYy8jKmhCXtvibBAfi1CubCmTWlCp7oiIGJN89t98M+cFds4MMTppR8ssfyshfV+QbJeiHeQYrio+WdzV6OfBUWaZBLd1RmcGh6KVDm4Dw1NW0VhUFxspBZmqtE9BYkP6w0i2GLWQwn54NJU3K2WmBspjOxEh8wum8mJZjUc6Hs7PZnGaz0RQSCI0Q1rC3WpNPd+PQQzNWUiBNgq2HDErUgX6QEhrxER9HpB4g5AkxNEIXg/e4g6E4FeXZfAyPoc8Iu7qgQdgHpiqiBEcicvda0QNUyGL75wut6NxR97RtAkyV08jUwfR68gj1jr3mp1PdYwlrtpWtDT99BudCxNXa7shA1rVCAvECVIZ8gOzrA5Bpuv9Pz1otP9/m18s1JNCgruPW6Rja5E3A8nzx++WqFzJMu7+TYT/y4nK1vs2X1+vrXuTNRf4+5u1949F0Pn4XdJW/XPibt1Ust1Sk5ZLKl985cvySAiPXId3awFd5gLZtNwmoCu8imEcjtuRPdlo5R36gi6wZppN0Bk8xea11brUSe8jgqlxYzj0FMgwvTa2LTvPTqLaznh+ZfCE2t54hOx0nEJ/wujqU7S1bYfUzM5sEPAVbe0Gx66K2JGqveP/ZGqZ77roVHRZKK1b02JpSQvYVFpfr2/OLL1cL2LRtpO8IaSeTny3u8NPZ/0PcSM1H8k4mbwXu1ocx/jORNwdTwz4I1qGvvSFOlWsmqXK3pfXfonO8ijKCdtO9ou8ai56BQwJsNfnnsR19oyxJMGSwsCuxJVnrOGZ2FImNM2jgraY02rk3xBSihVUYmHwcsS5idQPq8l4FDl1H/BvIJ7scOI2Gvov8iPHZK1YC9bmU1oRro/eHEzbRU2snkWnFHpnu9pHW6MrK3N10B4+D6P7GYINKY6EJslGcMHsXWVu+ie08uuvbTtnaezK8qKuC/HOhErJhApKC8iQPHZlu74sK4cD2klDuIRu27T8BAAD//4jAUNOWCQAA
    objectset.rio.cattle.io/id: ''
    objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
    objectset.rio.cattle.io/owner-name: adguard-remote-udp
    objectset.rio.cattle.io/owner-namespace: adguard
  managedFields: <removed>
spec:
  selector:
    matchLabels:
      app: svclb-adguard-remote-udp-0c5cf893
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: svclb-adguard-remote-udp-0c5cf893
        svccontroller.k3s.cattle.io/svcname: adguard-remote-udp
        svccontroller.k3s.cattle.io/svcnamespace: adguard
    spec:
      containers:
        - name: lb-udp-53
          image: rancher/klipper-lb:v0.4.7
          ports:
            - name: lb-udp-53
              hostPort: 53
              containerPort: 53
              protocol: UDP
          env:
            - name: SRC_PORT
              value: '53'
            - name: SRC_RANGES
              value: 0.0.0.0/0
            - name: DEST_PROTO
              value: UDP
            - name: DEST_PORT
              value: '31693'
            - name: DEST_IPS
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIPs
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
        - name: lb-udp-5443
          image: rancher/klipper-lb:v0.4.7
          ports:
            - name: lb-udp-5443
              hostPort: 5443
              containerPort: 5443
              protocol: UDP
          env:
            - name: SRC_PORT
              value: '5443'
            - name: SRC_RANGES
              value: 0.0.0.0/0
            - name: DEST_PROTO
              value: UDP
            - name: DEST_PORT
              value: '30283'
            - name: DEST_IPS
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIPs
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: svclb
      serviceAccount: svclb
      automountServiceAccountToken: false
      securityContext:
        sysctls:
          - name: net.ipv4.ip_forward
            value: '1'
*          - name: net.ipv6.conf.all.forwarding
*            value: '1'
      schedulerName: default-scheduler
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: CriticalAddonsOnly
          operator: Exists
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0
  revisionHistoryLimit: 10
status:
  currentNumberScheduled: 1
  numberMisscheduled: 0
  desiredNumberScheduled: 1
  numberReady: 1
  observedGeneration: 2
  updatedNumberScheduled: 1
  numberAvailable: 1

Example of an unaffected DaemonSet:

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: svclb-plex-remote-97c6642d
  namespace: kube-system
  uid: 0dbb1aa7-8296-4e16-bfd0-d02c51a3a570
  resourceVersion: '24222317'
  generation: 2
  creationTimestamp: '2023-07-08T15:47:21Z'
  labels:
    objectset.rio.cattle.io/hash: 29277008bc62cc604d50c233fd0b23fdc8e5ff99
    svccontroller.k3s.cattle.io/nodeselector: 'false'
    svccontroller.k3s.cattle.io/svcname: plex-remote
    svccontroller.k3s.cattle.io/svcnamespace: plex
  annotations:
    deprecated.daemonset.template.generation: '2'
    objectset.rio.cattle.io/applied: >-
      H4sIAAAAAAAA/7xUwW7jNhD9lWLOkqzY3mQjoIcgCYqgXcewvb0sjGBEjmLWFEmQI22MQP9eULZTpXU2QQ8LHwyR7z2Sb97MM6BTf5IPyhooAJ0Lo/YMEtgqI6GAG6TamiUxJFATo0RGKJ4BjbGMrKwJ8dOWf5HgQJx5ZTOBzJoyZUcqakDy5r79bsinj+0WChi1Z8kvvysjf12Sb5Wgd3kGa4ICnKan1FNt+WOU4FAcedAlIDz1L1mpmgJj7aAwjdYJaCxJ//B9GwwbKGB8Ob64yPPPpTgfC3GeT+WnXIwnk0rm5XhSSfGZPlXV5SUkEFohrGFvtSafbSdhoGaspECaBFsPBVSoA71DCa046cIHOP+24aATWqHLdKCWXl6I8/PpWMIec+Rtm5LSsAtMdaQHRyJa9c8DnqFGFps/XlxE5358QNclwFQ7jUw9f5C4D1TpXf2f5OXADWzY1rYxfMj0lRDxa2W3ZKDoK5xAVEdlyAcovj0Dmbb/P1xmubh+mN8vVpBAi7qJS5PxNM+hS15hFlez326XA1Se9b/RK+TN7XL1MF/cr+4HyNX1/L+Yd47sQXfz4YFneTadZOM8z8Y5dOsEVI2PccOjERvyo61WzpFPdVm0eTbNLuCAmTdaz61WYgcF3FUzy3NPgQzDSy51mbJw6f4eCTjreW/Xi3tz6xmKHpDAxgZ+tfCGjLdshdVHE9YJeAq28YJipGIlSTRe8e7aGqYn7qOIDkulFSva505KKL7B7Hb1cHXz5W4G666LSieoYRcE6zCsryHOlGunmXIPlfXf0cuhp9Ct+1sM4zMbtCokwFaTPw7jGKCqIsFQwMwuxYZko2OMtxTNjTMm9VZTFvvXG2IKMcg1BiYf56eLWv0Aun1SgUNf9P8jeWia1Gk09KbyXuPaK1YC9ZWU1oR7o3enCevYXI2TyLRkj0yPu2hr7E1lHr/2G/vJ8/TVYItKY6kJirM4WXYuurZ4he2blZGbvpKi8Z4Mz5q6JH98qIQiT0BSUJ7kqS3Tr31RIZxYXhDKHRR51/0dAAD//+LjkalsBwAA
    objectset.rio.cattle.io/id: ''
    objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
    objectset.rio.cattle.io/owner-name: plex-remote
    objectset.rio.cattle.io/owner-namespace: plex
  managedFields: <removed>
spec:
  selector:
    matchLabels:
      app: svclb-plex-remote-97c6642d
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: svclb-plex-remote-97c6642d
        svccontroller.k3s.cattle.io/svcname: plex-remote
        svccontroller.k3s.cattle.io/svcnamespace: plex
    spec:
      containers:
        - name: lb-tcp-32400
          image: rancher/klipper-lb:v0.4.7
          ports:
            - name: lb-tcp-32400
              hostPort: 32400
              containerPort: 32400
              protocol: TCP
          env:
            - name: SRC_PORT
              value: '32400'
            - name: SRC_RANGES
              value: 0.0.0.0/0
            - name: DEST_PROTO
              value: TCP
            - name: DEST_PORT
              value: '32400'
            - name: DEST_IPS
              value: 10.43.200.20
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: svclb
      serviceAccount: svclb
      automountServiceAccountToken: false
      securityContext:
        sysctls:
          - name: net.ipv4.ip_forward
            value: '1'
      schedulerName: default-scheduler
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: CriticalAddonsOnly
          operator: Exists
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0
  revisionHistoryLimit: 10
status:
  currentNumberScheduled: 1
  numberMisscheduled: 0
  desiredNumberScheduled: 1
  numberReady: 1
  observedGeneration: 2
  updatedNumberScheduled: 1
  numberAvailable: 1

The text was updated successfully, but these errors were encountered:

brandond · 2024-04-15T20:50:28Z

Can you show the output of kubectl get service -n adguard adguard-remote-udp -o yaml and kubectl get node -o yaml? The IPv6 sysctl will be automatically added if the backing service has ipv6 enabled, however it looks like your service only has IPv4 enabled so it should not be necessary:

k3s/pkg/cloudprovider/servicelb.go

Lines 451 to 452 in 06b6444

    
           case core.IPv6Protocol: 
        
           	sysctls = append(sysctls, core.Sysctl{Name: "net.ipv6.conf.all.forwarding", Value: "1"})

It looks like you've set externalTrafficPolicy: Local on this service:

k3s/pkg/cloudprovider/servicelb.go

Lines 565 to 572 in 06b6444

    
           core.EnvVar{ 
        
           	Name: "DEST_IPS", 
        
           	ValueFrom: &core.EnvVarSource{ 
        
           		FieldRef: &core.ObjectFieldSelector{ 
        
           			FieldPath: getHostIPsFieldPath(), 
        
           		}, 
        
           	}, 
        
           },

I suspect might cause the klipper-lb script try to send traffic to the node's IPv6 address, even if the service itself only supports IPv4. It shouldn't do that.

Ndr6 · 2024-04-16T14:45:02Z

Here is the output for kubectl get service -n adguard adguard-remote-udp -o yaml:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"adguard-remote-udp","namespace":"adguard"},"spec":{"externalTrafficPolicy":"Local","ports":[{"name":"dns-udp","port":53,"protocol":"UDP","targetPort":53},{"name":"dns-crypt-udp","port":5443,"protocol":"UDP","targetPort":5443}],"selector":{"app.kubernetes.io/instance":"adguard-home"},"type":"LoadBalancer"}}
  creationTimestamp: "2024-04-13T14:55:00Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: adguard-remote-udp
  namespace: adguard
  resourceVersion: "24320629"
  uid: 0c5cf893-71db-4679-a5b3-1b8329dad8fc
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.43.161.206
  clusterIPs:
  - 10.43.161.206
  externalTrafficPolicy: Local
  healthCheckNodePort: 31464
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: dns-udp
    nodePort: 31693
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-crypt-udp
    nodePort: 30283
    port: 5443
    protocol: UDP
    targetPort: 5443
  selector:
    app.kubernetes.io/instance: adguard-home
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: <my external ip>

The output for kubectl get node -o yaml:

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      alpha.kubernetes.io/provided-node-ip: <my ipv4>,<my ipv6>
      flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"<>"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: <ipv4>
      k3s.io/hostname: astra
      k3s.io/internal-ip: <my ipv4>,<my ipv6>
      k3s.io/node-args: '["server","--disable","traefik"]'
      k3s.io/node-config-hash: QZWAV47A5VAKFX3MULWXBH3UEML5TZNT3W63QB5RUOQTNEUM22XA====
      k3s.io/node-env: '{"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/3fcd4fcf3ae2ba4d577d4ee08ad7092538cd7a7f0da701efa2a8807d44a25f66"}'
      node.alpha.kubernetes.io/ttl: "0"
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2023-07-06T11:24:57Z"
    finalizers:
    - wrangler.cattle.io/node
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/instance-type: k3s
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: astra
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: "true"
      node-role.kubernetes.io/master: "true"
      node.kubernetes.io/instance-type: k3s
      plan.upgrade.cattle.io/server-plan: fbb298c23beabb26f6297e41bcd3117ffbafac84d9ef95df775e3b14
    name: astra
    resourceVersion: "24502646"
    uid: 17c1b88f-742e-4316-941b-9e87430b90d6
  spec:
    podCIDR: 10.42.0.0/24
    podCIDRs:
    - 10.42.0.0/24
    providerID: k3s://astra
  status:
    addresses:
    - address: <my ipv4>
      type: InternalIP
    - address: <my ipv6>
      type: InternalIP
    - address: astra
      type: Hostname
    allocatable:
      cpu: "8"
      ephemeral-storage: "1959802746163"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 65733840Ki
      pods: "110"
    capacity:
      cpu: "8"
      ephemeral-storage: 2014599864Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 65733840Ki
      pods: "110"
    conditions:
    - lastHeartbeatTime: "2024-04-16T14:39:24Z"
      lastTransitionTime: "2023-07-06T11:24:57Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2024-04-16T14:39:24Z"
      lastTransitionTime: "2023-07-06T11:24:57Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2024-04-16T14:39:24Z"
      lastTransitionTime: "2023-07-06T11:24:57Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2024-04-16T14:39:24Z"
      lastTransitionTime: "2024-04-13T15:31:30Z"
      message: kubelet is posting ready status. AppArmor enabled
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images: <deployed images>
    nodeInfo:
      architecture: amd64
      bootID: f1afbea7-14c8-4bd1-ac84-dc62ecbb3d82
      containerRuntimeVersion: containerd://1.7.11-k3s2
      kernelVersion: 6.1.0-20-amd64
      kubeProxyVersion: v1.29.3+k3s1
      kubeletVersion: v1.29.3+k3s1
      machineID: b3d19325a7ee411791c9288a15e00c0c
      operatingSystem: linux
      osImage: Debian GNU/Linux 12 (bookworm)
      systemUUID: 00000000-0000-0000-0000-ac1f6b0065dc
kind: List
metadata:
  resourceVersion: ""

brandond · 2024-04-16T17:53:23Z

OK, I can replicate this:

Start k3s with --node-ip=<ipv4>,<ipv6>
Patch the traefik pod to change the external traffic policy: kubectl patch service -n kube-system traefik -p '{"spec": {"externalTrafficPolicy": "Local"}}'
Note that the servicelb pods start crashing:

brandond@dev01:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS             RESTARTS      AGE
svclb-traefik-d855c4d4-b9v52   0/2     CrashLoopBackOff   8 (48s ago)   2m12s

brandond@dev01:~$ kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik -o yaml | grep -A3 podIP:
    podIP: 10.42.0.9
    podIPs:
    - ip: 10.42.0.9
    qosClass: BestEffort

brandond@dev01:~$ kubectl logs -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik -c lb-tcp-80
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '==' 1 ]
+ iptables -t filter -A FORWARD -d 172.17.0.8/32 -p TCP --dport 31503 -j DROP
+ iptables -t nat -I PREROUTING -p TCP --dport 80 -j DNAT --to 172.17.0.8:31503
+ iptables -t nat -I POSTROUTING -d 172.17.0.8/32 -p TCP -j MASQUERADE
+ echo fd7c:53a5:aef5::242:ac11:8
+ grep -Eq :
+ cat /proc/sys/net/ipv6/conf/all/forwarding
+ '[' 0 '==' 1 ]
+ exit 1

The pod and service are both ipv4 only but the node has an ipv6 address, so it incorrectly tries to set up ipv6 forwarding.

fmoral2 · 2024-06-05T15:11:35Z

Validated on Version:

-$   k3s version v1.30.1+k3s-df5db28a (df5db28a)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu
AMD

Cluster Configuration:
-1 nodes

Steps to validate the fix

start k3s with --node-ip=1pv4,1pv6
Patch service traefik to -p '{"spec": {"externalTrafficPolicy": "Local"}}'
Validate nodes, pods,serviceLB

Reproduction Issue:

 k3s version v1.30.1+k3s-5cf4d757 (5cf4d757)

 kubectl patch service traefik  -n kube-system -p '{"spec": {"externalTrafficPolicy": "Local"}}'
 service/traefik patched

 kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik

NAME                           READY   STATUS             RESTARTS      AGE
svclb-traefik-a436cc37-mwskr   0/2     CrashLoopBackOff   2 (13s ago)   15s



~$ kubectl  get pods -A
NAMESPACE     NAME                                      READY   STATUS             RESTARTS       AGE
kube-system   local-path-provisioner-75bb9ff978-bf7fw   1/1     Running            0              4m59s
kube-system   coredns-576bfc4dc7-h8nlj                  1/1     Running            0              4m59s
kube-system   helm-install-traefik-crd-x84bt            0/1     Completed          0              4m59s
kube-system   helm-install-traefik-4bbh4                0/1     Completed          1              4m59s
kube-system   metrics-server-557ff575fb-sfwzq           1/1     Running            0              4m59s
kube-system   traefik-5fb479b77-8nmm8                   1/1     Running            0              4m32s
kube-system   svclb-traefik-a436cc37-mwskr              0/2     CrashLoopBackOff   10 (12s ago)   2m56s

Validation Results:

 $ kubectl  get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-576bfc4dc7-pzfbg                  1/1     Running     0          35s
kube-system   helm-install-traefik-bnpvl                0/1     Completed   1          35s
kube-system   helm-install-traefik-crd-z76g6            0/1     Completed   0          35s
kube-system   local-path-provisioner-86f46b7bf7-crlk2   1/1     Running     0          35s
kube-system   metrics-server-557ff575fb-9b2mn           1/1     Running     0          35s
kube-system   svclb-traefik-8ea8106e-9txhr              2/2     Running     0          9s
kube-system   traefik-5fb479b77-cwwvk                   1/1     Running     0          9s


ubuntu@:~$ kubectl patch service traefik  -n kube-system -p '{"spec": {"externalTrafficPolicy": "Local"}}'
service/traefik patched


ubuntu@:~$  kubectl get pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik
NAME                           READY   STATUS    RESTARTS   AGE
svclb-traefik-8ea8106e-xthwt   2/2     Running   0          11s


ubuntu@:~$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-576bfc4dc7-pzfbg                  1/1     Running     0          64s
kube-system   helm-install-traefik-bnpvl                0/1     Completed   1          64s
kube-system   helm-install-traefik-crd-z76g6            0/1     Completed   0          64s
kube-system   local-path-provisioner-86f46b7bf7-crlk2   1/1     Running     0          64s
kube-system   metrics-server-557ff575fb-9b2mn           1/1     Running     0          64s
kube-system   svclb-traefik-8ea8106e-xthwt              2/2     Running     0          21s
kube-system   traefik-5fb479b77-cwwvk                   1/1     Running     0          38s

Ndr6 changed the title ~~LoadBalancers crashing after updating from 1.27.3~~ LoadBalancers crashing after updating from 1.27.3 to 1.29.3 Apr 15, 2024

brandond added the kind/bug Something isn't working label Apr 16, 2024

brandond self-assigned this Apr 16, 2024

brandond added this to the v1.30.1+k3s1 milestone Apr 16, 2024

brandond mentioned this issue Apr 17, 2024

Fix issue with externalTrafficPolicy: Local for single-stack services on dual-stack nodes #9963

Merged

caroline-suse-rancher modified the milestones: v1.30.1+k3s1, v1.30.2+k3s1 May 13, 2024

hmeerlo mentioned this issue May 30, 2024

K8s Ingress stopped working in 1.6.x orbstack/orbstack#1237

Open

ShylajaDevadiga assigned fmoral2 May 31, 2024

fmoral2 closed this as completed Jun 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LoadBalancers crashing after updating from 1.27.3 to 1.29.3 #9949

LoadBalancers crashing after updating from 1.27.3 to 1.29.3 #9949

Ndr6 commented Apr 15, 2024 •

edited

Loading

brandond commented Apr 15, 2024 •

edited

Loading

Ndr6 commented Apr 16, 2024

brandond commented Apr 16, 2024 •

edited

Loading

fmoral2 commented Jun 5, 2024

Validation Results:

LoadBalancers crashing after updating from 1.27.3 to 1.29.3 #9949

LoadBalancers crashing after updating from 1.27.3 to 1.29.3 #9949

Comments

Ndr6 commented Apr 15, 2024 • edited Loading

brandond commented Apr 15, 2024 • edited Loading

Ndr6 commented Apr 16, 2024

brandond commented Apr 16, 2024 • edited Loading

fmoral2 commented Jun 5, 2024

Validated on Version:

Environment Details

Steps to validate the fix

Reproduction Issue:

Validation Results:

Ndr6 commented Apr 15, 2024 •

edited

Loading

brandond commented Apr 15, 2024 •

edited

Loading

brandond commented Apr 16, 2024 •

edited

Loading