Repeated nodeAssigned events on ingress service

[frnmz]

Summary

Our app, which has a login section, always gets disconnected whenever the 'nodeAssigned' event occurs. This happened on MicroK8s version 1.27.4, revision 5643.
Below is the description of the ingress service:

Name:                     ingress-service
Namespace:                ingress
Labels:                   <none>
Annotations:              <none>
Selector:                 name=nginx-ingress-microk8s
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.152.183.224
IPs:                      10.152.183.224
IP:                       10.10.13.100
LoadBalancer Ingress:     10.10.13.100
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32176/TCP
Endpoints:                10.1.174.193:80,10.1.175.65:80,10.1.200.1:80 + 2 more...
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  32174/TCP
Endpoints:                10.1.174.193:443,10.1.175.65:443,10.1.200.1:443 + 2 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                     From             Message
  ----    ------        ----                    ----             -------
  Normal  nodeAssigned  2m16s (x16 over 9m19s)  metallb-speaker  announcing from node "dcim-worker2" with protocol "layer2"

What Should Happen Instead?

With the same app and the same manifest this does not happened on microk8s v1.23.17 revision 4916

Reproduction Steps

This is simple yaml that we use to deploy ingress and kube-dashboard

apiVersion: v1
kind: Service
metadata:
  name: ingress-service
  namespace: ingress
spec:
  type: LoadBalancer
  loadBalancerIP: 10.10.13.100
  selector:
    name: nginx-ingress-microk8s
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: public
  tls:
    - hosts:
        - k8s-dcim.nexcloud.id
      secretName: star-nexcloud
  rules:
    - host: "k8s-dcim.nexcloud.id"
      http:
        paths:
          - path: /dashboard(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: kubernetes-dashboard

Could this be due to a difference in MetalLB versions, or is it related to our configuration?

Thanks!
inspection-report-20230808_104904.tar.gz

[neoaggelos]

Hi @frnmz, sorry for missing this.

How often does the nodeAssigned event happen? It should happen once when first creating the LoadBalancer service but I would expect it not to occur afterwards (assuming a healthy cluster).

[frnmz]

Hi @neoaggelos sorry for late followup.
These 'nodeAssigned' events occur randomly but typically every minute. Here's the service description of Istio that I enabled via 'microk8s enable':"

Name:                     istio-ingressgateway
Namespace:                istio-system
Labels:                   app=istio-ingressgateway
                          install.operator.istio.io/owning-resource=unknown
                          install.operator.istio.io/owning-resource-namespace=istio-system
                          istio=ingressgateway
                          istio.io/rev=default
                          operator.istio.io/component=IngressGateways
                          operator.istio.io/managed=Reconcile
                          operator.istio.io/version=1.15.3
                          release=istio
Annotations:              <none>
Selector:                 app=istio-ingressgateway,istio=ingressgateway
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.152.183.237
IPs:                      10.152.183.237
LoadBalancer Ingress:     10.10.13.150
Port:                     status-port  15021/TCP
TargetPort:               15021/TCP
NodePort:                 status-port  31606/TCP
Endpoints:                10.1.175.72:15021
Port:                     http2  80/TCP
TargetPort:               8080/TCP
NodePort:                 http2  30015/TCP
Endpoints:                10.1.175.72:8080
Port:                     https  443/TCP
TargetPort:               8443/TCP
NodePort:                 https  30975/TCP
Endpoints:                10.1.175.72:8443
Port:                     tcp  31400/TCP
TargetPort:               31400/TCP
NodePort:                 tcp  32337/TCP
Endpoints:                10.1.175.72:31400
Port:                     tls  15443/TCP
TargetPort:               15443/TCP
NodePort:                 tls  31441/TCP
Endpoints:                10.1.175.72:15443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                   From             Message
  ----    ------        ----                  ----             -------
  Normal  nodeAssigned  3m41s (x17 over 29m)  metallb-speaker  announcing from node "dcim-worker3" with protocol "layer2"
  Normal  nodeAssigned  97s (x9 over 2m25s)   metallb-speaker  announcing from node "dcim-worker1" with protocol "layer2"
  Normal  nodeAssigned  97s (x5 over 97s)     metallb-speaker  announcing from node "dcim-worker3" with protocol "layer2"

This issue has occurred in both our in-office staging server and our data center production server. It started happening from microk8s version 1.24 through 1.27 but did not occur in microk8s version 1.23.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.