Reassignment Duplicate IP after deleting IP pool

[SihengCui]

Hello! I have find a problem caused by IP reassignment after deleting IP pool.
k8s version: 1.25.0 metallb: 0.13.5
At first, I have two autoAssign ip pools.
kubectl get ipaddresspools -n metallb-system -oyaml

apiVersion: v1
items:
- apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  metadata:
    creationTimestamp: "2022-09-06T08:01:03Z"
    generation: 1
    name: cheap
    namespace: metallb-system
    resourceVersion: "1739387"
    uid: 64c320ee-0570-4685-a619-e533afa53b65
  spec:
    addresses:
    - 192.168.1.0/24
    autoAssign: true
    avoidBuggyIPs: true
- apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  metadata:
    creationTimestamp: "2022-09-06T08:05:47Z"
    generation: 1
    name: default
    namespace: metallb-system
    resourceVersion: "1740838"
    uid: 436518ec-f63c-4921-9510-c4d2e8a4941f
  spec:
    addresses:
    - 192.168.10.0/24
    autoAssign: true
    avoidBuggyIPs: true
kind: List
metadata:
  resourceVersion: ""

The services may be divided into ips in two different ip pools.
kubectl get svc -A | grep Load

          test         sanxianyimian-ng                     LoadBalancer   10.240.181.10    192.168.10.1    80:26491/TCP                                 3d22h
istio-system           istio-ingressgateway                 LoadBalancer   10.240.160.224   192.168.1.1   15021:37434/TCP,80:39462/TCP,443:33942/TCP   4d22h
monitor                kube-state-metrics                   LoadBalancer   10.240.92.35     192.168.1.2   8080:21949/TCP,8081:22126/TCP                4d1h

Then I deleted the default1 ippool, restart controller, in my opinion, the reassigned ip for istio-ingressgateway and kube-state-metrics should not use 192.168.10.1.
kubectl delete ipaddresspools cheap -n metallb-system
But

test                   sanxianyimian-ng                     LoadBalancer   10.240.181.10    192.168.10.1   80:26491/TCP                                 3d23h
istio-system           istio-ingressgateway                 LoadBalancer   10.240.160.224   192.168.10.2   15021:37434/TCP,80:39462/TCP,443:33942/TCP   4d22h
monitor                kube-state-metrics                   LoadBalancer   10.240.92.35     192.168.10.1   8080:21949/TCP,8081:22126/TCP                4d1h

[woodgear]

try restart metallb controller pod?

[SihengCui]

try restart metallb controller pod?

yes , If the controller is not restarted, the IP will not be reallocated.

[SihengCui]

these are some errors in controller

{"caller":"service.go:99","error":"can't change sharing key for \"cuisiheng-test/sanxianyimian2\", address also in use by monitor/kube-state-metrics: services can't share key","event":"clearAssignment","level":"info","msg":"current IP not allowed by config, clearing","ts":"2022-09-06T08:27:34Z"}

{"caller":"service_controller.go:92","controller":"ServiceReconciler","endpoints":"{\"Type\":0,\"EpVal\":null,\"SlicesVal\":null}","event":"failed to handle service","level":"info","name":"cuisiheng-test/sanxianyimian2","service":"{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"sanxianyimian2\",\"namespace\":\"cuisiheng-test\",\"uid\":\"6603fc33-0c60-4e49-aa28-c869d113366c\",\"resourceVersion\":\"1746503\",\"creationTimestamp\":\"2022-09-06T08:13:49Z\",\"labels\":{\"app.kubernetes.io/component\":\"sanxianyimian-ng\",\"app.kubernetes.io/instance\":\"sanxianyimian\",\"app.kubernetes.io/managed-by\":\"Tiller\",\"app.kubernetes.io/name\":\"sanxianyimian\",\"application/name\":\"sanxianyimian\",\"helm.sh/chart\":\"sanxianyimian-0.1.0\"},\"annotations\":{\"app.kubernetes.io/component\":\"sanxianyimian-ng\",\"app.kubernetes.io/instance\":\"sanxianyimian\",\"app.kubernetes.io/name\":\"sanxianyimian\",\"helm.sh/chart\":\"sanxianyimian-0.1.0\"},\"managedFields\":[{\"manager\":\"caihcloud\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2022-09-06T08:24:17Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/instance\":{},\"f:app.kubernetes.io/name\":{},\"f:helm.sh/chart\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/instance\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{},\"f:application/name\":{},\"f:helm.sh/chart\":{}}},\"f:spec\":{\"f:allocateLoadBalancerNodePorts\":{},\"f:externalTrafficPolicy\":{},\"f:internalTrafficPolicy\":{},\"f:ipFamilies\":{},\"f:ipFamilyPolicy\":{},\"f:ports\":{\".\":{},\"k:{\\\"port\\\":80,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:name\":{},\"f:port\":{},\"f:protocol\":{},\"f:targetPort\":{}}},\"f:selector\":{},\"f:sessionAffinity\":{},\"f:type\":{}}}},{\"manager\":\"controller\",\"operation\":\"Update\",\"apiVersion\":\"v1\",\"time\":\"2022-09-06T08:26:46Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:loadBalancer\":{\"f:ingress\":{}}}},\"subresource\":\"status\"}]},\"spec\":{\"ports\":[{\"name\":\"80port\",\"protocol\":\"TCP\",\"port\":80,\"targetPort\":80,\"nodePort\":37857}],\"selector\":{\"app.kubernetes.io/component\":\"sanxianyimian-ng\",\"app.kubernetes.io/instance\":\"sanxianyimian\",\"app.kubernetes.io/name\":\"sanxianyimian\",\"application/name\":\"sanxianyimian\"},\"clusterIP\":\"10.240.246.193\",\"clusterIPs\":[\"10.240.246.193\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"192.168.10.1\"}]}}}","ts":"2022-09-06T08:27:34Z"}

{"level":"error","ts":1662452854.9475472,"msg":"Reconciler error","controller":"service","controllerGroup":"","controllerKind":"Service","service":{"name":"sanxianyimian2","namespace":"cuisiheng-test"},"namespace":"cuisiheng-test","name":"sanxianyimian2","reconcileID":"a6e8c78b-0370-49db-9b01-c9e8ffc9dfe8","error":"event handling failed, retrying","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234"}

[fedepaol]

Hi @SihengCui , sorry for the delay in getting back.
Can you provide also the definition of your services? I tried to reproduce this without specific annotations / loadbalancer ips and I couldn't

[SihengCui]

Hi @SihengCui , sorry for the delay in getting back. Can you provide also the definition of your services? I tried to reproduce this without specific annotations / loadbalancer ips and I couldn't

There will be a random part of the situation where the IP cannot be modified. The service template is exactly the same, only the name is different.

kind: Service
apiVersion: v1
metadata:
  name: service1 service2 service3 service4
  namespace: cuisiheng-test
spec:
  ports:
    - name: 80port
      protocol: TCP
      port: 80
      targetPort: 80port
  selector:
    application/name: sxym
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  allocateLoadBalancerNodePorts: false
  internalTrafficPolicy: Cluster

My kubernetes version is 1.25.2.
First, the lb ip are 192.168.2.1 192.168.2.2 192.168.2.3 192.168.2.4.(service1 - service4)
I annotate service3 and service4 with metallb.universe.tf/address-pool: cheap. so they change to 192.168.10.1 192.168.10.2. After that I removed the annotation.
Delete the cheap pool and restart the controller.
service3 ip is 192.168.2.3, service4 ip is 192.168.2.1.

At this point service1 began to report that the ip could not be shared.

Set the service of duplicate ip to clusterIP type and then change it back to loadbalancer, it works correctly.

[fedepaol]

I think I found the bug, need to think about how to address it properly. It depends on the order the controller receives the service when it gets back, and was introduced (I think) by #1230
What should've happened is, if the controller receives sanxianyimian-ng first, it can assign the IP (because it doesn't know yet another service holds it), but when it gets the previous holder of that IP, it must change it instead of skipping.