Strimizi kafka deployments dont delete LB

[pashafirdous]

hi,

we are observing a strange behaviour with strimzi deployments(0.17.0)
we have a lot of strimzi deployments in GKE env and we do our prod and dev envs in it.
when we delete kafka deployments, we see that the forwardingrules/Loadbalancers dont cleanup by itself/ mayby some finalizer is not removed from loadbalancer service upon CR removal

could you please advice how we can fix this?

regards,
PAsha

[scholzj]

Strimzi does not create the load balancers. We just create a Kubernetes service with type=LoadBalancer. And Kubernetes takes care of the load balancer creation and configuration. Similarly, we do not delete the load balancers but just the service.

So you can check if the related service using this load balancer exists => if it doesn't, this is purely Kubernetes thing and you would need to check in Kube logs / with GKE support.

If the service exists, we would need to look into why it is not deleted. Strimzi relies on the Kubernetes garbage collection for this, so the first thing to check would be the owner reference in the service.

[pashafirdous]

is there a plan to include the finalizer/ option to delete the loadbalancer?
i guess this will be a challenge for LB quota on any cloud platform.

[scholzj]

The loadbalancers are normally created, managed and deleted by Kubernetes. Strimzi does not create them or delete them. So I'm not sure what we could do about it.

[cleberwarmling]

I'm facing this exact problem. Hit our LB quota on GCP and after some investigation, I saw that Strimzi LBs were not being deleted on GCP after the service got deleted. I've to reach GCP support to confirm that the problem was the missing Finalizer on Services' metadata.

finalizers:
- service.kubernetes.io/load-balancer-cleanup

For some reason, Strimzi's services do not get those finalizers in place and I'm couldn't add them through templating as well.

Also, GCP answer below:

_

After checking other active services with similar naming convention I found that they are managed by Strimzi operators. The reason why this is important is because Strimzi operator managed clusters do not utilize or permit the use of Kubernetes finalizers. In particular, the finalizer responsible for removing associated load balancers, "service.kubernetes.io/load-balancer-cleanup". This is a known issue with Strimzi managed clusters [1]. For more details on the various resources that Strimzi permits, please review the following documentation [2]. You'll notice that finalizers are not within this list of Kubernetes resources permitted within Strimzi operators.

_

[scholzj]

Not sure whether the quoted text comes from and where do the links point. We can make finalizers configurable - please open an enhancement issue for it. I would need to check if this really works everywhere and can be done automatically by Strimzi ... last time I checked it was supported only somewhere and it would block the deletion of the service in some environments.

[cleberwarmling]

That was from GCP support, link 1 points here and 2 to template spec docs. But gotcha. I Will do that, thanks.

[scholzj]

Ok, thanks.

I tried it on my Kube 1.20 cluster on bare-metal and it looks like the finalizer never deletes. So I guess it is not something we can set by default because it is not supported everywhere. But we can make it configurable to allow you to set them in the Kafka CR.

PS: The Kube docs say it should happen in corner cases that the LB is not deleted ... so I guess it is a pretty big corner if none of the LBs gets deleted.

[scholzj]

FYI: Handled in #4500 / #4607.