Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pending load balancer security groups leak in AWS legacy cloud provider #79994

Open
2opremio opened this issue Jul 10, 2019 · 1 comment

Comments

Projects
None yet
2 participants
@2opremio
Copy link
Contributor

commented Jul 10, 2019

What happened:

A security group is leaked when a <pending> (due to having reached the ELB limit) type: LoadBalancer Service is deleted.

What you expected to happen:

The Security group should be deleted when deleting the service.

How to reproduce it (as minimally and precisely as possible):

  1. Create file memcache-svc.yaml:
---
apiVersion: v1
kind: Service
metadata:
  name: service$I
spec:
  ports:
    - name: memcached
      port: 11211
  type: LoadBalancer
  selector:
    name: memcached
  1. Create enough services to surpass the ELB limit of your account. In my case 25 is enough to reach it, since the cluster is in a region with the default limit of 20 ELBS:
for I in `seq 25`; do cat memcache-svc.yaml | I=$I envsubst  | kubectl apply -f - ; done
  1. Wait for the the services to be created (check with kubectl get services), the EXTERNAL-IP of some of them will remain pending due to the ELB limit having been reached (in my case service21, service22, service23 service24 and service25 are in that state)

  2. Delete all the services

for I in `seq 25`; do kubectl delete service service$I ; done
  1. Wait for their corresponding ELBs to be deleted by AWS the cloud provider. Wait some more.

  2. List the security groups in the cluster's AWS region. You will encounter that the ones matching the PENDING services (service21, service22, service23 service24 and service25) remain there and won't be deleted.

Anything else we need to know?:

I discovered this while working on weaveworks/eksctl#1010

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-eks-c57ff8", GitCommit:"c57ff8e35590932c652433fab07988da79265d5b", GitTreeState:"clean", BuildDate:"2019-06-07T20:43:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: Amazon EKS (Kubernetes 1.13)
  • Other: The cluster was created with version Release 0.1.39, runnig eksctl create cluster -f cluster.yaml with:

cluster.yaml:

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: fons-cluster-4
  region: eu-north-1

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 2
@2opremio

This comment has been minimized.

Copy link
Contributor Author

commented Jul 10, 2019

/sig AWS
/sig cloud-provider

2opremio added a commit to weaveworks/eksctl that referenced this issue Jul 11, 2019

Wait for the loadbalancer security groups to be deleted
Also:

1. print warnings when unexpected conditions occur
2. deleted orphaned security groups due to kubernetes/kubernetes#79994
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.