EKS Module does not allow clean destroy (dependency violation)

[archmangler]

It seems this EKS module implementation does not allow clean destroy.

I get the following on terraform destroy without modification of the code:

Error: error deleting EC2 Subnet (subnet-0cc3d27fc54396aea): DependencyViolation: The subnet 'subnet-0cc3d27fc54396aea' has dependencies and cannot be deleted.
        status code: 400, request id: fc2374cc-b57e-491b-8d0e-7ff5e0ea8c04



Error: error deleting EC2 Subnet (subnet-0aa0c6858a9610951): DependencyViolation: The subnet 'subnet-0aa0c6858a9610951' has dependencies and cannot be deleted.
        status code: 400, request id: 7e0adef1-f9cf-4f77-bd53-76ee3cadddb4



Error: error deleting EC2 Subnet (subnet-06c15b8c3fe729839): DependencyViolation: The subnet 'subnet-06c15b8c3fe729839' has dependencies and cannot be deleted.
        status code: 400, request id: 7abb9076-1f27-42a7-aa58-55f9d6c0684f



Error: error deleting EC2 VPC (vpc-085a1ba062c00608c): DependencyViolation: The vpc 'vpc-085a1ba062c00608c' has dependencies and cannot be deleted.
        status code: 400, request id: 86fc16a6-7486-4c13-ad8a-3ecc10d2f17
Error: error detaching EC2 Internet Gateway (igw-017db3cdb54073820) from VPC (vpc-085a1ba062c00608c): DependencyViolation: Network vpc-085a1ba062c00608c has some mapped public address(es). Please unmap those public address(es) before detaching the gateway.
        status code: 400, request id: f21a4901-278b-423a-83e6-9dec07fa8fc2

NOTE: I found the following article useful in cleanup: https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-dependency-error-delete-vpc/

[bryantbiggs]

This is most likely due to anything you provisioned onto the cluster that creates ENIs (ALB ingress controller, NGINX ingress, etc.)

[dwschulze]

So if I create the EKS cluster with terraform, but deploy an application with kubectl then terraform delete won't work?

This has left me in a mess where I have to try to delete the left over resources manually. What's the point of terraform if it can't destroy all the cluster resources just because you deployed an app to EKS?

[bryantbiggs]

If you deploy an app thats is just a pod on the cluster, you can safely delete the cluster with Terraform without deleting the app.

If you deploy something like the AWS load balancer controller, that creates additional AWS resources *outside of Terraform's control, and therefore it has no visibility into those resources, but those resources are consuming resources created by Terraform - using the OPs error message, I would suspect this is some form of a load balancer that is utilizing the subnets of the VPC and therefore that load balancer controller *HAS to be deleted before any terraform destroy command is issued

[dwschulze]

I have to wonder if Terraform should be used to create an EKS cluster.

A cluster will have deployments including load balancers in many cases. Then Terraform's state is stale. If you forget to destroy a load balancer or anything else a deployment has created and run terraform destroy you get a real mess. You're faced with a long manual process of trying to find the remaining resources and delete them one by one.

AWS has Resource Explorer, but it shows you all the default resources in every region which you don't want to destroy. You have to try and find your orphaned resources in a big haystack of defaults. It's practically worthless.

Do you have any suggestions for how to clean up these orphaned resources?

[bryantbiggs]

I think you are missing the crux of the issue - any IaC tool will face the same challenge. IaC tools will only manage those resources that they know about and are in control of, so you have to plan your workflow accordingly when bridging across different domains/tools

[dwschulze]

aws-nuke was able to clean up the mess. aws-nuke looks like something you'll need to clean up after terraform EKS clusters.

[bryantbiggs]

again, this is not specific to EKS. If I launch an EC2 instance with Terraform, where a custom program runs on that instance that launches other Ec2 instances or other AWS resources - if I run terraform destroy it will destroy the Terraform controlled instance and leave all of the other resources. If you take this one step further by deploying the VPC and the Ec2 instance at the same time, when you try to run terraform destroy it will eventually fail since it has dependencies that have not been removed (the additional EC2 instances and other AWS resources created by the custom program on the Terraform provisioned EC2 instance) because Terraform does not know about these other resources and nothing else is trying to remove them to clean up the resources created

[dwschulze]

Terraform destroy should not leave a mess. It should be able to destroy the things it created. If it cannot do that it needs to warn you in advance. This kind of check should be part of the plan step.

[BrianMMcClain]

Hey all, I'm going to go ahead and mark this one as closed since there hasn't been much activity lately.

I wanted to raise one option however. You can manage resources that you deploy on top of Kubernetes with Terraform too, such as the Kubernetes provider or the Helm provider. In this case, those resources would be in a Terraform state file and a terraform destroy would destroy them.