Skip to content
This repository has been archived by the owner on Jan 19, 2022. It is now read-only.

cfn_delete consistently fails #177

Closed
lgarvey opened this issue Jan 28, 2016 · 3 comments
Closed

cfn_delete consistently fails #177

lgarvey opened this issue Jan 28, 2016 · 3 comments

Comments

@lgarvey
Copy link

lgarvey commented Jan 28, 2016

CF output:

11:54:19 UTC+0000 DELETE_FAILED AWS::EC2::VPCGatewayAttachment AttachGateway Network vpc-69a8810c has some mapped public address(es). Please unmap those public address(es) before detaching the gateway.

pip freeze tells me:

bootstrap-cfn==0.7.5

If I manually find the IGW and detach it, then I can delete the stack.

@lgarvey lgarvey added the bug label Jan 28, 2016
@mattpep
Copy link
Contributor

mattpep commented Mar 1, 2016

Though I mostly get this error, once in a while (say 1 out of 5, perhaps a little less often) I've had a stack successfully delete. I suspect this is related to the speed of some operations, and on the occasions a stack has deleted, it's been because an ordinarily slow operation has completed before the depending VPC deleted.

The core work of the cfn_delete task is a single call (source file bootstrap_cfn/cloudformation.py, class method delete) to the boto library. So we can't resolve this at deletion time, but we can prevent the issue from cropping up by changing what we do at creation time. Advice found at http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-delete-stack-fails includes:

To resolve a dependency error, add a DependsOn attribute to resources that depend on other resources in your template.

Once the fix is in place, we'll still have this issue on stacks which have been created in the period between after the problematic VPC was added[1] and before that problematic VPC had its dependency added. But future stacks will be deletable without problem.

[1] Identifying this VPC is the crux of this issue

@mattpep
Copy link
Contributor

mattpep commented Apr 15, 2016

Our AWS Solutions Architect provided the following input:

I believe it may be due to a missing DependsOn in your CF template. The most common cause for this is when you have an EIP without an explicit DependsOn to the Internet Gateway. Looking at your template specifically, it seems like the Auto Scaling Launch Configuration (which specifies that instances get a public IP) should have “DependOn” : “AttachGateway”, so that the InternetGateway is created and attached before the instances launch with public Ips.
See here: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html#gatewayattachment

@mattpep mattpep added the sweep1 label Apr 19, 2016
sevenmachines pushed a commit that referenced this issue Jun 29, 2016
Some cloudformation resources depend on other resources before they
can be created or deleted. This especially causes problems when deleting
VPC's, meaning that the delete fails since there are dependent resources
still in the process of being removed form the VPC.

By adding the DependsOn attribute to these resources, this problem can be resolved.
This change uses the AWS guidance on when this is required to add this attribute to resources
that do, or may require it. The details are in the link below.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html#gatewayattachment

Following the advice in the document above, the resources below are considered,

Already has DependsOn the VPCGatewayAttachment
* Amazon VPC routes that include the Internet gateway

Added DependsOn the VPCGatewayAttachment since EC2 instances have public ips
* Auto Scaling groups
* Elastic Load Balancers

Added DependsOn the VPCGatewayAttachment since they can have public ips
* RDS instances

No action required
* Elastic IP addresses - we dont have any yet
* Amazon EC2 instances - these are handled by the ASG

(Closes #177)
@filipposc5
Copy link
Contributor

fixed by #214

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants