Apply fails when changing an AZ

[natefoxrax]

Found a bug? Maybe our Slack Community can help.

Describe the Bug

When changing an AZ (eg 1c to 1d), the apply fails.

Expected Behavior

The apply to not fail and the affected subnets to be recreated.

Steps to Reproduce

Steps to reproduce the behavior:

provider "aws" {
  region = "us-east-1"
}

module "vpc" {
  source = "git::https://github.com/cloudposse/terraform-aws-vpc.git?ref=tags/0.19.0"
  namespace  = "eg"
  stage      = "test"
  name       = "app"
  cidr_block = "10.207.0.0/16"
}

module "dynamic_subnets" {
  source = "git::https://github.com/cloudposse/terraform-aws-dynamic-subnets.git?ref=tags/0.36.1"

  namespace          = "eg"
  stage              = "test"
  name               = "app"
  availability_zones = ["us-east-1a","us-east-1b","us-east-1c"]
  vpc_id             = module.vpc.vpc_id
  igw_id             = module.vpc.igw_id
  cidr_block         = "10.207.0.0/16"
}

1. Save the above as a file and run terraform init && terraform apply -auto-approve
2. Now change us-east-1c to us-east-1d and run terraform apply -auto-approve

Output is similar to:

➔ terraform apply -auto-approve
module.dynamic_subnets.module.utils.data.aws_regions.default: Refreshing state... [id=aws]
module.dynamic_subnets.module.utils.data.aws_regions.opted_in: Refreshing state... [id=aws]
module.dynamic_subnets.module.utils.data.aws_regions.not_opted_in: Refreshing state... [id=aws]
module.dynamic_subnets.data.aws_availability_zones.available[0]: Refreshing state... [id=us-east-1]
module.dynamic_subnets.aws_eip.default[0]: Refreshing state... [id=eipalloc-0e1220c3f15593b16]
module.dynamic_subnets.aws_eip.default[1]: Refreshing state... [id=eipalloc-00a5ac071e46de3d1]
module.dynamic_subnets.aws_eip.default[2]: Refreshing state... [id=eipalloc-05176125e443dedf1]
module.vpc.aws_vpc.default[0]: Refreshing state... [id=vpc-039076351a10e99ce]
module.dynamic_subnets.data.aws_vpc.default[0]: Refreshing state... [id=vpc-039076351a10e99ce]
module.vpc.aws_default_security_group.default[0]: Refreshing state... [id=sg-04bb988c6086039e6]
module.vpc.aws_internet_gateway.default[0]: Refreshing state... [id=igw-03e9ac32dab06d2e5]
module.dynamic_subnets.aws_subnet.private[0]: Refreshing state... [id=subnet-0a535c05e53db7d39]
module.dynamic_subnets.aws_subnet.private[2]: Refreshing state... [id=subnet-03542b2ddbd0e6c66]
module.dynamic_subnets.aws_route_table.public[0]: Refreshing state... [id=rtb-0d4aa01fce891d909]
module.dynamic_subnets.aws_subnet.public[1]: Refreshing state... [id=subnet-02b8ddffb4fa6333d]
module.dynamic_subnets.aws_subnet.private[1]: Refreshing state... [id=subnet-0858c11aa700b2c9b]
module.dynamic_subnets.aws_route_table.private[1]: Refreshing state... [id=rtb-0c2ee5f670fb66801]
module.dynamic_subnets.aws_subnet.public[0]: Refreshing state... [id=subnet-0eeba78e16c541109]
module.dynamic_subnets.aws_route_table.private[0]: Refreshing state... [id=rtb-090f9c3de1419df97]
module.dynamic_subnets.aws_route_table.private[2]: Refreshing state... [id=rtb-0209ab8fc6282e3d9]
module.dynamic_subnets.aws_subnet.public[2]: Refreshing state... [id=subnet-0059582536cdf24f3]
module.dynamic_subnets.aws_route.public[0]: Refreshing state... [id=r-rtb-0d4aa01fce891d9091080289494]
module.dynamic_subnets.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-0996301cdebd4143f]
module.dynamic_subnets.aws_nat_gateway.default[2]: Refreshing state... [id=nat-02d46720a0c3353cb]
module.dynamic_subnets.aws_nat_gateway.default[0]: Refreshing state... [id=nat-09f0d9beaddba0605]
module.dynamic_subnets.aws_route_table_association.public[2]: Refreshing state... [id=rtbassoc-08d7e55a18128aaba]
module.dynamic_subnets.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-04dbaebb5d0cb0699]
module.dynamic_subnets.aws_nat_gateway.default[1]: Refreshing state... [id=nat-0187993afba1dfb3b]
module.dynamic_subnets.aws_network_acl.public[0]: Refreshing state... [id=acl-09d95fe8bf98bbc01]
module.dynamic_subnets.aws_network_acl.private[0]: Refreshing state... [id=acl-004b96187c2bf5965]
module.dynamic_subnets.aws_route_table_association.private[2]: Refreshing state... [id=rtbassoc-0512545810fbab6dd]
module.dynamic_subnets.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-0baff6276d51bd578]
module.dynamic_subnets.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-00e4bfc4d05ade610]
module.dynamic_subnets.aws_route.default[1]: Refreshing state... [id=r-rtb-0c2ee5f670fb668011080289494]
module.dynamic_subnets.aws_route.default[2]: Refreshing state... [id=r-rtb-0209ab8fc6282e3d91080289494]
module.dynamic_subnets.aws_route.default[0]: Refreshing state... [id=r-rtb-090f9c3de1419df971080289494]
module.dynamic_subnets.aws_route_table_association.private[2]: Destroying... [id=rtbassoc-0512545810fbab6dd]
module.dynamic_subnets.aws_route_table_association.public[2]: Destroying... [id=rtbassoc-08d7e55a18128aaba]
module.dynamic_subnets.aws_subnet.public[2]: Creating...
module.dynamic_subnets.aws_route_table.private[2]: Modifying... [id=rtb-0209ab8fc6282e3d9]
module.dynamic_subnets.aws_eip.default[2]: Modifying... [id=eipalloc-05176125e443dedf1]
module.dynamic_subnets.aws_route_table_association.public[2]: Destruction complete after 0s
module.dynamic_subnets.aws_route_table_association.private[2]: Destruction complete after 0s
module.dynamic_subnets.aws_subnet.private[2]: Destroying... [id=subnet-03542b2ddbd0e6c66]
module.dynamic_subnets.aws_route_table.private[2]: Modifications complete after 1s [id=rtb-0209ab8fc6282e3d9]
module.dynamic_subnets.aws_eip.default[2]: Modifications complete after 1s [id=eipalloc-05176125e443dedf1]
module.dynamic_subnets.aws_subnet.private[2]: Destruction complete after 2s
module.dynamic_subnets.aws_subnet.private[2]: Creating...
module.dynamic_subnets.aws_subnet.private[2]: Creation complete after 1s [id=subnet-018d17ea596a3f190]
module.dynamic_subnets.aws_route_table_association.private[2]: Creating...
module.dynamic_subnets.aws_network_acl.private[0]: Modifying... [id=acl-004b96187c2bf5965]
module.dynamic_subnets.aws_route_table_association.private[2]: Creation complete after 1s [id=rtbassoc-0b5e2fb6c76916ce2]
module.dynamic_subnets.aws_network_acl.private[0]: Modifications complete after 2s [id=acl-004b96187c2bf5965]

Error: error creating subnet: InvalidSubnet.Conflict: The CIDR '10.207.128.0/20' conflicts with another subnet
	status code: 400, request id: f8b97b78-3f4c-4120-b06f-f8363b5e020e

It appears to not try to destroy the public subnet for the old 1c region.

Environment (please complete the following information):

Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/aws v3.26.0
+ provider registry.terraform.io/hashicorp/local v1.4.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/template v2.2.0

[Nuru]

This appears to be an issue with AWS's "eventual consistency" model of resource management. Although you can see the old subnet was deleted

module.dynamic_subnets.aws_subnet.private[2]: Destruction complete after 2s

the old CIDR is not released in time for it to be reallocated. Usually running plan and apply a second time resolves the issue. So this is more of an issue for Terraform and AWS than for this module specifically. However, if you find a solution we can implement, please let us know, or better yet open a PR that implements it.

[Nuru]

It turns out this was an unexpected and unintended side-effect of enabling create_before_destroy behavior on the NAT Gateways and NAT Instances, which rippled into create_before_destroy behavior of the subnets.

Fixed by #159