Bug with aws_eip while using count

[madhurranjan]

Hi,
Below is my terraform file.

provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region = "us-west-2"
}

resource "aws_instance" "test" {
  count = "2"
  ami = "ami-20927640"
  key_name = "<keyname>"
  instance_type = "m3.medium"
  availability_zone = "us-west-2a"
  vpc_security_group_ids = ["sg-<>"]
  subnet_id = "subnet-<fo>"
  tenancy = "default"
  tags {
    Name = "test-${count.index + 1}"
    application = "foo"
  }

}

resource "aws_eip" "eip" {
    count = "2"
    instance = "${element(aws_instance.test.*.id,count.index)}"
    vpc = true
    depends_on = ["aws_instance.test"]
}

After increasing the count from 1 to 2 (Basically anytime I change the count ) , I hit the below error. Running terraform apply second time works but this seems to be a bug:

aws_instance.test.0: Refreshing state... (ID: i-2dac44f5)
aws_eip.eip.0: Refreshing state... (ID: eipalloc-2dc5d348)
aws_instance.test.1: Creating...
  ami:                               "" => "ami-20927640"
  availability_zone:                 "" => "us-west-2a"
  ebs_block_device.#:                "" => "<computed>"
  ephemeral_block_device.#:          "" => "<computed>"
  instance_type:                     "" => "m3.medium"
  key_name:                          "" => "<key>"
  placement_group:                   "" => "<computed>"
  private_dns:                       "" => "<computed>"
  private_ip:                        "" => "<computed>"
  public_dns:                        "" => "<computed>"
  public_ip:                         "" => "<computed>"
  root_block_device.#:               "" => "<computed>"
  security_groups.#:                 "" => "<computed>"
  source_dest_check:                 "" => "1"
  subnet_id:                         "" => "subnet-yo"
  tags.#:                            "" => "2"
  tags.Name:                         "" => "test-2"
  tags.application:                  "" => "foo"
  tenancy:                           "" => "default"
  vpc_security_group_ids.#:          "" => "1"
  vpc_security_group_ids.2786422505: "" => "<sg>"
aws_instance.test.1: Creation complete
aws_eip.eip.1: Creating...
  allocation_id:     "" => "<computed>"
  association_id:    "" => "<computed>"
  domain:            "" => "<computed>"
  instance:          "" => "i-b4af476c"
  network_interface: "" => "<computed>"
  private_ip:        "" => "<computed>"
  public_ip:         "" => "<computed>"
  vpc:               "" => "1"
aws_eip.eip.1: Creation complete
Error applying plan:

1 error(s) occurred:

* aws_eip.eip.0: diffs didn't match during apply. This is a bug with Terraform and should be reported.

[ElliotG]

We might have a different problem, but we are also seeing a race condition related to multiple EIP's and a nat_gateway. Without posting the whole file, the thing we are seeing is:

-Create a mini VPC with 4 public/private subnets
-Create 4 EIPs
-Create 4 NAT's, linked to the 4 EIPs through count.

Bug: the 4 NAT's are all given the same EIP
Workaround: Add the EIP set as a direct dependency.

If needed, I could add the statefile. For now, seems like there's an issue related to EIP+Count. Will wait to see if the above is confirmed as a bug.

[diroussel]

@ElliotG Can you provide a cut down example?

[diroussel]

This is how I'm configuring multiple EIPs, seems to work.

resource "aws_eip" "nat" {
  vpc = true
  count = "${length(split(",", var.private_subnet_cidr))}"
}

resource "aws_nat_gateway" "gw" {
  allocation_id = "${element(aws_eip.nat.*.id, count.index)}"

  count = "${length(split(",", var.private_subnet_cidr))}"
  subnet_id = "${element(split(",", module.vpc.private_subnets), count.index)}"
}

[ElliotG]

resource "aws_eip" "nat_gateway" {
    vpc = true
    count = "${length(split(",", var.azs))}"
}

resource "aws_nat_gateway" "gateway" {
    count = "${length(split(",", var.azs))}"
    subnet_id = "${element(aws_subnet.public-subnet.*.id, count.index)}"
    allocation_id = "${element(aws_eip.nat_gateway.*.id, count.index)}"
}

For whatever reason, we saw this behavior (on Terraform 6.8 or 6.9, i can't remember, we have since upgraded). We were able to trivially fix it with this:

depends_on = ["aws_internet_gateway.gw", "aws_eip.nat_gateway"]

[Gary-Armstrong]

I'm seeing this today when I tried to create a couple new instances using -var compute_count=3 when the default is 1. Normally I'd expect to see 2 new instances spring to life. Today I get something else:

    Terraform Version: 0.6.12
    Resource ID: aws_eip.wxmix_compute_eip.0
    Mismatch reason: attribute mismatch: instance
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"instance":*terraform.ResourceAttrDiff{Old:"i-0fc2238c", New:"${element(aws_instance.compute.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

I did find it interesting that TF wanted to make changes to the EIP on the existing instance despite there being no actual change to the code other than the count variable:

~ aws_eip.wxmix_compute_eip.0
    instance: "i-0fc2238c" => "${element(aws_instance.compute.*.id, count.index)}"

I think it may be related to starting off where count = 1 being entered into state like so:

aws_instance.compute: Refreshing state... (ID: i-0fc2238c)

And so there isn't a aws_instance-compute.0 while count = 1.

Managing instances by manipulating the count via CLI has worked brilliantly up to this point.

[ArnaudBrousseau]

This just happened to me. Here's the relevant code:

resource "aws_eip" "REDACTED_GROUP_NAME_eip" {
  count = "${var.count}"
  instance = "${element(REDACTED_GROUP_NAME.REDACTED_INSTANCE_NAME.*.id, count.index)}"
  vpc = true
}

Path to error:

• attached EIPs to 2 (existing) instances by adding code above
• checked on AWS console, everything looked good
• lowered the instance count from 2 to 1
• at this point, in AWS console I was able to see the terminated instance (correct)
• bumped the count from 1 to 2: make apply => error!

The exact error:

* aws_eip.railgun_eip.0: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.6.16
    Resource ID: aws_eip.REDACTED_NAME.0
    Mismatch reason: attribute mismatch: instance
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"instance":*terraform.ResourceAttrDiff{Old:"i-2a582ab7", New:"${element(yelpaws_instance.REDACTED_NAME.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Worth noting that

• in AWS console I was able to see the terminated instance, and 2 instances with elastic IPs (what I expected). Things ran fine despite this error?
• make apply succeeded the second time

[Gary-Armstrong]

@ArnaudBrousseau I concur that so far a second apply does bring state into consistency and all is well. I didn't actually check the console to verify that the process was successful, but historically the AWS functions do complete while TF itself trips over it's state management.

[hsergei]

Try adding

  lifecycle {
    ignore_changes = [ "instance" ]
  }

to the aws_eip definition. Seem to work OK so far.

[Gary-Armstrong]

Just for the record, I have

  lifecycle {
    prevent_destroy = true
    ignore_changes = ["aws_instance"]
  }

inside each aws_eip def and I still need to apply twice.

[hsergei]

@Gary-Armstrong I think you might want try remove aws part from aws_instance. Although in my case it was not that I had to apply twice. It was this error

Please include the following information in your report:

    Terraform Version: 0.7.3
    Resource ID: aws_eip.my_instance.0
    Mismatch reason: attribute mismatch: instance

[mitchellh]

Hello! This should be fixed in later versions of Terraform (0.7.11 at time of writing).

Its hard to know 100% for a couple reasons:

• This is on a now much-older version of Terraform. This is our fault, not yours, for not trying sooner, but we've fixed many diff mismatch issues since then.
• We've fixed issues with exact error messages as this in those versions.

Please try again! If the issue persists please just open a new issue. Thanks!

[OlivierCuyp]

Hi,

I still have the issue when incrementing the count of instance existing eip associations are re-created.

Here is the plan log :

-/+ aws_eip_association.io-mvstud-eip-assoc-apps-production.1
    allocation_id:        "eipalloc-fbc22b9c" => "eipalloc-fbc22b9c"
    instance_id:          "i-06386f27f8c161946" => "${element(aws_instance.io-mvstud-ec2-apps-production.*.id, count.index)}" (forces new resource)
    network_interface_id: "eni-c310f483" => "<computed>"

I basically have a pool of eip that I associate with ec2 instances.
Here is my code :

# EC2
resource "aws_instance" "io-mvstud-ec2-apps-production" {
    count = "${var.instances_production}"
    # ...
}
# EC2: EIP
resource "aws_eip" "io-mvstud-eip-apps-production" {
    count = "${var.elastic_ips_production}"
    vpc = true
}
# EC2: EIP Association
resource "aws_eip_association" "io-mvstud-eip-assoc-apps-production" {
    count = "${var.instances_production}"
    instance_id = "${element(aws_instance.io-mvstud-ec2-apps-production.*.id, count.index)}"
    allocation_id = "${element(aws_eip.io-mvstud-eip-apps-production.*.id, count.index)}"
}

I don't understand why terraform can't keep the existing association when incrementing the var.instances_production ...

[ZachMealing]

I am also seeing an issue where Elastic IP address are being recreated when using count when I run has any one found a solution for this yet?

  vpc        = true
  count      = "${var.webserver_count}"
  instance   = "${element(aws_instance.webserver.*.id, count.index)}"
  depends_on = ["aws_instance.webserver"]
}

[jerrywardlow]

When using a similar pattern, I also encounter an issue where Terraform assumes the EIP resource should be modified instead of correctly interpolating the instance ID.

instance = "${element(aws_instance.pdfs.*.id, count.index)}"

Terraform will perform the following actions:

  ~ module.test.aws_eip.test[0]
      instance:                     "i-05e99<redacted>" => "${element(aws_instance.test.*.id, count.index)}"

[jsnod]

I'm seeing this with NAT Gateways too when using an interpolated allocation_id. If I increase the nat_gateway_count TF destroys the existing NAT Gateway and then recreates it with the same EIP. This causes a Black Hole in the route table, and I need to manually update the route table in AWS console (our route tables aren't in TF... yet!). See below:

resource "aws_eip" "nat_gateway_eip" {
  count = "${var.nat_gateway_count}"

  vpc = "true"

  tags {
    Name        = "${var.name_short}-nat-gateway-${count.index}"
    Terraform   = "true"
    Environment = "${terraform.workspace}"
  }
}

resource "aws_nat_gateway" "gw" {
  count = "${var.nat_gateway_count}"

  depends_on    = ["aws_internet_gateway.this"]
  allocation_id = "${element(aws_eip.nat_gateway_eip.*.id, count.index)}"
  subnet_id     = "${element(aws_subnet.public.*.id, count.index)}"

  tags {
    Name        = "${var.name_short}-nat-gateway-${count.index}"
    Terraform   = "true"
    Environment = "${terraform.workspace}"
  }
}

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.