aws_instance/volume_tags immedately out of date after creating new EC2 instance

[richard-browne]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

• Terraform 1.0.8
• hashicorp/aws v3.62.0

Affected Resource(s)

• aws_instance

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 3.62.0"
    }
  }
  required_version = "~> 1.0.8"
}

provider "aws" {
  region = "ap-southeast-2"
}

resource "aws_instance" "testvm" {
  ami = "ami-01b8916bb0ca18666"
  instance_type = "t3a.small"
  iam_instance_profile = "MyInstanceProfile"
  subnet_id = "subnet-06e1721e7aaaadea0"

  tags = {
    Name = "testvm"
  }

  root_block_device {
    tags = {
      Name = "testvm"
    }
  }

  user_data = "<script>echo Hello</script>"

  lifecycle { ignore_changes = [ami, user_data, volume_tags] }  
}

Expected Behavior

Running terraform apply a second time immediately after the first time, we expect Terraform to find nothing to do.

Actual Behavior

But when terraform apply is run for a second time, it claims the aws_instance is out of date and wants to change the aws_instance to set an empty volume_tags. See the following output:

aws_instance.testvm: Refreshing state... [id=i-087ea2a598da49904]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_instance.testvm will be updated in-place
  ~ resource "aws_instance" "testvm" {
        id                                   = "i-087ea2a598da49904"
        tags                                 = {
            "Name" = "testvm"
        }
      + volume_tags                          = {}
        # (28 unchanged attributes hidden)





        # (5 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Steps to Reproduce

1. terraform apply and observe the EC2 instance is created successfully with expected tags.
2. terraform apply again immediately and notice Terraform claims volume_tags are out of date

Important Factoids

• Although it seems unlikely, this problem occurs when user_data is given and user_data is in ignore_changes. Remove user_data, and the problem goes away.

• I note in the first .tfstate file, we see this line:

            "volume_tags":  null,

and after the second apply, it is changed to:

            "volume_tags": {},

• Note that the .tf file does not have a volume_tags at all.

• Adding volume_tags to ignore_changes does not fix the problem.

[justinretzolk]

Hey @richard-browne 👋 Thank you for taking the time to file this. This feels like a bug to me, so I'm going to tag it as such so that the team can take a look into it as time permits.

As a workaround, if you explicitly set volume_tags = {} in the configuration, does that prevent the issue?

[richard-browne]

Hi @justinretzolk yes is definitely a bug. No you can't add volume_tags = {} in this scenario, because volume_tags is mutually exclusive with root_block_device/tags (you can only use one or the other).

Next you will suggest use volume_tags instead of root_block_device/tags, and that would be ok for this example. But suppose you want different tags on root_block_device vs an additional ebs_block_device.

[ewbankkit]

Related: #20852.

[justinretzolk]

Thanks for the udpate @richard-browne; I'd not caught that those were mutually exclusive, but had considered that suggesting only using volume_tags would only really be useful in the event that you want similar tags, so glad to hear we were on the same page there! I've marked this one as a bug so that we can look into it as soon as time permits. Thank you again for reporting this!

[rba1-source]

This bug has been open for over a year, is there any progress on it? This is impacting our ability to see what changes will be made during our Production terraform apply.

If you have 200 deployed EC2 instances and are only updating something trivial like an EC2 security group, the terraform plan will show 201 resources to update. If we have more involved changes, we end up spending a long time verifying the plan is correct because of all the noise generated from this volume_tags bug.