aws_kms_grant does not fail if a grant by given name already exists

[mkielar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform: 0.13.3
AWS Provider: 3.3.0

Affected Resource(s)

• aws_kms_grant

Terraform Configuration Files

resource "aws_kms_grant" "bastion_grant" {
  count = var.enabled ? 1 : 0

  name               = "bastion-grant"
  key_id             = "arn:aws:kms:eu-west-1:${var.shared_account_id}:key/${var.packer_kms_key}"
  grantee_principal  = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
  retiring_principal = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${var.packer_kms_key_grant_retiring_principal_role}"
  operations         = ["Encrypt", "Decrypt", "ReEncryptFrom", "ReEncryptTo", "GenerateDataKey", "GenerateDataKeyWithoutPlaintext", "DescribeKey", "CreateGrant", "RetireGrant"]
  retire_on_delete   = true
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

Deployment of the second environment should fail, because the grant by the name bastion-grant already exists.

Actual Behavior

Each terraform workspace "thinks" it has it's own instance of KMS Grant resources. Then when I destroy one environment, the KMS Grant gets destroyed, and the other environment is unable to scale more EC2s.

Steps to Reproduce

1. Create new workspace with terraform workspace new one
2. terraform apply
3. Create new workspace with terraform workspace new two
4. terraform apply

Important Factoids

We're creating AWS environments for every Git Branch. Each environment gets a VPC and a Autoscaling Group with Bastion Host. The Bastion AMI is built with packer on another account and encrypted with CMK on that account (referenced as var.shared_account_id). On the "DEV" AWS Account - where all dev environments are created - we create a KMS Grant, so that ASG would be able to re-encrypt the EBS of the AMI.

Mistakenly, we did not prefix the grant name with the environment name, causing each environment to try to create KMS Grant named bastion-grant. This led to terraform on all environments thinking it has KMS Grant resource of its own, when in fact, AWS always returned the same grant. AWS did not fail, so Terraform did not fail. And put the same GrantID into each environment's statefile.

The problem arised when one of the Git Branchech was merged, and the dev environment destroyed. Because terraform "thinks" the KMS Grant was created especially for it, it deletes it, but because there was always only one Grant, all other dev environments are broken and unable to reencrypt bastion AMIs.

[dalgibbard]

Got bit by this when using terragrunt; moved one existing grant to a new def, and then changed the existing grant to a new IAM role -- there's a condition here where the parallel run from terragrunt results in the second grant being created, pointing at the same object, and then the first (original) grant runs it's modification; and this results in only a single grant being attached to the key, where two should be.

If it isn't already, the AWS KMS API should really be returning 409 Conflicts on identical/conflicting key grant requests, rather than blindly returning the existing grant ID!