Missing AWS SNS subscriptions after state move

[exawolve]

Missing AWS SNS subscriptions after state move

AWS SNS might result in some SNS subscriptions missing, after moving state and recreating resources.

terraform should make sure resources are deleted before recreating, but seems to end up in a race condition or something occasionally.

Strange about this, is also that an initial terraform plan states the infrastructure to be in sync.
When doing a follow up terraform plan the missing resources show up, though.

See this git repo to reproduce this issue.

description

Given the following SQS/SNS infrastructure:

resource "aws_sns_topic" "test-topic" {
  count = length(var.event_names)
  name  = var.event_names[count.index]
}

resource "aws_sqs_queue" "test-event-queue" {
  name = "test-event-queue"
}

resource "aws_sns_topic_subscription" "sns-subscriptions" {
  count = length(var.event_names)
  topic_arn = format("arn:aws:sns:%s:%s:%s", data.aws_region.current.name, data.aws_caller_identity.current.account_id, var.event_names[count.index])
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.test-event-queue.arn
}

---

When switching to the following module based infrastructure:

resource "aws_sns_topic" "test-topic" {
  count = length(var.event_names)
  name  = var.event_names[count.index]
}

module "sqs" {
  source = "./module"
  event_names = var.event_names
  queue-name = "test-event-queue"
}

with the ./module:

resource "aws_sqs_queue" "test-event-queue" {
  name = var.queue-name
}

resource "aws_sns_topic_subscription" "sns-subscriptions" {
  for_each  = toset(var.event_names)
  topic_arn = format("arn:aws:sns:%s:%d:%s", data.aws_region.current.name, data.aws_caller_identity.current.account_id, each.key)
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.test-event-queue.arn
}

---

and moving state like:

terraform state mv aws_sqs_queue.test-event-queue module.sqs.aws_sqs_queue.test-event-queue -no-color
for i in {0..2}; do terraform state mv aws_sns_topic_subscription.sns-subscriptions[$i] module.sqs.aws_sns_topic_subscription.sns-subscriptions[$i] -no-color; done

you will end up occasionally with the SNS subscriptions missing.

(another issue here as a side note:
moving state will result in new subscriptions being recreated, because each in aws_sns_topic_subscription type is changed from list to map.)

This will usually show up only, when doing a terraform plan for the 2nd time...

Reproduce

See the script run.sh to reproduce.

Yet TODO: provide some terraform S3 backend:

terraform {
  backend "s3" {
    [...]
  }
}

Output from run.sh script run

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "template" (hashicorp/template) 2.2.0...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
data.aws_caller_identity.current: Refreshing state...
data.aws_region.current: Refreshing state...
aws_sns_topic.test-topic[1]: Creating...
aws_sqs_queue.test-event-queue: Creating...
aws_sns_topic.test-topic[2]: Creating...
aws_sns_topic.test-topic[0]: Creating...
aws_sns_topic.test-topic[1]: Creation complete after 0s id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2]
aws_sns_topic.test-topic[2]: Creation complete after 0s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3]
aws_sns_topic.test-topic[0]: Creation complete after 0s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1]
aws_sqs_queue.test-event-queue: Creation complete after 0s [id=https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxx/test-event-queue]
aws_sns_topic_subscription.sns-subscriptions[0]: Creating...
aws_sns_topic_subscription.sns-subscriptions[2]: Creating...
aws_sns_topic_subscription.sns-subscriptions[1]: Creating...
aws_sns_topic_subscription.sns-subscriptions[2]: Creation complete after 1s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3:0bb667d2-04ec-4794-a40c-7089a0a78118]
aws_sns_topic_subscription.sns-subscriptions[1]: Creation complete after 1s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2:4ae5580d-24eb-430d-9bca-72daf8079e8b]
aws_sns_topic_subscription.sns-subscriptions[0]: Creation complete after 1s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1:bc529eef-8bfc-404e-b846-f9677ebd9ede]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Initializing modules...
- sqs in module

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "template" (hashicorp/template) 2.2.0...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Move "aws_sqs_queue.test-event-queue" to "module.sqs.aws_sqs_queue.test-event-queue"
Successfully moved 1 object(s).
Move "aws_sns_topic_subscription.sns-subscriptions[0]" to "module.sqs.aws_sns_topic_subscription.sns-subscriptions[0]"
Successfully moved 1 object(s).
Move "aws_sns_topic_subscription.sns-subscriptions[1]" to "module.sqs.aws_sns_topic_subscription.sns-subscriptions[1]"
Successfully moved 1 object(s).
Move "aws_sns_topic_subscription.sns-subscriptions[2]" to "module.sqs.aws_sns_topic_subscription.sns-subscriptions[2]"
Successfully moved 1 object(s).
data.aws_caller_identity.current: Refreshing state...
module.sqs.data.aws_caller_identity.current: Refreshing state...
module.sqs.aws_sqs_queue.test-event-queue: Refreshing state... [id=https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxx/test-event-queue]
module.sqs.data.aws_region.current: Refreshing state...
data.aws_region.current: Refreshing state...
aws_sns_topic.test-topic[0]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1]
aws_sns_topic.test-topic[2]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3]
aws_sns_topic.test-topic[1]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2]
module.sqs.aws_sns_topic_subscription.sns-subscriptions: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1:bc529eef-8bfc-404e-b846-f9677ebd9ede]
module.sqs.aws_sns_topic_subscription.sns-subscriptions[1]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2:4ae5580d-24eb-430d-9bca-72daf8079e8b]
module.sqs.aws_sns_topic_subscription.sns-subscriptions[2]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3:0bb667d2-04ec-4794-a40c-7089a0a78118]
module.sqs.aws_sns_topic_subscription.sns-subscriptions[2]: Destroying... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3:0bb667d2-04ec-4794-a40c-7089a0a78118]
module.sqs.aws_sns_topic_subscription.sns-subscriptions: Destroying... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1:bc529eef-8bfc-404e-b846-f9677ebd9ede]
module.sqs.aws_sns_topic_subscription.sns-subscriptions[1]: Destroying... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2:4ae5580d-24eb-430d-9bca-72daf8079e8b]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e2"]: Creating...
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e1"]: Creating...
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e3"]: Creating...
module.sqs.aws_sns_topic_subscription.sns-subscriptions: Destruction complete after 1s
module.sqs.aws_sns_topic_subscription.sns-subscriptions[2]: Destruction complete after 1s
module.sqs.aws_sns_topic_subscription.sns-subscriptions[1]: Destruction complete after 1s
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e1"]: Creation complete after 1s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1:bc529eef-8bfc-404e-b846-f9677ebd9ede]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e2"]: Creation complete after 1s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2:4ae5580d-24eb-430d-9bca-72daf8079e8b]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e3"]: Creation complete after 1s [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3:1d42d9bd-f120-448e-ae73-a5b4345d6fa7]

Apply complete! Resources: 3 added, 0 changed, 3 destroyed.

Initializing modules...
- sqs in module

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "template" (hashicorp/template) 2.2.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.aws_region.current: Refreshing state...
aws_sns_topic.test-topic[0]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1]
data.aws_caller_identity.current: Refreshing state...
module.sqs.data.aws_caller_identity.current: Refreshing state...
module.sqs.data.aws_region.current: Refreshing state...
aws_sns_topic.test-topic[2]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3]
aws_sns_topic.test-topic[1]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2]
module.sqs.aws_sqs_queue.test-event-queue: Refreshing state... [id=https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxx/test-event-queue]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e3"]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3:1d42d9bd-f120-448e-ae73-a5b4345d6fa7]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e1"]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1:bc529eef-8bfc-404e-b846-f9677ebd9ede]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e2"]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2:4ae5580d-24eb-430d-9bca-72daf8079e8b]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

running plan one more time...

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_sns_topic.test-topic[1]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2]
data.aws_region.current: Refreshing state...
data.aws_caller_identity.current: Refreshing state...
aws_sns_topic.test-topic[2]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3]
aws_sns_topic.test-topic[0]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1]
module.sqs.data.aws_caller_identity.current: Refreshing state...
module.sqs.data.aws_region.current: Refreshing state...
module.sqs.aws_sqs_queue.test-event-queue: Refreshing state... [id=https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxx/test-event-queue]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e2"]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2:4ae5580d-24eb-430d-9bca-72daf8079e8b]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e1"]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1:bc529eef-8bfc-404e-b846-f9677ebd9ede]
module.sqs.aws_sns_topic_subscription.sns-subscriptions["e3"]: Refreshing state... [id=arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e3:1d42d9bd-f120-448e-ae73-a5b4345d6fa7]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.sqs.aws_sns_topic_subscription.sns-subscriptions["e1"] will be created
  + resource "aws_sns_topic_subscription" "sns-subscriptions" {
      + arn                             = (known after apply)
      + confirmation_timeout_in_minutes = 1
      + endpoint                        = "arn:aws:sqs:eu-central-1:xxxxxxxxxxxx:test-event-queue"
      + endpoint_auto_confirms          = false
      + id                              = (known after apply)
      + protocol                        = "sqs"
      + raw_message_delivery            = false
      + topic_arn                       = "arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e1"
    }

  # module.sqs.aws_sns_topic_subscription.sns-subscriptions["e2"] will be created
  + resource "aws_sns_topic_subscription" "sns-subscriptions" {
      + arn                             = (known after apply)
      + confirmation_timeout_in_minutes = 1
      + endpoint                        = "arn:aws:sqs:eu-central-1:xxxxxxxxxxxx:test-event-queue"
      + endpoint_auto_confirms          = false
      + id                              = (known after apply)
      + protocol                        = "sqs"
      + raw_message_delivery            = false
      + topic_arn                       = "arn:aws:sns:eu-central-1:xxxxxxxxxxxx:e2"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.