SNS Topic delete/create failure #10473

Tirke · 2019-10-11T14:55:52Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.10

provider.aws v2.29.0

Affected Resource(s)

aws_sns_topic

Terraform Configuration Files

First config :

First file is module

resource "aws_sns_topic" "stream_in_sns_topic" {
  name = "another_sns_bug"
}

output "sns-topic-arn" {
  value = aws_sns_topic.stream_in_sns_topic.arn
}

Root

terraform {
  backend "s3" {
    bucket = "s3-states-dev-iot"
    key    = "sns-bug"
    region = "eu-west-1"
  }
}

provider "aws" {
  region = "eu-west-1"
}

module "sns" {
  source = "./module"
}

output "sns-topic-arn" {
  value = module.sns.sns-topic-arn
}

Second config :

terraform {
  backend "s3" {
    bucket = "s3-states-dev-iot"
    key    = "sns-bug"
    region = "eu-west-1"
  }
}
provider "aws" {
  region = "eu-west-1"
}

resource "aws_sns_topic" "stream_in_sns_topic" {
  name = "another_sns_bug"
}

output "sns-topic-arn" {
  value = aws_sns_topic.stream_in_sns_topic.arn
}

Debug Output

Expected Behavior

No error on sns_topic creation.

Actual Behavior

aws_sns_topic.stream_in_sns_topic: Creating...
module.sns.aws_sns_topic.stream_in_sns_topic: Destroying... [id=arn:aws:sns:eu-west-1:787685363335:another_sns_bug]
module.sns.aws_sns_topic.stream_in_sns_topic: Destruction complete after 0s

Error: Provider produced inconsistent result after apply

When applying changes to aws_sns_topic.stream_in_sns_topic, provider "aws"
produced an unexpected new value for was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Steps to Reproduce

terraform apply first config with the module
switch to the second config.
terraform apply -> failure

Important Factoids

Badly fixed on our side by using depends_on on the topic to introduce some delay.

References

I have another issue on aws_sns_topic #9645. With this new bug, I think that this resource really need some eyes on it, it's a bit broken ATM.

The text was updated successfully, but these errors were encountered:

Tirke · 2019-10-11T15:05:15Z

Looking at the trace log, terraform is creating the new topic then deleting it immediately after.
Maybe it all comes down to the two topics having the same name.

2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] DEBUG: Request sns/DeleteTopic Details:
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: POST / HTTP/1.1
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Host: sns.eu-west-1.amazonaws.com
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: User-Agent: aws-sdk-go/1.23.15 (go1.12.6; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Length: 107
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=ASIA3OZNUZ2DQLIS7XPL/20191011/eu-west-1/sns/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=c5ca311e09b7d71130a11292a12209d658709b5bf0753420e1c36b8de45c4eea
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Date: 20191011T145859Z
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Security-Token: FQoGZXIvYXdzEPn//////////wEaDG2O0ono84+bg2vDXCKnAl1qvfM5eMcP8Kixw6ZinH5GmWlRwo+arPtTiarA+C+59hrPWEYcKNYR/3z5hhUO738FSQQXyXxBVcQqkG/4ZuSjL0DEP/Jg+LfXhsm1D3ajvdG5BYNBVausn5SWjP5Rayoy62vbTZ1GLmbsNjnwGfxv3ZuwlYttK6SkW5qChMSkoYHO1kl4k7wmYMDSTA28UY5MDK3jZGWnltFGxlHVDK0CP6Q3fPHZWHjVuWj2gd9iy2sUjuGbNeC08hvGm2laKUzisEkTcIf6Be+Y8nsO5adCOh5KpLifHx5sRLs+vnw+6y2sd75JB54igQRI/OIsqlj8oj3IRc8FrxEc0CvmIc/WTBnwtdob3tlcNGhyB4M24mnI8LpnHmmhsYL+apb1XHN/dVOsaIYo9+eA7QU=
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Accept-Encoding: gzip
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Action=DeleteTopic&TopicArn=arn%3Aaws%3Asns%3Aeu-west-1%3A787685363335%3Aanother_sns_bug&Version=2010-03-31
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------
2019-10-11T16:58:59.427+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] SNS create topic: another_sns_bug
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] DEBUG: Request sns/CreateTopic Details:
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: POST / HTTP/1.1
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Host: sns.eu-west-1.amazonaws.com
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: User-Agent: aws-sdk-go/1.23.15 (go1.12.6; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Length: 64
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=ASIA3OZNUZ2DQLIS7XPL/20191011/eu-west-1/sns/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=f082559290caeb72ed60fca8b940b635de76b1a26f87be397626d69b8cc70b84
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Date: 20191011T145859Z
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Security-Token: FQoGZXIvYXdzEPn//////////wEaDG2O0ono84+bg2vDXCKnAl1qvfM5eMcP8Kixw6ZinH5GmWlRwo+arPtTiarA+C+59hrPWEYcKNYR/3z5hhUO738FSQQXyXxBVcQqkG/4ZuSjL0DEP/Jg+LfXhsm1D3ajvdG5BYNBVausn5SWjP5Rayoy62vbTZ1GLmbsNjnwGfxv3ZuwlYttK6SkW5qChMSkoYHO1kl4k7wmYMDSTA28UY5MDK3jZGWnltFGxlHVDK0CP6Q3fPHZWHjVuWj2gd9iy2sUjuGbNeC08hvGm2laKUzisEkTcIf6Be+Y8nsO5adCOh5KpLifHx5sRLs+vnw+6y2sd75JB54igQRI/OIsqlj8oj3IRc8FrxEc0CvmIc/WTBnwtdob3tlcNGhyB4M24mnI8LpnHmmhsYL+apb1XHN/dVOsaIYo9+eA7QU=
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Accept-Encoding: gzip
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Action=CreateTopic&Name=another_sns_bug&Tags=&Version=2010-03-31
2019-10-11T16:58:59.428+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] DEBUG: Response sns/CreateTopic Details:
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ RESPONSE ]--------------------------------------
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: HTTP/1.1 200 OK
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Connection: close
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Length: 322
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Type: text/xml
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Date: Fri, 11 Oct 2019 14:58:58 GMT
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amzn-Requestid: 64c4bec6-026c-5d8a-afc4-e8dc989fad30
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] <CreateTopicResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:   <CreateTopicResult>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:     <TopicArn>arn:aws:sns:eu-west-1:787685363335:another_sns_bug</TopicArn>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:   </CreateTopicResult>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:   <ResponseMetadata>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:     <RequestId>64c4bec6-026c-5d8a-afc4-e8dc989fad30</RequestId>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:   </ResponseMetadata>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: </CreateTopicResponse>
2019-10-11T16:58:59.603+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] Reading SNS Topic Attributes for arn:aws:sns:eu-west-1:787685363335:another_sns_bug
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] DEBUG: Request sns/GetTopicAttributes Details:
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: POST / HTTP/1.1
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Host: sns.eu-west-1.amazonaws.com
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: User-Agent: aws-sdk-go/1.23.15 (go1.12.6; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Length: 114
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=ASIA3OZNUZ2DQLIS7XPL/20191011/eu-west-1/sns/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=c5b2b66344f454d489078cd2af2854faf70f14177d3d5636fb74e4dbf953d2b3
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Date: 20191011T145859Z
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amz-Security-Token: FQoGZXIvYXdzEPn//////////wEaDG2O0ono84+bg2vDXCKnAl1qvfM5eMcP8Kixw6ZinH5GmWlRwo+arPtTiarA+C+59hrPWEYcKNYR/3z5hhUO738FSQQXyXxBVcQqkG/4ZuSjL0DEP/Jg+LfXhsm1D3ajvdG5BYNBVausn5SWjP5Rayoy62vbTZ1GLmbsNjnwGfxv3ZuwlYttK6SkW5qChMSkoYHO1kl4k7wmYMDSTA28UY5MDK3jZGWnltFGxlHVDK0CP6Q3fPHZWHjVuWj2gd9iy2sUjuGbNeC08hvGm2laKUzisEkTcIf6Be+Y8nsO5adCOh5KpLifHx5sRLs+vnw+6y2sd75JB54igQRI/OIsqlj8oj3IRc8FrxEc0CvmIc/WTBnwtdob3tlcNGhyB4M24mnI8LpnHmmhsYL+apb1XHN/dVOsaIYo9+eA7QU=
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Accept-Encoding: gzip
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Action=GetTopicAttributes&TopicArn=arn%3Aaws%3Asns%3Aeu-west-1%3A787685363335%3Aanother_sns_bug&Version=2010-03-31
2019-10-11T16:58:59.604+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] DEBUG: Response sns/DeleteTopic Details:
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ RESPONSE ]--------------------------------------
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: HTTP/1.1 200 OK
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Connection: close
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Length: 201
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Content-Type: text/xml
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: Date: Fri, 11 Oct 2019 14:58:59 GMT
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: X-Amzn-Requestid: 481e7f59-a047-567f-acca-29433ab4eac3
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2019/10/11 16:58:59 [DEBUG] [aws-sdk-go] <DeleteTopicResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:   <ResponseMetadata>
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:     <RequestId>481e7f59-a047-567f-acca-29433ab4eac3</RequestId>
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4:   </ResponseMetadata>
2019-10-11T16:58:59.616+0200 [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: </DeleteTopicResponse>

ChernikovP · 2019-11-19T14:45:21Z

The same problem (the same sequence of log events) still exists with

terraform version: 0.12.12
terraform-provider-aws version 2.37.0

@Tirke , have you found a workaround to avoid this failure? As it looks like rerunning apply gracefully recreates topic, as there's nothing to destroy at the 2nd apply run.

Tirke · 2019-11-19T20:12:28Z

Hello @ChernikovP ,

Yes we wrote a script to check and repair (if needed) all our SNS topics and all our SQS subscriptions using the AWS SDK. Terraform is very buggy around SNS and SNS subscriptions.
As you said a second apply may work but in some specific cases it won't. And this is not an ideal workaround because Terraform very often fails silently around this. What I mean is we had SNS topics and SQS subs deleted without being re-created and without any warning or error. And when you manage a lot of resources it can be a real pain.
This bite us twice on production infrastructure and had an impact on our clients so we don't even trust Terraform on this anymore.

ewbankkit · 2020-10-22T19:09:54Z

@Tirke Thanks for raising this issue.
As the new resource has a new name, Terraform will treat the old and new resources as being independent and will act on them in parallel, destroying the old SNS topic and creating the new in parallel and effectively a race condition is being hit.
See https://discuss.hashicorp.com/t/destroy-before-create/3980 for some details.
The way I would approach such a resource renaming would be to use the terraform state mv CLI command to rename the resource in state and then rename in the HCL code.
See also #14394 for a discussion of other thought around resource uniqueness.

ghost · 2020-11-22T17:11:12Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Oct 11, 2019

nywilken added the service/sns Issues and PRs that pertain to the sns service. label Oct 11, 2019

ChernikovP mentioned this issue Nov 19, 2019

SNS Topic Subscription delete/create failure when resource name is changed #10930

Closed

jochen42 mentioned this issue Jan 23, 2020

aws_sns_topic_subscription resource produced new value for was present but now absent #11737

Closed

ewbankkit mentioned this issue Oct 22, 2020

Plan-Time Resource Uniqueness Errors #14394

Open

ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Oct 22, 2020

ewbankkit closed this as completed Oct 22, 2020

ghost locked as resolved and limited conversation to collaborators Nov 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNS Topic delete/create failure #10473

SNS Topic delete/create failure #10473

Tirke commented Oct 11, 2019 •

edited

Tirke commented Oct 11, 2019 •

edited

ChernikovP commented Nov 19, 2019

Tirke commented Nov 19, 2019 •

edited

ewbankkit commented Oct 22, 2020

ghost commented Nov 22, 2020

SNS Topic delete/create failure #10473

SNS Topic delete/create failure #10473

Comments

Tirke commented Oct 11, 2019 • edited

Community Note

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

Tirke commented Oct 11, 2019 • edited

ChernikovP commented Nov 19, 2019

Tirke commented Nov 19, 2019 • edited

ewbankkit commented Oct 22, 2020

ghost commented Nov 22, 2020

Tirke commented Oct 11, 2019 •

edited

Tirke commented Oct 11, 2019 •

edited

Tirke commented Nov 19, 2019 •

edited