aws_cloudwatch_log_group creation intermittently fails with "already exists" error

[camlow325]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

› terraform -v
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.4.0

Affected Resource(s)

• aws_cloudwatch_log_group

Terraform Configuration Files

resource "aws_cloudwatch_log_group" "test" {
  name = "/aws/lambda/test"
}

Debug Output

The debug output shows that the provider appeared to encounter an error while waiting for a response to the initial creation attempt. During an attempted retry, an "already exists" error was received. This is presumably because AWS actually did create the log group during the first try.

2021-03-29T18:34:03.922Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2021/03/29 18:34:03 [DEBUG] [aws-sdk-go] DEBUG: Request logs/CreateLogGroup Details:
2021-03-29T18:34:03.922Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2021-03-29T18:34:03.922Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: POST / HTTP/1.1
…
2021-03-29T18:34:03.922Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: {"logGroupName":"/aws/lambda/test"}
...
2021-03-29T18:34:27.108Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2021/03/29 18:34:27 [DEBUG] [aws-sdk-go] DEBUG: Send Request logs/CreateLogGroup failed, attempt 0/25, error RequestError: send request failed
2021-03-29T18:34:27.108Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: caused by: Post "https://logs.us-west-2.amazonaws.com/": EOF
2021-03-29T18:34:27.168Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2021/03/29 18:34:27 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request logs/CreateLogGroup, attempt 1
2021-03-29T18:34:27.168Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2021/03/29 18:34:27 [DEBUG] [aws-sdk-go] DEBUG: Request logs/CreateLogGroup Details:
2021-03-29T18:34:27.168Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ REQUEST POST-SIGN ]-----------------------------
2021-03-29T18:34:27.168Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: POST / HTTP/1.1
…
2021-03-29T18:34:27.168Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: {"logGroupName":"/aws/lambda/test"}
2021-03-29T18:34:27.195Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2021/03/29 18:34:27 [DEBUG] [aws-sdk-go] DEBUG: Response logs/CreateLogGroup Details:
2021-03-29T18:34:27.195Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: ---[ RESPONSE ]--------------------------------------
2021-03-29T18:34:27.195Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: HTTP/1.1 400 Bad Request
...
2021-03-29T18:34:27.195Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: -----------------------------------------------------
2021-03-29T18:34:27.195Z [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: 2021/03/29 18:34:27 [DEBUG] [aws-sdk-go] {"__type":"ResourceAlreadyExistsException","message":"The specified log group already exists"}

Expected Behavior

The apply should have succeeded, with the log group having been created successfully.

Actual Behavior

Normally, the expected behavior occurs. Intermittently, though, the log group was created in AWS but the apply failed with an error:

aws_cloudwatch_log_group.test: Still creating... [20s elapsed]
...
Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists:  The CloudWatch Log Group '/aws/lambda/test' already exists.

Steps to Reproduce

1. terraform apply

[bryantbiggs]

Relates to #17839