Crash while destroying state

[aaron-lane]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.15.4
AWS Provider v3.44.0

Affected Resource(s)

Resources reporting Error: Plugin did not respond:

• aws_security_group
• aws_security_group_rule
• aws_s3_bucket_public_access_block
• aws_iam_role_policy

Resources reporting Error: Request cancelled:

• aws_security_group_rule
• aws_iam_role_policy_attachment
• aws_instance
• aws_kms_alias
• aws_lb_target_group
• aws_route53_record
• aws_s3_bucket_public_access_block
• aws_iam_role_policy
• aws_launch_configuration

Terraform Configuration Files

https://github.com/hashicorp/terraform-aws-terraform-enterprise/tree/master/tests/private-tcp-active-active

Panic Output

https://gist.github.com/aaron-lane/c5b4f37486b94fc3293a33b95e573c23

Expected Behavior

Terraform should have successfully destroyed resources.

Actual Behavior

Terraform crashed while destroying resources.

Steps to Reproduce

1. terraform apply successfully:

• tests create network terraform-aws-terraform-enterprise#181 (comment)
• https://github.com/hashicorp/terraform-aws-terraform-enterprise/runs/2786266696?check_suite_focus=true
• https://app.terraform.io/app/terraform-enterprise-modules-test/workspaces/aws-private-tcp-active-active/runs/run-m7MX5ZSrLeZANaJq

2. terraform apply unsuccessfully against the same state with different inputs for resource names:

• randomize key alias terraform-aws-terraform-enterprise#182 (comment)
• https://github.com/hashicorp/terraform-aws-terraform-enterprise/runs/2786782810?check_suite_focus=true
• https://app.terraform.io/app/terraform-enterprise-modules-test/workspaces/aws-private-tcp-active-active/runs/run-CrKfkHgb87G262v2

3. terraform destroy unsuccessfully:

• https://app.terraform.io/app/terraform-enterprise-modules-test/workspaces/aws-private-tcp-active-active/runs/run-pXu3AnmLagqxx2FL

Important Factoids

The applies were triggered by GitHub Actions and used a TFC workspace as the remote backend. The destroy was triggered manually via TFC. Access can be granted to the workspace as necessary.

[ewbankkit]

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 67 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsSecurityGroupRead(0xc001314e80, 0x0, 0x0, 0xb3de340, 0xc000075000)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_security_group.go:346 +0x15df
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc000b928c0, 0x7c2df68, 0xc00128b600, 0xc001314e80, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000b928c0, 0x7c2df68, 0xc00128b600, 0xc000558540, 0x0, 0x0, 0xc00000f080, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000122dc8, 0x7c2df68, 0xc00128b600, 0xc00128bf80, 0xc00128b600, 0x40b965, 0x6bfa020)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc0014f7e20, 0x7c2e010, 0xc00128b600, 0xc0010388a0, 0xc0014f7e20, 0xc00122fc50, 0xc000f54ba0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x6ec9000, 0xc0014f7e20, 0x7c2e010, 0xc00122fc50, 0xc001038840, 0x0, 0x7c2e010, 0xc00122fc50, 0xc000061c00, 0x325)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000cd340, 0x7c4f918, 0xc00154c600, 0xc001a45b00, 0xc000a88fc0, 0xb39d690, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0000cd340, 0x7c4f918, 0xc00154c600, 0xc001a45b00, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000ace300, 0xc0000cd340, 0x7c4f918, 0xc00154c600, 0xc001a45b00)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Similar (same panic message interface conversion: interface {} is nil, not *aws.AWSClient): #19601.

[aaron-lane]

We found that manually deleting the resources which reported that the plugin did not respond was a workaround to resolve the issue.

[anGie44]

Hi @aaron-lane , thank you for reporting this issue. Do you mind providing access to the workspace to see if there's anything we can find in the logs (if that's also viewabled?) when the destroy was run? Our team has noticed this crash before when authentication fails, so maybe worth looking around for signs of that as well

[elisedaurio]

Experiencing a similar issue during a destroy using version 0.15.5

[DenisBY]

The similar with version 3.51.0

│ Error: Plugin did not respond
│ 
│   with module.proxy.module.asg.aws_launch_configuration.this[0],
│   on .terraform/modules/proxy.asg/main.tf line 4, in resource "aws_launch_configuration" "this":
│    4: resource "aws_launch_configuration" "this" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.proxy.module.asg.aws_launch_configuration.this[0],
│   on .terraform/modules/proxy.asg/main.tf line 4, in resource "aws_launch_configuration" "this":
│    4: resource "aws_launch_configuration" "this" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.proxy.aws_vpc_endpoint_service.vpc_endpoint_service,
│   on .terraform/modules/proxy/proxy-asg/endpoint.tf line 1, in resource "aws_vpc_endpoint_service" "vpc_endpoint_service":
│    1: resource "aws_vpc_endpoint_service" "vpc_endpoint_service" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.proxy.aws_security_group.security_group_endpoint,
│   on .terraform/modules/proxy/proxy-asg/endpoint.tf line 51, in resource "aws_security_group" "security_group_endpoint":
│   51: resource "aws_security_group" "security_group_endpoint" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain more details.
╵
Releasing state lock. This may take a few moments...

Stack trace from the terraform-provider-aws_v3.51.0_x5 plugin:

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 65 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsLaunchConfigurationRead(0xc0000bad00, 0x0, 0x0, 0xb6b71c0, 0xc000088800)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_launch_configuration.go:562 +0x12b8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0006ef960, 0x7e14c08, 0xc0021637c0, 0xc0000bad00, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.0.2-0.20210714181518-b5a3dc95a675/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0006ef960, 0x7e14c08, 0xc0021637c0, 0xc002524690, 0x0, 0x0, 0xc00000e658, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.0.2-0.20210714181518-b5a3dc95a675/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00000c7b0, 0x7e14c08, 0xc0021637c0, 0xc002163800, 0xc0021637c0, 0x40b965, 0x6da0ae0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.0.2-0.20210714181518-b5a3dc95a675/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc001b5f4a0, 0x7e14cb0, 0xc0021637c0, 0xc00219d260, 0xc001b5f4a0, 0xc0012ed770, 0xc0021baba0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x707d4c0, 0xc001b5f4a0, 0x7e14cb0, 0xc0012ed770, 0xc00219d200, 0x0, 0x7e14cb0, 0xc0012ed770, 0xc001968c00, 0x387)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00059d6c0, 0x7e36b38, 0xc001c02480, 0xc0002a7300, 0xc0015daae0, 0xb675f70, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00059d6c0, 0x7e36b38, 0xc001c02480, 0xc0002a7300, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000cb2130, 0xc00059d6c0, 0x7e36b38, 0xc001c02480, 0xc0002a7300)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.51.0_x5 plugin crashed!

[justinretzolk]

Hi @aaron-lane 👋 Terraform v1.0.9 was released yesterday which includes a fix that should correct this issue. Can you test again to verify that you're no longer experience this issue?

[aaron-lane]

@justinretzolk I'll give it a whirl tomorrow on Monday (🎉) and let you know! 🍳

[aaron-lane]

I went through the reproduction steps again, using Terraform 1.0.0 for steps 1 and 2, and using Terraform 1.0.9 for step 3. The problem is resolved! 🥳 #️⃣

• step 1
• step 2
• step 3

[justinretzolk]

Thanks for confirming @aaron-lane! 🎉 With that in mind, we'll go ahead and close this issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.