Plugin crash on the refreshing state stage

[speller]

module.backend.module.backend-admin.aws_instance.instance (deposed object c5240807): Refreshing state... [id=i-09ddad078818ac310]
Error: Plugin did not respond
  with module.lb.aws_lb_target_group.frontend,
  on ../../lb/main.tf line 260, in resource "aws_lb_target_group" "frontend":
 260: resource "aws_lb_target_group" "frontend" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain
more details.
Error: Plugin did not respond
  with module.deployment-domain.aws_acm_certificate.cert,
  on ../../subdomain/main.tf line 33, in resource "aws_acm_certificate" "cert":
  33: resource "aws_acm_certificate" "cert" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain
more details.
Error: Plugin did not respond
  with module.backend.module.backend-admin.aws_instance.instance,
  on ../../module/ec2-docker/main.tf line 19, in resource "aws_instance" "instance":
  19: resource "aws_instance" "instance" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
details.
Error: Plugin did not respond
  with module.backend.module.backend-admin.aws_instance.instance,
  on ../../module/ec2-docker/main.tf line 19, in resource "aws_instance" "instance":
  19: resource "aws_instance" "instance" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
details.
Error: Plugin did not respond
  with module.backend.module.backend-admin.aws_instance.instance,
  on ../../module/ec2-docker/main.tf line 19, in resource "aws_instance" "instance":
  19: resource "aws_instance" "instance" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
details.
Error: Plugin did not respond
  with module.backend.module.frontend.aws_iam_instance_profile.iam-instance-profile,
  on ../../module/ec2-docker/main.tf line 41, in resource "aws_iam_instance_profile" "iam-instance-profile":
  41: resource "aws_iam_instance_profile" "iam-instance-profile" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain
more details.
Error: Plugin did not respond
  with module.backend.module.frontend.aws_cloudwatch_log_group.docker-log-group,
  on ../../module/ec2-docker/main.tf line 110, in resource "aws_cloudwatch_log_group" "docker-log-group":
 110: resource "aws_cloudwatch_log_group" "docker-log-group" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain
more details.
Error: Plugin did not respond
  with module.backend.module.frontend.aws_security_group.security-group,
  on ../../module/ec2-docker/main.tf line 120, in resource "aws_security_group" "security-group":
 120: resource "aws_security_group" "security-group" {
The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may contain
more details.
module.backend.module.backend-admin.aws_instance.instance (deposed object 26830708): Refreshing state... [id=i-0fe5d374afd7f0ee7]
module.backend.module.backend-admin.aws_instance.instance (deposed object 9b85b620): Refreshing state... [id=i-09f5306b864412ce4]
Warning: Experimental feature "module_variable_optional_attrs" is active
  on main.tf line 10, in terraform:
  10:   experiments = [module_variable_optional_attrs]
Experimental features are subject to breaking changes in future minor or
patch releases, based on feedback.
If you have feedback on the design of this feature, please open a GitHub
issue to discuss it.
(and 6 more similar warnings elsewhere)
Stack trace from the terraform-provider-aws_v3.53.0_x5 plugin:
panic: interface conversion: interface {} is nil, not *aws.AWSClient
goroutine 79 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsInstanceRead(0xc000afe880, 0x0, 0x0, 0xb82dd20, 0xc000098800)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_instance.go:885 +0x44a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc000c542a0, 0x7f0e828, 0xc000af3000, 0xc000afe880, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.0.2-0.20210714181518-b5a3dc95a675/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000c542a0, 0x7f0e828, 0xc000af3000, 0xc001c12150, 0x0, 0x0, 0xc0021b8c30, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.0.2-0.20210714181518-b5a3dc95a675/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00019e588, 0x7f0e828, 0xc000af3000, 0xc000af3040, 0xc000af3000, 0x40b965, 0x6e76140)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.0.2-0.20210714181518-b5a3dc95a675/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc000bb7e80, 0x7f0e8d0, 0xc000af3000, 0xc000b56540, 0xc000bb7e80, 0xc00071d7d0, 0xc002706ba0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x7159860, 0xc000bb7e80, 0x7f0e8d0, 0xc00071d7d0, 0xc000b564e0, 0x0, 0x7f0e8d0, 0xc00071d7d0, 0xc00074b000, 0x2f85)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000f4e700, 0x7f30878, 0xc0015c9e00, 0xc002206100, 0xc0015dca80, 0xb7ec2f0, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000f4e700, 0x7f30878, 0xc0015c9e00, 0xc002206100, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000265bb0, 0xc000f4e700, 0x7f30878, 0xc0015c9e00, 0xc002206100)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd
Error: The terraform-provider-aws_v3.53.0_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[ewbankkit]

Similar:

• Error: The terraform-provider-aws_v3.60.0_x5 plugin crashed! Jenkins job failed. #21079
• terraform destroy gives "Error: The terraform-provider-aws_v3.59.0_x5 plugin crashed!" #21019
• "terraform-provider-aws_v3.57.0_x5 plugin crashed" when calling terraform destroy (EKS cluster) #20941
• Terraform Destroy crashed  #20878
• Plugin crashed #20461
• terraform-provider-aws_v3.52.0_x5_plugin_crashed #20438
• Terraform AWS plugin crashes when destroying items #20131
• terraform-provider-aws_v3.47.0_x5 plugin - interface {} is nil, not *aws.AWSClient #20015
• error running terraform 1.0 destroy on local windows env #19807
• aws provider v3.45 crashes during deleting resources #19788
• Crash while destroying state #19740
• Error: Plugin did not respond #19601

[justinretzolk]

Hi @speller 👋 Terraform v1.0.9 was released yesterday which includes a fix that should correct this. Can you test again with this newer version to verify that you're no longer experience this issue?

[justinretzolk]

👋 The other reports we've had of this issue were all resolved by Terraform 1.0.9; since we haven't heard back on this issue, I'll assume the same and close this one out. If you run into additional issues, please open a new issue so that we may continue to investigate.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.