aws provider v3.45 crashes during deleting resources

[witnsby]

When aws_api_gateway_deployment has a parameter stage_name where a value inside exists in the api_gateway then AWS provider 3.45.0 crashes during deleting resources.

Stack trace from the terraform-provider-aws_v3.45.0_x5 plugin:

...
panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 114 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsApiGatewayMethodSettingsRead(0xc00065a480, 0x0, 0x0, 0xc022270, 0xc000601b00)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_api_gateway_method_settings.go:137 +0xea5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0008382a0, 0x8859ae8, 0xc00063c240, 0xc00065a480, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0008382a0, 0x8859ae8, 0xc00063c240, 0xc00067c150, 0x0, 0x0, 0xc00064a220, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00000c840, 0x8859ae8, 0xc00063c240, 0xc00063c280, 0xc00063c240, 0x100b665, 0x781f9a0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.6.1/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc00237b480, 0x8859b90, 0xc00063c240, 0xc00062e3c0, 0xc00237b480, 0xc00063ea80, 0xc0023b6ba0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x7aefd20, 0xc00237b480, 0x8859b90, 0xc00063ea80, 0xc00062e360, 0x0, 0x8859b90, 0xc00063ea80, 0xc0006721c0, 0x1a7)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000a3b340, 0x887b818, 0xc000c8c300, 0xc00067a000, 0xc00172b890, 0xbfe19d0, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000a3b340, 0x887b818, 0xc000c8c300, 0xc00067a000, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0015f69b0, 0xc000a3b340, 0x887b818, 0xc000c8c300, 0xc00067a000)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.45.0_x5 plugin crashed!
...

and the issue from my own module, I hope it is useful too.

│ Error: Plugin did not respond │ │ with module.serverless_tf_deals.aws_api_gateway_deployment.apigw_deployment, │ on ../modules/sls/apigw_deployment.tf line 1, in resource "aws_api_gateway_deployment" "apigw_deployment": │ 1: resource "aws_api_gateway_deployment" "apigw_deployment" { │ │ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.

[ewbankkit]

Suspiciously similar to

• Crash while destroying state #19740
• Error: Plugin did not respond #19601

[ewbankkit]

@witnsby Thanks for raising this issue.
Do you have an example Terraform configuration that you can share that causes the crash?

[ewbankkit]

@witnsby Which version of Terraform are you running?

[witnsby]

@witnsby Which version of Terraform are you running?

@ewbankkit 0.15.5

[witnsby]

@witnsby Thanks for raising this issue.
Do you have an example Terraform configuration that you can share that causes the crash?

unfortunately, no.

[jseiser]

Stack trace from the terraform-provider-aws_v3.59.0_x5 plugin:

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 51 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsLbTargetGroupRead(0xc00273cb80, 0x0, 0x0, 0xbb3a1c0, 0xc000078800)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_lb_target_group.go:385 +0x967
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc001371420, 0x81251e8, 0xc00207c180, 0xc00273cb80, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc001371420, 0x81251e8, 0xc00207c180, 0xc002752540, 0x0, 0x0, 0xc0027564d8, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00000cf90, 0x81251e8, 0xc00207c180, 0xc00207c1c0, 0xc00207c180, 0x40b965, 0x703ca80)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc001953980, 0x8125290, 0xc00207c180, 0xc00270a240, 0xc001953980, 0xc0020c6690, 0xc0019b1ba0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x732afa0, 0xc001953980, 0x8125290, 0xc0020c6690, 0xc00270a1e0, 0x0, 0x8125290, 0xc0020c6690, 0xc000108d80, 0x44f)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001cba40, 0x8147a18, 0xc00249a000, 0xc002748100, 0xc00139e990, 0xbaf8750, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0001cba40, 0x8147a18, 0xc00249a000, 0xc002748100, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000fb23f0, 0xc0001cba40, 0x8147a18, 0xc00249a000, 0xc002748100)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.59.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

This is running a destroy on an older version of the terraform-aws-alb module from the registry

[kpashov]

I have this same issue - I have tried 3.30, 3.40, 3.59, 3.60, 3.61 with Terraform 1.0.6 on MacOS BigSur- they all failed when retrieving state for the same resources.

I also tried with 3.61 on Terraform 1.0.7, still the same behaviour.

I have also tried 3.61 on Terraform 1.0.8 inside a Debian Linux container - that also fails with the same error, on the same resource.

In my case, it fails when trying to retrieve an aws_launch_configuration.

[ewbankkit]

@kpashov Is it possible to paste in your configuration? The maintainers are having a hard time creating a configuration which reliably reproduces this (and similar) crashes.

[kpashov]

I wish I could, but it is a massive 100+ resource terraform configuration. I have just now managed to get mine unstuck by:

• Manually destroying a few resources
• Manually removing them from the state with terraform state rm
• Running a terraform apply
• And then running a terraform destroy.

Given that other people have reported something like the above working, I don't think there's a configuration that will reliably reproduce the problem - I think it has something to do with a series of failures. Next time it happens, I will have a look at the TF state to see if there is something strange happening there.

[justinretzolk]

Hi @witnsby 👋 Terraform v1.0.9 was released yesterday which includes a fix that should correct this issue. Can you test again to verify that you're no longer experience this issue?

[martinwrightgithub]

I've just had this issue, and I'm on terraform v1.0.9. So I can confirm the issue is still present.

Error

tack trace from the terraform-provider-aws plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 81 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.SetTagsDiff(0x7670548, 0xc001e00dc0, 0xc0020ac900, 0x5e5dd80, 0xc001b4db80, 0x768a390, 0xc00012fb00)
	/home/rhughes/terraform-provider-aws/aws/tags.go:123 +0x3a5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0xc0006ad2c0, 0x7670548, 0xc001e00dc0, 0xc000474c40, 0xc001d94d80, 0x6cf2d60, 0x5e5dd80, 0xc001b4db80, 0xab93d00, 0x559f600, ...)
	/home/rhughes/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/schema.go:540 +0xb29
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0xc00088c380, 0x7670548, 0xc001e00dc0, 0xc000474c40, 0xc001d94d80, 0x5e5dd80, 0xc001b4db80, 0x0, 0x0, 0x0)
	/home/rhughes/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:506 +0xa5
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0xc0003047f8, 0x7670548, 0xc001e00dc0, 0xc0018e46e0, 0xc001e00dc0, 0x69a6640, 0xc0012eb600)
	/home/rhughes/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/grpc_provider.go:693 +0x7c5
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).PlanResourceChange(0xc001e5f840, 0x76705f0, 0xc001e00dc0, 0xc000474a10, 0xc001e5f840, 0xc0012eb680, 0xc001c7bba0)
	/home/rhughes/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:315 +0xb5
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x69a6640, 0xc001e5f840, 0x76705f0, 0xc0012eb680, 0xc0021c0c00, 0x0, 0x76705f0, 0xc0012eb680, 0xc00039e000, 0x301)
	/home/rhughes/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:362 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000241500, 0x7690758, 0xc001742480, 0xc001c74200, 0xc0003e6ab0, 0xab52bc8, 0x0, 0x0, 0x0)
	/home/rhughes/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000241500, 0x7690758, 0xc001742480, 0xc001c74200, 0x0)
	/home/rhughes/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0003c0830, 0xc000241500, 0x7690758, 0xc001742480, 0xc001c74200)
	/home/rhughes/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/rhughes/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws plugin crashed!

Terraform version

Terraform v1.0.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.64.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.3
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/time v0.7.2
+ provider registry.terraform.io/hashicorp/tls v3.1.0

[martinwrightgithub]

Fixed. My problem was caused by a custom override.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.