Cannot destroy MSK resources

[emilienbtn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Hi All,

We are making some tests to deploy MSK,
Recently we try to destroy one of our clusters and end up with this panic error.

Any ideas where this could be from?

Thank you for your help :)

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.8
on darwin_amd64

• provider registry.terraform.io/hashicorp/aws v3.62.0

Affected Resource(s)

• aws_msk_cluster
• aws_msk_configuration

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  region = var.aws_region
}

terraform {
  backend "s3" {}
}

resource "aws_msk_cluster" "msk_cluster" {
  cluster_name            = var.cluster_name
  kafka_version           = var.kafka_version
  number_of_broker_nodes  = var.num_broker_nodes
  
  broker_node_group_info {
    instance_type       = var.instance_type
    ebs_volume_size  = var.broker_disk_size
    client_subnets      = var.subnet_ids
    security_groups  = var.security_groups
  }

  encryption_info {
    encryption_in_transit {
      client_broker   = "TLS_PLAINTEXT"
    }
  }

  configuration_info {
    arn = aws_msk_configuration.msk_configuration.arn
    revision = aws_msk_configuration.msk_configuration.latest_revision
  }

  enhanced_monitoring = var.enhanced_monitoring

  logging_info {
    broker_logs {
      cloudwatch_logs {
        enabled   = true
        log_group = var.log_group.name
      }
    }
  }

  open_monitoring {
    prometheus {
      jmx_exporter {
        enabled_in_broker = var.open_monitoring
      }
      node_exporter {
        enabled_in_broker = var.open_monitoring
      }
    }
  }

  lifecycle {
    ignore_changes = [broker_node_group_info["ebs_volume_size"]]
  }

  tags = var.tags
}

# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_configuration
resource "aws_msk_configuration" "msk_configuration" {
  kafka_versions    = [var.kafka_version]
  name              = var.config_name
  server_properties = var.config_properties

  # Allow the update of the configuration when the kafka version changes
  lifecycle {
    create_before_destroy = true
  }
}

# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_target
resource "aws_appautoscaling_target" "kafka_storage" {
  max_capacity       = var.storage_scaling_max_capacity
  min_capacity       = 1
  resource_id        = aws_msk_cluster.msk_cluster.arn
  scalable_dimension = "kafka:broker-storage:VolumeSize"
  service_namespace  = "kafka"
}

# https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_policy
resource "aws_appautoscaling_policy" "kafka_broker_scaling_policy" {
  name                  = var.scaling_policy_name
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_msk_cluster.msk_cluster.arn
  scalable_dimension = aws_appautoscaling_target.kafka_storage.scalable_dimension
  service_namespace  = aws_appautoscaling_target.kafka_storage.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "KafkaBrokerStorageUtilization"
    }

    target_value = var.storage_scaling_target_value
  }
}

Panic Output

https://gist.github.com/emilienbtn/0c3cebcfd7f2390b61badf95ac9a0f0b

Expected Behavior

The cluster and its configuration should be destroyed

Actual Behavior

The provider is encountering errors

Steps to Reproduce

1. terraform init
2. terraform plan
3. terraform destroy

Important Factoids

We also use Terragrunt to trigger terraform modules

[ewbankkit]

Stack trace from the terraform-provider-aws_v3.62.0_x5 plugin:

panic: interface conversion: interface {} is nil, not *aws.AWSClient

goroutine 25 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsMskConfigurationRead(0xc00291a300, 0x0, 0x0, 0xc9838b0, 0xc000396d80)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_msk_configuration.go:86 +0x11b9
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc000efe700, 0x8ebb1c8, 0xc00036c2c0, 0xc00291a300, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/resource.go:335 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000efe700, 0x8ebb1c8, 0xc00036c2c0, 0xc0023481c0, 0x0, 0x0, 0xc0006f61d8, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/resource.go:624 +0x1cb
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc0001a0fc0, 0x8ebb1c8, 0xc00036c2c0, 0xc00036c300, 0xc00036c2c0, 0x100b665, 0x7d93c20)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/gdavison/terraform-plugin-sdk/v2@v2.7.1-0.20210913224932-c7c2dbd9e010/helper/schema/grpc_provider.go:575 +0x43b
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc0018cbae0, 0x8ebb270, 0xc00036c2c0, 0xc00035c3c0, 0xc0018cbae0, 0xc00035fbf0, 0xc0007fdba0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/server/server.go:298 +0x105
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x808ab80, 0xc0018cbae0, 0x8ebb270, 0xc00035fbf0, 0xc00035c360, 0x0, 0x8ebb270, 0xc00035fbf0, 0xc002346000, 0x265)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.3.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002af6c0, 0x8edf058, 0xc0003e4600, 0xc002342000, 0xc0015ea510, 0xc942150, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0002af6c0, 0x8edf058, 0xc0003e4600, 0xc002342000, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00101ecd0, 0xc0002af6c0, 0x8edf058, 0xc0003e4600, 0xc002342000)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.62.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[ewbankkit]

Relates:

• NodeApplyableProvider Does Not Call ConfigureProvider for Destroy Plan With Deposed Instances terraform#29712
• v1.0 backport of #29720: skip refreshing deposed during destroy plan terraform#29722
• The terraform-provider-aws_v3.61.0_x5 plugin crashed #21215
• Invalid OpenAPI spec in aws_api_gateway_rest_api leads to plugin crash during destroy state refresh #21181
• plug in crash when doing a terraform apply on EKS. #21140
• Plugin crash on the refreshing state stage #21132
• Error: The terraform-provider-aws_v3.60.0_x5 plugin crashed! Jenkins job failed. #21079
• terraform destroy gives "Error: The terraform-provider-aws_v3.59.0_x5 plugin crashed!" #21019
• "terraform-provider-aws_v3.57.0_x5 plugin crashed" when calling terraform destroy (EKS cluster) #20941
• Terraform Destroy crashed  #20878
• Plugin crashed #20461
• terraform-provider-aws_v3.52.0_x5_plugin_crashed #20438
• Terraform AWS plugin crashes when destroying items #20131
• terraform-provider-aws_v3.47.0_x5 plugin - interface {} is nil, not *aws.AWSClient #20015
• error running terraform 1.0 destroy on local windows env #19807
• aws provider v3.45 crashes during deleting resources #19788
• Crash while destroying state #19740
• Error: Plugin did not respond #19601

[ewbankkit]

@emilienbtn Thanks for raising this issue.
The crash is caused by an issue in the Terraform CLI which should be fixed in an upcoming patch release.

[justinretzolk]

Hi @emilienbtn 👋 Terraform v1.0.9 was released yesterday which includes a fix that should correct this issue. Can you test again to verify that you're no longer experience this issue?

[emilienbtn]

Hi @justinretzolk,

Sorry for the late reply. We tested several terraform destroy plans and they all went well.
Thank you very much for being so reactive to the problem.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.