Terraform private endpoint on an IoT Hub: deletion provokes crash and state corruption

[jzaegel]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.13.5

• provider registry.terraform.io/hashicorp/azurerm v2.33.0

Affected Resource(s)

• azurerm_private_endpoint
• azurerm_iothub

Terraform Configuration Files

locals {
  location            = "canadacentral"
  iothub_tier         = "B1"
  try_number          = "4"
  resource_group_name = "rg-iot-bug-repro-try-${local.try_number}"
}

resource "azurerm_resource_group" "rg_iothub_repro" {
  name     = local.resource_group_name
  location = local.location
}

resource "azurerm_iothub" "iothub" {
  name                = "iot-repro-${local.try_number}"
  location            = local.location
  resource_group_name = azurerm_resource_group.rg_iothub_repro.name

  sku {
    name     = local.iothub_tier
    capacity = 1
  }

  public_network_access_enabled = true
}

resource "azurerm_virtual_network" "vnet" {
  name                = "iot-repro-vnet"
  address_space       = ["10.0.0.0/16"]
  location            = local.location
  resource_group_name = azurerm_resource_group.rg_iothub_repro.name
}


resource "azurerm_subnet" "subnet_upstream" {
  name                                           = "iot-repro-subnet-upstream"
  resource_group_name                            = azurerm_resource_group.rg_iothub_repro.name
  virtual_network_name                           = azurerm_virtual_network.vnet.name
  address_prefixes                               = ["10.0.1.0/24"]
  enforce_private_link_endpoint_network_policies = true
}

resource "azurerm_private_endpoint" "iothub_input_private_endpoint" {
  name                = "pep-in-iot-repro"
  location            = local.location
  resource_group_name = azurerm_resource_group.rg_iothub_repro.name
  subnet_id           = azurerm_subnet.subnet_upstream.id

  private_service_connection {
    name                           = azurerm_iothub.iothub.name
    private_connection_resource_id = azurerm_iothub.iothub.id
    subresource_names              = ["iothub"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "zone-group"
    private_dns_zone_ids = [azurerm_private_dns_zone.iothub_dns.id]
  }
}

resource "azurerm_subnet" "subnet-downstream" {
  name                                           = "iot-repro-subnet-downstream"
  resource_group_name                            = azurerm_resource_group.rg_iothub_repro.name
  virtual_network_name                           = azurerm_virtual_network.vnet.name
  address_prefixes                               = ["10.0.2.0/24"]
  enforce_private_link_endpoint_network_policies = true
}

resource "azurerm_private_endpoint" "iothub_downstream_private_endpoint" {
  name                = "pep-out-iot-repro"
  location            = local.location
  resource_group_name = azurerm_resource_group.rg_iothub_repro.name
  subnet_id           = azurerm_subnet.subnet-downstream.id

  private_service_connection {
    name                           = azurerm_iothub.iothub.name
    private_connection_resource_id = azurerm_iothub.iothub.id
    subresource_names              = ["iotHub"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "zone-group"
    private_dns_zone_ids = [azurerm_private_dns_zone.iothub_dns.id]
  }
}

resource "azurerm_private_dns_zone" "iothub_dns" {
  name                = "privatelink.iothub.core.windows.net"
  resource_group_name = azurerm_resource_group.rg_iothub_repro.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "iothub_dns_link" {
  name                  = "iothub_dns"
  resource_group_name   = azurerm_resource_group.rg_iothub_repro.name
  private_dns_zone_name = azurerm_private_dns_zone.iothub_dns.name
  virtual_network_id    = azurerm_virtual_network.vnet.id
}

Debug Output

https://gist.github.com/jzaegel/891f9baea9e92fa8992a433490961dba

Panic Output

https://gist.github.com/jzaegel/f1f05ad87830dd17e6e2230f8c5a56cb

Expected Behavior

After the above config has been deployed, commenting the private endpoints related sources (iothub_input_private_endpoint, iothub_downstream_private_endpoint, iothub_dns, iothub_dns_link) and running terraform apply should lead to the smooth deletion of these resources.

Actual Behavior

terraform apply ends like this:

azurerm_private_dns_zone_virtual_network_link.iothub_dns_link: Destruction complete after 2m2s

Error: waiting for deletion of Private Endpoint "pep-out-iot-repro" (Resource Group "rg-iot-bug-repro-try-4"): Code="Failed" Message="The async operation failed." InnerError={"unmarshalError":"json: cannot unmarshal array into Go struct field respErr.error of type map[string]interface {}"} AdditionalInfo=[{"error":{"code":"412001","details":[],"innerError":"Source: Nrp.Frontend.ClientCommon. Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.OperationException: Either bad ETag was supplied or IotHub description was changed. If you contact a support representative please include this correlation identifier: 0abb17cd-2210-42b5-a7eb-f5ec6c5001b7, timestamp: 2020-10-28 19:36:04Z, errorcode: IH412001.\\r\\nCode: 412001\\r\\nInnerError: \\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cReadResult\u003ed__34`2.MoveNext() in X:\\\\bt\\\\1022955\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 413\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Client.Common.OperationChannel.\u003cExecuteOperation\u003ed__30`2.MoveNext() in X:\\\\bt\\\\1022955\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontendClientCommon\\\\OperationChannel.cs:line 0\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Common.ArmAuthProviderFacade.PrivateLinkServiceRpFacade.\u003cDeletePrivateEndpointConnectionProxy\u003ed__13.MoveNext() in X:\\\\bt\\\\1022955\\\\repo\\\\src\\\\sources\\\\Common\\\\ArmAuthProviderFacade\\\\PrivateLinkServiceRpFacade.cs:line 121\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cCallDeleteOnFirstPartyAsync\u003ed__18.MoveNext() in X:\\\\bt\\\\1022955\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 443\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Csm.DeletePrivateEndpointOperation.\u003cRunBackgroundTaskAsync\u003ed__14.MoveNext() in X:\\\\bt\\\\1022955\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Csm\\\\DeletePrivateEndpointOperation.cs:line 105\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\\r\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n   at Microsoft.WindowsAzure.Networking.Nrp.Frontend.Operations.Operation`1.\u003cStartRunBackgroundTaskAsync\u003ed__134.MoveNext() in X:\\\\bt\\\\1022955\\\\repo\\\\src\\\\sources\\\\Frontend\\\\FrontEndOperations\\\\Operation.cs:line 379","message":"Call to Microsoft.Devices/IotHubs failed."},"status":"Failed"}]

Running terraform apply again leads to a Terraform crash:

azurerm_private_endpoint.iothub_downstream_private_endpoint: Destroying... [id=/subscriptions/91f943b7-b4aa-4e32-ae06-2977b666d95d/resourceGroups/rg-iot-bug-repro-try-4/providers/Microsoft.Network/privateEndpoints/pep-out-iot-repro]

Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Canceled desc = context canceled
panic: interface conversion: interface {} is nil, not map[string]interface {}
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: goroutine 115 [running]:
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/network.resourceArmPrivateEndpointDelete(0xc000a5a770, 0x440e9c0, 0xc0002cc2c0, 0x0, 0x0)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/network/private_endpoint_resource.go:446 +0x929
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).Apply(0xc0007974d0, 0xc000c40be0, 0xc000f2ae40, 0x440e9c0, 0xc0002cc2c0, 0x40d401, 0x20, 0x4736660)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:283 +0x49f
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Apply(0xc0000faf00, 0xc000b93a10, 0xc000c40be0, 0xc000f2ae40, 0x8467880, 0xc000c40be0, 0x0)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:294 +0xa0
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ApplyResourceChange(0xc000007c20, 0x536a660, 0xc000c435c0, 0xc000a5a230, 0xc000007c20, 0xc000c435c0, 0xc000161b78)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:885 +0x8bb
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0x4824ac0, 0xc000007c20, 0x536a660, 0xc000c435c0, 0xc00002a600, 0x0, 0x536a660, 0xc000c435c0, 0xc000f24c00, 0x5ba)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3305 +0x21e
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0005edc80, 0x53ac060, 0xc000704780, 0xc00019e800, 0xc000ba26f0, 0x842a720, 0x0, 0x0, 0x0)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:1024 +0x508
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: google.golang.org/grpc.(*Server).handleStream(0xc0005edc80, 0x53ac060, 0xc000704780, 0xc00019e800, 0x0)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:1313 +0xd44
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000b0a610, 0xc0005edc80, 0x53ac060, 0xc000704780, 0xc00019e800)
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:722 +0xa8
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-10-28T15:38:33.284-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:720 +0xa8
2020/10/28 15:38:33 [DEBUG] azurerm_private_endpoint.iothub_downstream_private_endpoint: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalWriteState
2020/10/28 15:38:33 [TRACE] EvalWriteState: writing current state object for azurerm_private_endpoint.iothub_downstream_private_endpoint
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalApplyPost
2020/10/28 15:38:33 [ERROR] eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:38:33 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:38:33 [ERROR] eval: *terraform.EvalOpFilter, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:38:33 [TRACE] [walkApply] Exiting eval tree: azurerm_private_endpoint.iothub_downstream_private_endpoint (destroy)
2020/10/28 15:38:33 [TRACE] vertex "azurerm_private_endpoint.iothub_downstream_private_endpoint (destroy)": visit complete
2020/10/28 15:38:33 [TRACE] vertex "azurerm_private_dns_zone.iothub_dns (destroy)": evaluating
2020/10/28 15:38:33 [TRACE] [walkApply] Entering eval tree: azurerm_private_dns_zone.iothub_dns (destroy)
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalOpFilter
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalSequence
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalGetProvider
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalReadDiff
2020/10/28 15:38:33 [TRACE] EvalReadDiff: Read Delete change from plan for azurerm_private_dns_zone.iothub_dns
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalReduceDiff
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalIf
2020/10/28 15:38:33 [TRACE] eval: terraform.EvalNoop
2020/10/28 15:38:33 [TRACE] eval: *terraform.EvalReadState
2020/10/28 15:38:33 [TRACE] EvalReadState: reading state for azurerm_private_dns_zone.iothub_dns
2020/10/28 15:38:33 [TRACE] UpgradeResourceState: schema version of azurerm_private_dns_zone.iothub_dns is still 0; calling provider "azurerm" for any other minor fixups
2020/10/28 15:38:33 [TRACE] GRPCProvider: UpgradeResourceState
2020-10-28T15:38:33.300-0400 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/azurerm/2.33.0/windows_amd64/terraform-provider-azurerm_v2.33.0_x5.exe pid=13092 error="exit status 2"
2020/10/28 15:38:33 [ERROR] eval: *terraform.EvalReadState, err: rpc error: code = Canceled desc = context canceled
2020/10/28 15:38:33 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2020/10/28 15:38:33 [ERROR] eval: *terraform.EvalOpFilter, err: rpc error: code = Canceled desc = context canceled
2020/10/28 15:38:33 [TRACE] [walkApply] Exiting eval tree: azurerm_private_dns_zone.iothub_dns (destroy)
2020/10/28 15:38:33 [TRACE] vertex "azurerm_private_dns_zone.iothub_dns (destroy)": visit complete
2020/10/28 15:38:33 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" errored, so skipping
2020/10/28 15:38:33 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/10/28 15:38:33 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/10/28 15:38:33 [TRACE] statemgr.Filesystem: creating backup snapshot at terraform.tfstate.backup
2020/10/28 15:38:33 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 17
2020/10/28 15:38:33 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2020/10/28 15:38:33 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2020/10/28 15:38:33 [TRACE] statemgr.Filesystem: unlocked by closing terraform.tfstate
2020-10-28T15:38:33.323-0400 [DEBUG] plugin: plugin exited





!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

After that even terraform plan fails:

Microsoft.Network/virtualNetworks/iot-repro-vnet/subnets/iot-repro-subnet-downstream]

Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closingpanic: interface conversion: interface {} is nil, not map[string]interface {}
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: goroutine 33 [running]:
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/network.resourceArmPrivateEndpointRead(0xc000ac84d0, 0x440e9c0, 0xc001488000, 0x0, 0x0)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/network/private_endpoint_resource.go:411 +0x12b6
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000aef320, 0xc0006905f0, 0x440e9c0, 0xc001488000, 0xc0003632c0, 0xc0006905f0, 0x0)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:460 +0x120
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc000b327b8, 0x536a660, 0xc0004f8cf0, 0xc0009bc1e0, 0xc000b327b8, 0xc0004f8cf0, 0xc000223b78)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:525 +0x3df
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x4824ac0, 0xc000b327b8, 0x536a660, 0xc0004f8cf0, 0xc0009bc180, 0x0, 0x536a660, 0xc0004f8cf0, 0xc001009800, 0x5b0)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3269 +0x21e
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000685c80, 0x53ac060, 0xc000644780, 0xc001016000, 0xc000774000, 0x842a6f0, 0x0, 0x0, 0x0)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:1024 +0x508
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: google.golang.org/grpc.(*Server).handleStream(0xc000685c80, 0x53ac060, 0xc000644780, 0xc001016000, 0x0)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:1313 +0xd44
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000036cc0, 0xc000685c80, 0x53ac060, 0xc000644780, 0xc001016000)
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:722 +0xa8
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-10-28T15:39:33.289-0400 [DEBUG] plugin.terraform-provider-azurerm_v2.33.0_x5.exe:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-azurerm/vendor/google.golang.org/grpc/server.go:720 +0xa8
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [TRACE] [walkRefresh] Exiting eval tree: azurerm_private_endpoint.iothub_downstream_private_endpoint
2020/10/28 15:39:33 [TRACE] vertex "azurerm_private_endpoint.iothub_downstream_private_endpoint": visit complete
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [TRACE] [walkRefresh] Exiting eval tree: azurerm_subnet.subnet-downstream
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet-downstream": visit complete
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet-downstream": dynamic subgraph encountered errors
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet-downstream": visit complete
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet-downstream (expand)": dynamic subgraph encountered errors
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet-downstream (expand)": visit complete
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [TRACE] [walkRefresh] Exiting eval tree: azurerm_iothub.iothub
2020/10/28 15:39:33 [TRACE] vertex "azurerm_iothub.iothub": visit complete
2020/10/28 15:39:33 [TRACE] vertex "azurerm_iothub.iothub": dynamic subgraph encountered errors
2020/10/28 15:39:33 [TRACE] vertex "azurerm_iothub.iothub": visit complete
2020/10/28 15:39:33 [TRACE] vertex "azurerm_iothub.iothub (expand)": dynamic subgraph encountered errors
2020/10/28 15:39:33 [TRACE] vertex "azurerm_iothub.iothub (expand)": visit complete
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/28 15:39:33 [TRACE] [walkRefresh] Exiting eval tree: azurerm_subnet.subnet_upstream
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet_upstream": visit complete
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet_upstream": dynamic subgraph encountered errors
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet_upstream": visit complete
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet_upstream (expand)": dynamic subgraph encountered errors
2020/10/28 15:39:33 [TRACE] vertex "azurerm_subnet.subnet_upstream (expand)": visit complete
2020/10/28 15:39:33 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" errored, so skipping
2020/10/28 15:39:33 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/10/28 15:39:33 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2020/10/28 15:39:33 [TRACE] statemgr.Filesystem: unlocked by closing terraform.tfstate
2020-10-28T15:39:33.312-0400 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/azurerm/2.33.0/windows_amd64/terraform-provider-azurerm_v2.33.0_x5.exe pid=12556 error="exit status 2"
2020-10-28T15:39:33.312-0400 [DEBUG] plugin: plugin exited





Error: rpc

er!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
r
oTerraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
rworking directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.
:
When reporting bugs, please include your terraform version. That
 information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.
c
oSECURITY WARNING: the "crash.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
don the issue tracker.
e
 [1]: https://github.com/hashicorp/terraform/issues

=!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
 Unavailable desc = transport is closing

The only way to get out of this is to restore the state with a backup

Steps to Reproduce

1. terraform apply
2. Work around the failure of terraform apply (see Terraform private endpoint on an IoT Hub: "Either bad ETag was supplied or IotHub description was changed" #9060) with import terraform import azurerm_private_endpoint.iothub_input_private_endpoint xxxxx
3. terraform apply to finish the deployment
4. Comment the private endpoints related sources (iothub_input_private_endpoint, iothub_downstream_private_endpoint, iothub_dns, iothub_dns_link)
5. terraform apply
6. terraform apply
7. terraform plan

Important Factoids

References

• Related to Terraform private endpoint on an IoT Hub: "Either bad ETag was supplied or IotHub description was changed" #9060. Feels like the etag issue is causing havoc in the deletion process.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!