Terraform Crash

[jchenseated]

Terraform Version

1.2.6

Terraform Configuration Files

We use terraform to map our entire infrastructure which expands across multiple aws module

Debug Output

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

value is null
goroutine 3308 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/hashicorp/terraform/internal/logging.PanicHandler()
        /home/circleci/project/project/internal/logging/panic.go:55 +0x153
panic({0x221dca0, 0x2c4a630})
        /usr/local/go/src/runtime/panic.go:838 +0x207
github.com/zclconf/go-cty/cty.Value.LengthInt({{{0x2c71838?, 0xc002e34970?}}, {0x0?, 0x0?}})
        /home/circleci/go/pkg/mod/github.com/zclconf/go-cty@v1.10.0/cty/value_ops.go:1063 +0x27d
github.com/hashicorp/terraform/internal/plans/objchange.assertPlanValid(0xc00418caa0, {{{0x2c718a8?, 0xc0033d9b20?}}, {0x22fa380?, 0xc00955d5c0?}}, {{{0x2c718a8?, 0xc0033d98d0?}}, {0x22fa380?, 0xc00955d230?}}, {{{0x2c718a8, ...}}, ...}, ...)
        /home/circleci/project/project/internal/plans/objchange/plan_valid.go:105 +0x9d2
github.com/hashicorp/terraform/internal/plans/objchange.assertPlanValid(0xc004041720, {{{0x2c718a8?, 0xc009440010?}}, {0x22fa380?, 0xc00955df20?}}, {{{0x2c718a8?, 0xc0033d99c0?}}, {0x22fa380?, 0xc00955d440?}}, {{{0x2c718a8, ...}}, ...}, ...)
        /home/circleci/project/project/internal/plans/objchange/plan_valid.go:127 +0x2e78
github.com/hashicorp/terraform/internal/plans/objchange.assertPlanValid(0xc004139bc0, {{{0x2c718a8?, 0xc009440140?}}, {0x22fa380?, 0xc00955df80?}}, {{{0x2c718a8?, 0xc0033d9a70?}}, {0x22fa380?, 0xc00955d4a0?}}, {{{0x2c718a8, ...}}, ...}, ...)
        /home/circleci/project/project/internal/plans/objchange/plan_valid.go:127 +0x2e78
github.com/hashicorp/terraform/internal/plans/objchange.AssertPlanValid(...)
        /home/circleci/project/project/internal/plans/objchange/plan_valid.go:36
github.com/hashicorp/terraform/internal/terraform.(*NodeAbstractResourceInstance).plan(0xc00488afc0, {0x2c7f9a0, 0xc006a96380}, 0x0, 0xc00092c720, 0x0, {0x0, 0x0, 0xc0092c8660?})
        /home/circleci/project/project/internal/terraform/node_resource_abstract_instance.go:790 +0x16fe
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstance).managedResourceExecute(0xc0046e3ec0, {0x2c7f9a0, 0xc006a96380})
        /home/circleci/project/project/internal/terraform/node_resource_plan_instance.go:218 +0xd6e
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstance).Execute(0x0?, {0x2c7f9a0?, 0xc006a96380?}, 0x80?)
        /home/circleci/project/project/internal/terraform/node_resource_plan_instance.go:60 +0x90
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0025b4c30, {0x2c7f9a0, 0xc006a96380}, {0x7f416a7c0ad0, 0xc0046e3ec0})
        /home/circleci/project/project/internal/terraform/graph_walk_context.go:135 +0xc2
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x266a620, 0xc0046e3ec0})
        /home/circleci/project/project/internal/terraform/graph.go:74 +0x2f0
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000b63260, {0x266a620, 0xc0046e3ec0}, 0xc00914be40)
        /home/circleci/project/project/internal/dag/walk.go:381 +0x2f1
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update
        /home/circleci/project/project/internal/dag/walk.go:304 +0xf45
Operation failed: failed running terraform plan (exit 11)

Actual Behavior

We started to run into issue without new commit to the code base. The last commit was planned and applied without crash.

Additional Context

I am submitting this issue so I hopefully can get a direction to debug (for example, which module should I target based on the console error)

[jbardin]

Hi @jchenseated,

Thanks for reporting the issue! We are going to need more information here, specifically a configuration which can reproduce the crash. The best place to start looking is with whatever changes were most recently made to the config, including changing versions of providers. Looking at the core trace logs may help too, chances are that the resource config which was involved in the crash logged something right before that point which could help narrow things down. Taking shot in the dark, something with ignore_changes could be involved here too.

Thanks!

[jchenseated]

Thank you @jbardin ! I followed your guideline and was able to pinpoint the issue to our appmesh resource's dynamic backend block. Here is the resource block code. I fixed the issue by manually removing the backend resource we configured on AWS UI. No terraform code changes needed. Whenever we added the backend resource on AWS UI, the error came back. Not sure what's the root cause for this, might be a provider issue.

Thank you anyway!

resource "aws_appmesh_virtual_node" "xxx" {

  for_each = xxx

  mesh_name = aws_appmesh_mesh.meshes[each.value.env].name
  name      = "${each.value.name}-${each.value.env}"

  spec {
    dynamic backend {
      ## virtual node's backend is a concatenation of its configured backend and (if it's logic layer, all logic layer nodes)
      for_each = [
        for DNS in distinct(concat(
          [ for s in each.value.java_backends : "${s}-${each.value.env}.internal" ],
          each.value.other_backends[each.value.env],
          contains(local.VIRTUAL_BACKEND_AUDIT_TIER, each.value.tier) ? local.ALL_LOGIC_LAYER_JAVA_BACKEND[each.value.env] : [])) : DNS if DNS != "${each.value.name}-${each.value.env}.internal"
      ]
      content {
        virtual_service {
          virtual_service_name = backend.value
        }
      }
    }
}

[jbardin]

Thanks for the added context @jchenseated. I have not yet been able to piece together how this example could create a null nested block at any level here. If a provider is involved in the error, there may be warnings logged about the specific resources, and looking for WARN level logs about the aws resources being used could help narrow down the search.

At this point my only hypothesis is a misbehaving legacy provider combined with ignore_changes, since otherwise only Terraform core should be tasked with evaluating the configuration.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.