crash from null nested blocks when included in ignore_changes

[farnsworth2008]

Terraform Version

Terraform v1.3.5
on linux_arm64
+ provider registry.terraform.io/hashicorp/aws v4.42.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/external v2.2.3
+ provider registry.terraform.io/hashicorp/kubernetes v2.16.0
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/time v0.9.1

Terraform Configuration Files

resource "kubernetes_job" "this" {
  # stuff here
}

Debug Output

N/A

Expected Behavior

Terraform should not crash. The job should be replaced

Actual Behavior

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.
When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.
[1]: https://github.com/hashicorp/terraform/issues
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
value is null
goroutine 3646 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/hashicorp/terraform/internal/logging.PanicHandler()
/home/circleci/project/project/internal/logging/panic.go:55 +0x153
panic({0x2144f60, 0x2b44e90})
/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/zclconf/go-cty/cty.Value.LengthInt({{{0x2b6dbb0?, 0xc002a60b60?}}, {0x0?, 0x0?}})
/home/circleci/go/pkg/mod/github.com/zclconf/go-cty@v1.12.1/cty/value_ops.go:1066 +0x1e5
github.com/hashicorp/terraform/internal/plans/objchange.assertPlanValid(0xc001f60320, {{{0x2b6dc20?, 0xc003f3afc0?}}, {0x221c7a0?, 0xc006ea5ec0?}}, {{{0x2b6dc20?, 0xc003ea9810?}}, {0x221c7a0?, 0xc006ea4480?}}, {{{0x2b6dc20, ...}}, ...}, ...)
/home/circleci/project/project/internal/plans/objchange/plan_valid.go:105 +0x9e5
github.com/hashicorp/terraform/internal/plans/objchange.assertPlanValid(0xc001f602d0, {{{0x2b6dc20?, 0xc003f3b010?}}, {0x221c7a0?, 0xc006ea5f20?}}, {{{0x2b6dc20?, 0xc003ea9880?}}, {0x221c7a0?, 0xc006ea44e0?}}, {{{0x2b6dc20, ...}}, ...}, ...)
/home/circleci/project/project/internal/plans/objchange/plan_valid.go:127 +0x2eb8
github.com/hashicorp/terraform/internal/plans/objchange.assertPlanValid(0xc001f41a10, {{{0x2b6dc20?, 0xc003f3b110?}}, {0x221c7a0?, 0xc006efa000?}}, {{{0x2b6dc20?, 0xc003ea9920?}}, {0x221c7a0?, 0xc006ea45a0?}}, {{{0x2b6dc20, ...}}, ...}, ...)
/home/circleci/project/project/internal/plans/objchange/plan_valid.go:127 +0x2eb8
github.com/hashicorp/terraform/internal/plans/objchange.AssertPlanValid(...)
/home/circleci/project/project/internal/plans/objchange/plan_valid.go:36
github.com/hashicorp/terraform/internal/terraform.(*NodeAbstractResourceInstance).plan(0xc002616000, {0x2b84098, 0xc0019660e0}, 0x0, 0xc002fabda0, 0x0, {0x0, 0x0, 0x0?})
/home/circleci/project/project/internal/terraform/node_resource_abstract_instance.go:857 +0x19fe
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstance).managedResourceExecute(0xc0025eedc0, {0x2b84098, 0xc0019660e0})
/home/circleci/project/project/internal/terraform/node_resource_plan_instance.go:223 +0xdae
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstance).Execute(0x0?, {0x2b84098?, 0xc0019660e0?}, 0x0?)
/home/circleci/project/project/internal/terraform/node_resource_plan_instance.go:60 +0x90
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0033dd1d0, {0x2b84098, 0xc0019660e0}, {0x7f5d5c0d2ab8, 0xc0025eedc0})
/home/circleci/project/project/internal/terraform/graph_walk_context.go:136 +0xc2
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x256cb40, 0xc0025eedc0})
/home/circleci/project/project/internal/terraform/graph.go:74 +0x2f0
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000d41ce0, {0x256cb40, 0xc0025eedc0}, 0xc0025ef100)
/home/circleci/project/project/internal/dag/walk.go:381 +0x2f6
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update
/home/circleci/project/project/internal/dag/walk.go:304 +0xf65

Steps to Reproduce

This crash occurs at random times when replacing k8 jobs

Additional Context

No response

References

No response

[kmoe]

Thanks for the bug report. Please provide some Terraform configuration that reproduces this issue, and/or a debug log.

It's possible this is an issue with the Kubernetes provider, so you may want to open a bug report there: https://github.com/hashicorp/terraform-provider-kubernetes

This issue looks similar, though without narrowing down which resource causes your crash, it's hard to tell. hashicorp/terraform-provider-kubernetes#1552

[jbardin]

Yes, the configuration will be important here, because the crash happens when handling the configuration value in Terraform. I suspect there is an ignore_changes in there, which is forcing Terraform form to use an invalid value from the provider as configuration, since decoding a block should never directly result in a null value. So while there may be a bug in the provider, it is slipping through the validation somehow to cause a crash in Terraform, which we would like to track down.

Thanks!

[farnsworth2008]

Thanks! This is from a complex system that I manage for my so it will take some time before I create a case that works in isolation. Once I have it isolated and reproducible I will update this item or if it is closed I will open a new issue.

The issue is intermittent but it does happen with some regularity. I think if I spend a bit of time I can create a reproducible situation.

I think is the cause is the following TF where when the value of var.name changes we often see the crash. I have a situation where jobs are renamed and when that happens I expect and often observe that TF will correctly delete the old job and then create the a new job with the new name. Sometimes however the TF crash occurs.

My workaround is to delete the job and then allow terraform to recreate it.

resource "kubernetes_job" "this" {
  wait_for_completion = true

  metadata {
    namespace = var.namespace
    name      = var.name
  }

  spec {
    template {
      metadata {}

      spec {
        dns_policy           = "Default"
        host_network         = true
        service_account_name = var.service_account_name

        affinity {
          node_affinity {
            required_during_scheduling_ignored_during_execution {
              node_selector_term {
                match_expressions {
                  key      = "eks.amazonaws.com/nodegroup"
                  operator = "In"
                  values   = var.nodegroups
                }
              }
            }
          }
        }

        container {
          command = var.command
          image   = var.image
          name    = var.name

          dynamic "env" {
            for_each = var.env

            content {
              name  = env.key
              value = env.value
            }
          }
        }
      }
    }
  }

  timeouts {
    create = var.create_timeout
  }

  lifecycle {
    ignore_changes = [spec]
  }
}
``

[jbardin]

Thanks @farnsworth2008, I think the ignore_changes there is the key. We may not be able to reproduce the exact state from the kubernetes provider, but the this config should give us enough clues to try and replicate the misbehavior locally.

[jbardin]

combining with #31536

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.