terraform test crash on multiple run blocks in file

[matt-FFFFFF]

Terraform Version

❯ t version
Terraform v1.6.0-beta1
on linux_amd64
+ provider registry.terraform.io/azure/azapi v1.9.0
+ provider registry.terraform.io/hashicorp/azurerm v3.71.0

Terraform Configuration Files

# HINT: make sure to run `terraform init` in this directory before running `terraform test`

# Default variable values (can be overridden inside the `run` block)
variables {
  subscription_alias_enabled = true
  subscription_alias_name    = "my-subscription-alias"
  subscription_billing_scope = "/providers/Microsoft.Billing/billingAccounts/0000000/enrollmentAccounts/000000"
  subscription_display_name  = "test-subscription-alias"
  subscription_workload      = "Production"
}

# Stop Terraform moaning about the provider not being configured
provider "azurerm" {
  features {}
}


# Example of basic test checking resource values against input variables
run "basic_valid" {
  command = plan

  assert {
    condition     = azurerm_subscription.this[0].alias == var.subscription_alias_name
    error_message = "Subscription alias name is not correct"
  }

  assert {
    condition     = azurerm_subscription.this[0].billing_scope_id == var.subscription_billing_scope
    error_message = "Subscription billing scope not correct"
  }

  assert {
    condition     = azurerm_subscription.this[0].subscription_name == var.subscription_display_name
    error_message = "Subscription name is not correct"
  }

  assert {
    condition     = azurerm_subscription.this[0].workload == var.subscription_workload
    error_message = "Subscription workload is not correct"
  }
}

# Example of chekiang failure on variable validation
run "expect_failure_invalid_billing_scope" {
  command = plan

  # Create an invalid billing account scope
  variables {
    subscription_billing_scope = "/providrs/Microft.Billing/billingAccouts/0000000/enrollmencounts/000000"
  }

  expect_failures = [
    var.subscription_billing_scope,
  ]
}

Debug Output

https://gist.github.com/matt-FFFFFF/629b3fa191f3618b07c0d32a628c01c8

Expected Behavior

Not crash!

Actual Behavior

Crashed when both run blocks are in the test file.

Either one works on its own.

Steps to Reproduce

1. terraform init
2. terraform test

Additional Context

Link to sub-module: https://github.com/Azure/terraform-azurerm-lz-vending/tree/spike/terraformtest/modules/subscription

References

No response

[liamcervante]

Hi @matt-FFFFFF! Thanks for this and sorry for the crash.

Do you have the crash stack trace available to share? I will try and reproduce this as well, but that might speed things up.

Thanks!

[matt-FFFFFF]

Hey @liamcervante

Thanks for the response - here it is

runtime error: invalid memory address or nil pointer dereference
goroutine 45 [running]:
runtime/debug.Stack()
        /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
        /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/debug/stack.go:16 +0x13
github.com/hashicorp/terraform/internal/logging.PanicHandler()
        /home/runner/work/terraform/terraform/internal/logging/panic.go:58 +0x13b
panic({0x2972c00?, 0x4bad6b0?})
        /opt/hostedtoolcache/go/1.21.0/x64/src/runtime/panic.go:914 +0x21f
github.com/hashicorp/terraform/internal/backend/local.(*TestFileRunner).ctx.func1(0xc00054fd08, 0x0?)
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:1348 +0x5ab
github.com/hashicorp/terraform/internal/backend/local.(*TestFileRunner).ctx(0xc00054fd08, 0xc000892b90, 0xc000892be0, 0xc00054f150?)
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:1368 +0x31a
github.com/hashicorp/terraform/internal/backend/local.(*TestFileRunner).buildInputVariablesForTest(0xc00054fd08, 0xc000892b90, 0xc000892be0, 0xc0005fed20)
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:941 +0x779
github.com/hashicorp/terraform/internal/backend/local.(*TestFileRunner).plan(0xc001b4fd08, 0xc0005fed20, 0xc0008817d0, 0xc000892b90, 0xc000892be0, {0x0, 0x0, 0x0})
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:596 +0x288
github.com/hashicorp/terraform/internal/backend/local.(*TestFileRunner).run(0xc001b4fd08, 0xc000892b90, 0xc000892be0, 0xc0008817d0, 0xc0005fed20)
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:346 +0x733
github.com/hashicorp/terraform/internal/backend/local.(*TestFileRunner).Test(0xc001b4fd08, 0xc000892be0)
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:282 +0x4be
github.com/hashicorp/terraform/internal/backend/local.(*TestSuiteRunner).Test(0xc000274a80)
        /home/runner/work/terraform/terraform/internal/backend/local/test.go:111 +0x349
github.com/hashicorp/terraform/internal/command.(*TestCommand).Run.func1()
        /home/runner/work/terraform/terraform/internal/command/test.go:186 +0x7e
created by github.com/hashicorp/terraform/internal/command.(*TestCommand).Run in goroutine 1
        /home/runner/work/terraform/terraform/internal/command/test.go:180 +0xbed

[liamcervante]

Thanks for that, super helpful!

So the crash is originating here: https://github.com/hashicorp/terraform/blob/main/internal/backend/local/test.go#L1348

It's caused by the outputs you have defined that are returning null. Terraform is just not including them in the set of outputs and the test file is expecting them to be there as it is looking in the config for the true set of outputs.

This is definitely a bug, so we'll get it fixed for the next beta release. Thanks for finding it.

[liamcervante]

Unfortunately, I don't have a good workaround for you here. The only thing I can think of is ensuring that in your tests that the null results in your outputs are not triggered if at all possible.

[matt-FFFFFF]

Thanks for confirming. Any idea why it doesn't happen with only one of the run blocks enabled?

Appreciate the speedy response :)

[liamcervante]

Yes - Terraform attempts to make the outputs of all the previous run blocks available to each run block as it executes. If there is only a single run block then it never has any previous run blocks to process, and so it never encounters the null output values that are causing the crash.

[liamcervante]

This should be fixed in the next beta release!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.