terraform test: produces invalid warning during cleanup phase

[liamcervante]

Terraform Version

1.7.5 / 1.8.1

Terraform Configuration Files

# main.tf

variable "input" {}

resource "tfcoremock_simple_resource" "resource" {
    string = var.input
}

# main.tftest.hcl

run "test" {
    variables {
        input = "Hello, world!"
        validation = "Hello, world!"
    }

    assert {
        condition = tfcoremock_simple_resource.resource.string == var.validation
        error_message = "bad!"
    }
}

Debug Output

N/A

Expected Behavior

> terraform test
main.tftest.hcl... in progress
  run "test"... pass
main.tftest.hcl... tearing down
main.tftest.hcl... pass

Success! 1 passed, 0 failed.

Actual Behavior

> terraform test
main.tftest.hcl... in progress
  run "test"... pass
main.tftest.hcl... tearing down
╷
│ Warning: Value for undeclared variable
│ 
│   on main.tftest.hcl line 5, in run "test":
│    5:         validation = "Hello, world!"
│ 
│ The module under test does not declare a variable named "validation", but it is declared in run block "test".
╵
main.tftest.hcl... pass

Success! 1 passed, 0 failed.

Steps to Reproduce

1. terraform init
2. terraform test

Additional Context

No response

References

No response

[liamcervante]

terraform test is re-performing the validation on variables during the cleanup, when this was already validated during the actual execution of the test.

The particular problematic validation is that we try to make sure that variables defined within run blocks are actually used within the execution of that run block. We print a warning if we find an unused variable. In the example test file, we provide values for two variables input and validation. input is used within the actual configuration, while validation is used within the assertion block. We shouldn't produce a warning for either of these, as they are being used.

From the output, you can see that the run block executes with no warnings and then we get a warning after the tearing down message is printed. The warning is printed only about the validation variable. From this, we can see that terraform test is not properly calculating the actual required variables. It is including the variables in the configuration as being required, but not the variables from the assertions.

Indeed, you can see that during the run(...) function we are collecting and providing the references from the run block:

terraform/internal/backend/local/test.go

Lines 427 to 434 in 4487751

	references, referenceDiags := run.GetReferences()
	run.Diagnostics = run.Diagnostics.Append(referenceDiags)
	if referenceDiags.HasErrors() {
	run.Status = moduletest.Error
	return state, false
	}
	variables, variableDiags := runner.GetVariables(config, run, references)

But, during the destroy(...) function we are not including the providing the references from the run block:

terraform/internal/backend/local/test.go

Line 640 in 4487751

variables, variableDiags := runner.GetVariables(config, run, nil)

One potential solution is to modify the GetVariables(...) function so that it only conditionally adds the warning during the run(...) function but doesn't during the destroy(...) function. We don't need to repeat the warnings, so only adding them during the actual test execution is fine.

Another potential solution would be to move the source of the warning from here into the FilterVariablesToModule function. Then the warnings from that function would simply be discarded during the destroy(...) function, as happens at the moment anyway. We'd need to tweak the function so that it accepts the references that are currently provided to GetVariables, and we can remove the relevantVariables check from the GetVariables function.

I'd be happy with either of these solutions.

[MicahKimel]

@liamcervante Are nil references only used in the GetVariables() function during the destroy(...) function? If so would the following solution work?

Code to update

// Do not display warnings during cleanup phase
if references != nil {
    diags = diags.Append(&hcl.Diagnostic{
    	Severity: hcl.DiagWarning,
	Summary:  "Value for undeclared variable",
	Detail:   fmt.Sprintf("The module under test does not declare a variable named %q, but it is declared in run block %q.", name, run.Name),
	Subject:  expr.Range().Ptr(),
    })
}

If this is a good solution let me know and I'll make a pull request.