Terraform fails to refresh the state properly when an item from a resource map is deleted outside of terraform.

[MarkKharitonov]

Terraform Version

Terraform -version
Terraform v0.12.23

Your version of Terraform is out of date! The latest version
is 0.12.24. You can update by downloading from https://www.terraform.io/downloads.html

Terraform Configuration Files

locals {
    locations = toset(["a", "b"])
}

resource "local_file" "instance" {
    for_each = local.locations

    content  = each.value
    filename = "${path.module}/${each.value}.txt"
}

output "primary_filename" {
    value = local_file.instance["a"].filename
}

Expected Behavior

terraform apply succeeds

Actual Behavior

terraform apply fails

Steps to Reproduce

1. terraform init
2. terraform apply
3. del a.txt
4. terraform apply

Additional Context

C:\work\test> dir


    Directory: C:\work\test


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        4/15/2020  11:47 PM            280 main.tf


C:\work\test> terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "local" (hashicorp/local) 1.4.0...

...
C:\work\test> terraform apply -auto-approve
local_file.instance["b"]: Creating...
local_file.instance["a"]: Creating...
local_file.instance["a"]: Creation complete after 0s [id=86f7e437faa5a7fce15d1ddcb9eaeaea377667b8]
local_file.instance["b"]: Creation complete after 0s [id=e9d71f5ee7c92d6dc9e92ffdad17b8bd49418f98]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

primary_filename = ./a.txt
C:\work\test> del .\a.txt
C:\work\test> terraform apply -auto-approve
local_file.instance["a"]: Refreshing state... [id=86f7e437faa5a7fce15d1ddcb9eaeaea377667b8]
local_file.instance["b"]: Refreshing state... [id=e9d71f5ee7c92d6dc9e92ffdad17b8bd49418f98]

Error: Invalid index

  on main.tf line 13, in output "primary_filename":
  13:     value = local_file.instance["a"].filename
    |----------------
    | local_file.instance is object with 1 attribute "b"

The given key does not identify an element in this collection value.

References

• https://discuss.hashicorp.com/t/what-is-wrong-with-this-trivial-usage-of-local-variable/7835
• https://stackoverflow.com/questions/61218173/why-terraform-is-unable-to-compute-a-local-variable-correctly-in-the-following-t

[danieldreier]

Hi @MarkKharitonov this is a great bug report! I have reproduced it on 0.12.25 on my OS X Catalina machine. Thank you for coming up with a simple reproduction case. I'll get this on our backlog as a valid bug.

To make reproduction trivial for the engineer who picks this up, I've created a reproduction case in https://github.com/danieldreier/terraform-issue-reproductions/tree/master/24883 where you just run run.sh.

[pselle]

Thanks for this helpful report.

Unfortunately, we won't be able to fix this in the near-term, as this is related to larger issues related to the refresh phase. I'll describe some of the why of the problem for future readers and to inform our work as we seek to reconcile the refresh phase so it is more reliable in reconciling drift.

In the config presented here, we have an output depending on a resource instance. In the refresh phase, Terraform waits for that resource to be refreshed before evaluating the output depending on that resource. When resource is refreshed, if an instance is removed, that instance is removed from state.

This means that for any resource with count or for_each, if one or more (but not all) instances of the resource is removed outside of Terraform, the state for that resource is updated such that the resource still exists, but the instance we are looking for does not (because it was deleted). When Terraform is ready to evaluate the output, there are no instances with those keys in the state, thus the error described here.

It is not recommended to remove resources outside of Terraform, but instead to use Terraform to manage your infrastructure. However, we recognize that the refresh phase is meant to address exactly this problem! (reconciling the resources in the wild with state) So we are motivated to fix this, but it's not likely to happen soon.

As an interim path, for you or future readers, one can use terraform state rm 'local_file.instance["a"]' to manually update the state, until we are able to address these underlying causes.

[jbardin]

I have confirmed this is fixed in 0.14.
There is no longer a separate refresh phase, so the output can be correctly evaluated against the plan for filename.instance.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.