Add new terraform regenerate command

[garysassano]

Use Cases

I'm interested in a command that combines removed (v1.7) and import (v1.5) blocks functionalities into a single operation. This command, potentially named terraform regenerate, would initiate a Terraform destroy plan for the entire workspace as if each resource was placed inside a removed block. It would then automatically extract the Import ID for each resource, take those IDs and feed them into import blocks, thereby generating a new Terraform state file from the existing one.

The primary benefit of this approach is that it gets rid of all existing moved blocks, while also dissolving all previously established modules. Essentially, it allows for an easy refactoring of your Terraform repository, moving from a complex and cluttered state into a streamlined structure. By the end of this process, you'd consolidate all resources from the previous configuration into a single .hcl file. For instance, if your workspace initially consisted of 10 different modules and 20 directories, generating a total of 100 resources, you would now manage the same 100 resources within a singular .hcl file, which you can then start to split at occurence.

Attempted Solutions

see above

Proposal

No response

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[garysassano]

We also welcome additional use case descriptions.

The primary application is for refactoring; thus, the command might actually be named terraform refactor. I saw situations where it was decided to reimport all resources managed by Terraform (using import blocks) and subsequently remove the old .tfstate file, a process which was manually intensive. This could be fully automated by extracting the import IDs from the Terraform plan generated by removed blocks, and then incorporating those into new import blocks.

The idea is that you don't want to create nor destroy anything, just take all existing resources deployed by your workspace and put them under a single .hcl file, so that you can then start restructuring the repo differently.

[apparentlymart]

Thanks for sharing this proposal, @garysassano.

You've presented it as a new command, which is a potentially-viable way to expose it, but I think for initial discussion I'd rather focus on the details of what this operation includes rather than how it is called or what it is named, since I hope that pinning down exactly what this operation would be doing will then lead to clearer options about how to offer it, whether that be as a new command, a new planning option, or something else entirely.

---

I'd like to start by restating what I understood from your proposal in terms of the concepts in the resource instance change lifecycle.

The idea as I understood it is to visit each resource instance that is either declared in the configuration or already tracked in the state, and for each one:

1. Determine an "import id" that could be used to refer to the same object.
2. Call ImportResourceState with that ID to produce an Import Stub State, and then pass that to ReadResource to calculate a new "Prior State", which for the sake of this discussion I'll call the "Replacement State".
3. Assuming that all of the above succeeds, replace the Prior State with the Replacement State, discarding the Prior State entirely.

The most significant challenge in the above is in step 1, because Terraform currently has no way to transform a prior state into an import ID. Providers are in full control of both their import IDs and their state data formats, and so Terraform currently relies on the ImportResourceState operation to transform an import ID into an Import Stub State which can then be elaborated into a new "Prior State" to import. There is not currently any provider operation to perform the opposite mapping from Prior State to Import ID. We could add such a thing, but then we'd need to wait for it to be implemented in enough providers across enough resource types for it to be likely that there's sufficient coverage for an arbitrary existing Terraform configuration.

If that problem were solved then I think the rest of it is relatively straightforward, although there are still some minor edge-cases to consider:

• What should Terraform do if there's a resource instance declared in the configuration that isn't currently represented in the state?

  Ignoring it entirely seems like the safest option, since then a subsequent terraform plan would propose to create it just as it would have before performing this transformation.

• What should Terraform do if there's a "deposed" object present in the state?

  Those are there to represent that a create_before_destroy object succeeded creating but failed deletion and so there's a remnant remote object that still needs to be cleaned up, and so today the only possible operation against a deposed object is to destroy it. However, we could potentially try to run the same sequence of steps against it to "re-import" the object and then replace the deposed object state with the new object.

• What should Terraform do if there's a resource instance tracked in the state that isn't mentioned anywhere in the configuration? (Let's assume there are no import, moved, or removed blocks referring to it)

  In this case I think the only options are to discard it entirely (as if there had been a removed block with destroy = false) or to leave it in the state at its old address. The former would cause the object to no longer be tracked by Terraform at all, while the later would not achieve the desired goal of reconciling the state with the new configuration.

  Perhaps it would be best to return an error in this case, since you explained the intent of this command as to force reconciling the existing state with the current configuration, in which case I would think I'd prefer to see an error telling me I need to give Terraform more information to achieve that reconciliation, rather than it just quietly producing an incomplete result.

---

Having run that thought experiment in my head, I'm left with a question: why is this extra step necessary, vs. just running terraform apply? That command's job is to try to reconcile desired state with actual state so that the two are converged, it should be able to achieve that in all of the same situations where this hypothetical command would also achieve that.

The only extra thing this new command would do is losing any information that the provider can't reconstruct based on the API response. Re-importing the existing object is at best a no-op (the new state is identical to the old), but is often messier and lossier than that, causing the provider to lose track of anything the remote API can't track.

Therefore if there's a situation that terraform apply can't currently deal with, I'd expect to prefer to add something new to terraform apply to make that situation work, vs. adding a new mode that would have the side-effect of throwing away information that might be important, and might therefore leave the system in a worse state than it started. 🤔