Module state migrations

[dunnill]

This feature does not relate to a specific version but Terraform in general, as I'm currently using v0.11.8 and looking to move towards v0.12.8.

Use-cases

Migrate from different terraform versions and between different modules. I have a number of v0.11 modules that run a number of infrastructure and want to upgrade them to v0.12. As this version reduces the number of modules / complexity and also improves customisation (within our acceptable limits). However I need a way to migrate the state file so it can read either 0.11 or 0.12 state files and save in 0.12 format. We push changes through our environments and as such need these 'data' changes to be pushed alongside it.

Attempted Solutions

The suggest approach is to use mv command. However this is a manual (and error prone) process so does not work in this context. Instead I have to create a duplicate stack, including separate CI/CD pipelines and state file. This allows me to create the stacks simultaneously and switch at a DNS level (then decommission when fully QA'd). This is messy and risks outages occurring (especially if there is a defect in the new version).

Proposal

Add a data model like layer where each module defines the output (in many ways this sort of exists as outputs). And also a translation / upgrade layer. This takes the data model layers of the old and new module(s) and translates from one to the other. Allowing state files to be upgraded. E.g. module-a1.EIP -> module-d2.EIP and module-b1.Instance -> module-d2.instance.

References

Could not see anything in the backlog already.

[teamterraform]

Hi @dunnill!

We'd love to hear a little more about what you are trying to do here. Terraform 0.12 can already read state snapshots created by Terraform 0.11, so we're assuming there's an additional subtlety in your situation where that isn't sufficient for what you're trying to do, but we're not sure from what you described here exactly what that is.

If you upgrade to Terraform 0.12 and run terraform apply with an existing configuration in an existing workspace, Terraform should read the existing Terraform 0.11 state snapshot, use it as the basis of the plan, and then write a new 0.12-format state snapshot as its result.

[dunnill]

Hi @teamterraform

The nuance is not strictly upgrade related. I have done this without changing terraform version but on a much lesser scale (and ended up murdering the entire stack to do so, which is not desirable). To put it simply, v0.12 is brilliant. So brilliant that many of my current modules are essentially defunct as they were created around the limitations of v0.11. But everyone complained that they were extremely hard to understand as I worked to the limits of v0.11 to achieve what I wanted (so were sometimes counter-intuitive).

Now v0.12 is here, I refactored these modules to take advantage of the new features. Vastly simplifying the modules and how they're used. But now I have the problem that the current state file has (illustrative):
DATA:

ASG {asg details}
ASG Scaling {common scaling details with flags to turn off/on specific scaling types/events etc}
to:
asg { asg details and [scaling event details] }

The problem seems fairly trivial (namely terraform mv ASG Scaling into ASG list). But the problem is using terraform mv is a very manual thing. We integrate these stacks in our development process which is fully automated. This means our options (that I know of):

• Create a totally new stack using 0.12 modules (ensuring names etc are unique and have new pipelines etc). Then scare everyone when you switch it over :P
• Don't bother - leave everything where it is and keep 0.11 modules ;-(
• Somehow integrate terraform mv into our pipeline. :-8

The latter option is very risky as we'd have to test for every environment the mv commands needed. Then add it to the deploy script, presumably with some "test" performed first to check if it has been upgraded already. Tbh - not sure how feasible it is and the risk is massive (e.g. corrupt state stack in live when trying to deploy urgent bug fix).

So it would be nice if we used a similar process that developers use in data base/model migration. Namely adding some representation that tells it how to upgrade safely - so either it is correctly upgraded or aborted. Hope that makes sense.

[apparentlymart]

Hello!

I think this request was eventually solved with the addition of moved blocks, which allow a module to carry with it information about how objects were renamed or moved over time so that Terraform can update the state automatically as part of a normal plan/apply round.

The initial incarnation of that feature only supported moving objects between resources of the same type, but more recently it grew the additional ability to migrate from one resource type to another as long as the target resource type's implementation (in its provider plugin) includes a migration rule to translate from the original state shape to the new state shape. Because that is relatively new and relies on provider support, it's not yet broadly available but will become more useful over time through additions to providers.

Provider changes are not developed in this repository, so if you have a request for a specific resource type migration then I'd suggest opening a feature request issue for it in the repository of the provider that contains the resource type you would be migrating to. (That's the codebase where the migration rule would need to be implemented.)

Since I think this request was eventually met in a past version of Terraform, I'm going to close this now. Thanks!