New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documentation for state file versioning considerations #19290
Comments
Hi @edahlseng! I think you're right that there isn't a single place where all of the details here are documented right now. For now, I'll try to summarize the situation here and then I'll label this as a documentation issue to remind us to write something more explicit about it the docs in future. A key piece of context here is that the state contains a mixture of data generated by Terraform Core and data generated by provider plugins. Since provider plugins are on a separate release cycle than Terraform Core, there are two different version handling mechanisms here to deal with those two problems, though both have similar high-level behavior. Terraform Core controls the overall structure of the state file, the syntax of which is versioned explicitly using its own version number. The current state file format as I write this is version 3, and Terraform v0.12 will introduce version 4. Terraform Core generally knows how to upgrade forwards through state format versions, but an older version of Terraform cannot downgrade a state to read it. The version number I covered in the previous paragraph represents the syntax of the state file format. The file can also include some semantic details that emerged from the behavior of Terraform in a particular version, and so as a safety mechanism to avoid unpredictable behavior we additionally require the use of a version of Terraform at least as new as the one that most recently updated the state. As Terraform's behavior stabilizes in future versions it is likely that this constraint will be relaxed and we will use only the file format version number, but the current implementation is conservative to reduce the risk of problems. Alongside both of those mechanisms, each separate resource instance in the state has data that is structured in a way decided by the provider, and so this structure has its own per-resource-type version number that the provider manages. If a breaking change is made to the state format for a particular resource type then the next provider release will increase this version number and include some logic to upgrade from prior versions. Just as with the core format version, upgrades can only move forwards and so once you have run
|
Thanks for the helpful answer @apparentlymart! I'm starting to understand how Terraform handles the state file much better now. To make sure I'm fully grokking everything, are the following statements correct?
|
That is very close! Two refinements: The version on Terraform stored is the most recent one that applied changes, not the one that started it. Also, any command that modifies the state may upgrade one of these versions. |
Thanks for the clarifications @apparentlymart! |
@apparentlymart Just wanted to check in on this, since it's been a few years. I found a bit about state here: https://www.terraform.io/docs/language/state/index.html but wasn't sure if we wanted to circle back to this to update the docs further, or if this has been resolved and we can close. Thank you! |
This seems to be, so far, the only definitive thing I've found about the version fields in the state file. I would say it's worth a mention in the documentation, because it's not entirely clear there (and a language reference should be thorough wherever possible). |
noted! Thank you both. I'm going to make a ticket for myself for this work. Behind the scenes, @apparentlymart and I were discussing a possible revamp of the entire state section. I'm not sure I can fit that in this quarter, but I'm going to add it to the documentation backlog so we make a note of it and can plan it in the next couple of months. :-) |
The main thing that changed since my original writeup above is that we did subsequently reach the point I was alluding to where Terraform's behavior is fixed enough that we can rely only on versioning of the state snapshot syntax and no longer check that a snapshots was created by the current or an earlier version of Terraform CLI. The current situation is therefore a little simpler:
I still don't have a good sense of how best to incorporate this into the docs, so again I'm leaving this here in the hope that it's useful for those who are interested. @laurapacilio, let's talk about this again some more when you have some time to dig in! |
I have a general question about how Terraform handles state file versions. I couldn't find this information anywhere in the online docs; I hope I haven't just missed it! Please point me in the right direction if I should be asking this somewhere else!
I notice that if I create a remote state file in one version of Terraform (e.g. on my local computer, using v0.11.8), and then try to run a plan/apply against that state file in a lower version of Terraform (e.g. as part of a CI build, using v0.11.1), Terraform complains with an error message along the lines of "this state file was created with a newer version of Terraform." This suggests to me that state files have an explicit version stored with them. I don't understand the full implications of this, however. For example, are we only limited with using older versions of Terraform? Can we always use a newer version of Terraform to operate on a given state file? Is there a process for upgrading the version of the state file? Would there be any benefit in doing such an operation, or is that a silly idea to even consider?
The text was updated successfully, but these errors were encountered: