Documentation Update - Slightly clearer description of 'Refresh' and 'Plan' execution

[manmedia]

Hello,

Current Terraform docs suggest the following on Refresh and Plan respectively:

The terraform refresh command is used to reconcile the state Terraform knows about (via its state file) with the real-world infrastructure. This can be used to detect any drift from the last-known state, and to update the state file.

This does not modify infrastructure, but does modify the state file. If the state is changed, this may cause changes to occur during the next plan or apply.

The terraform plan command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

The optional -out argument can be used to save the generated plan to a file for later execution with terraform apply, which can be useful when running Terraform in automation.

If Terraform detects no changes to resource or to root module output values, terraform plan will indicate that no changes are required.

It's not well-explained on the above that "Plan" will not do a real refresh on the local/remote state files. It does the refresh in-memory. Therefore, an explicit "Refresh" is required before the plan.

Reason

1. Suppose we have a Gitlab CI/CD file which is mean to run terraform plan on a feature branch before doing terraform apply -auto-approve- this is a typical code review situation.
2. When it's approved, the apply command is invoked - there is a risk that the state has not been updated correctly.

It's therefore, required to do an explicit state refresh before the plan to have a 100% like-to-like behaviour for plan+apply. After all, plan is meant to be dry-run of apply. But this doesn't happen in real scenario. Therefore, I am requesting that the documentation should be updated to ensure that (at least) a cautionary statement is added for "Refresh" or "Plan" suggesting the above.

Regards,

[apparentlymart]

Hi @manmedia,

It's true that terraform plan and terraform apply don't directly update the state, but the result of the refreshing is considered part of the plan and so if you apply the plan Terraform will save that updated state as part of that work. Therefore you should never need to run terraform refresh in the normal Terraform workflow; it's unsafe in a situation where you've got your provider credentials misconfigured in a way that would make it seem like none of your objects exist, and we continue to support it only for backward compatibility with workflows built around earlier versions of Terraform.

If you see terraform apply not saving changes to the state that terraform refresh would've detected then I'd love to hear more about that, because that sounds like a bug rather than the intended behavior.

Funnily enough, I'm currently working on some changes over in #28297 which I think will make the relationships between these commands clearer as part of the documentation update. That'll eventually also include #26921, which will make it explicit in the UI when Terraform detects a difference during refresh and that you can apply the result to accept those changes.

In particular, the updated documentation for terraform refresh will present it as being a shorthand for the following command, which uses a new option that #28297 aims to add:

terraform apply -refresh-only -auto-approve

That's effectively what terraform refresh is today, and so this new -refresh-only option will provide a way to get the effect of terraform refresh without the risk I mentioned above of misconfigured credentials leading to an incorrect refresh (because you'll get an opportunity to review the detected changes before committing them as a state snapshot).

[manmedia]

@apparentlymart Understood. But here is the problem:

1. My team manages our infrastructure using Terraform on a provider - say AWS.
2. Some group (in the name if security bla bla) goes and invokes their executive privilege and screws up the resource configuration from web console - which the developers/elevated users cannot do - because console change has been blocked.
3. Result = state has messed up.
4. We MUST do a terraform refresh - at least - before/during terraform validate ->

The above is possible, and can inadvertently destroy resources. Because super users can do things as and when they please. But we can update the state with terraform refresh. I know the real fix is to update your CI/CD definition e.g. gitlab/jenkins file - but would you agree that this case should be documented with examples?

[apparentlymart]

Does something unwanted happen if you don't run terraform refresh, and just run terraform apply as normal? In that case, Terraform should detect and react to the changes made elsewhere, and create a new state snapshot including those once you accept the proposed changes.

Once we include something like the change I prototyped in #26921, Terraform will also announce that it has detected those changes, allowing you to find out that they occurred without this other meddling team having to inform you first. But that's just a cosmetic change: Terraform is already tracking those changes internally and will record them once you apply the plan, it just does so quietly today.

[manmedia]

Yes it does. An example: 1. In my Terraform plan (before doing a forced refresh), I saw 3 added, 0 change, 1 replace. 2. After I did a forced `terraform refresh` and then `terraform plan` - it mentioned 19 to add, 0 to change, 0 to replace. Clearly - it says something during plan, but after refresh it shows something else. I think where I am too demanding is the behaviour. Terraform is nothing but a "Git" for Infrastructure. If you check git, after every `git fetch --prune` it cleans up the local cache and a following `git rebase` rebases off of all recent changes made in remote. Terraform should allow doing something like that, if not exactly like that. But anyway, feel free to think about my suggestion and improve things if that suits your roadmap :) And we are looking forward to your -refresh-only flag changes to be released.

…

On Tue, 13 Apr 2021 at 01:27, Martin Atkins ***@***.***> wrote: Does something unwanted happen if you *don't* run terraform refresh, and just run terraform apply as normal? In that case, Terraform should detect and react to the changes made elsewhere, and create a new state snapshot including those once you accept the proposed changes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28342 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQC3KHEMMNZMAYGJ6J5OTDTIOFWNANCNFSM422BQDVA> .

[pll]

@apparentlymart

In particular, the updated documentation for terraform refresh will present it as being a shorthand for the following command, which uses a new option that #28297 aims to add:

terraform apply -refresh-only -auto-approve

That's effectively what terraform refresh is today, and so this new -refresh-only option will provide a way to get the effect of terraform refresh without the risk I mentioned above of misconfigured credentials leading to an incorrect refresh (because you'll get an opportunity to review the detected changes before committing them as a state snapshot).

The current docs now claim the -refresh-only option exists.

https://www.terraform.io/docs/cli/commands/refresh.html

Yet in 0.15.3, they don't exist:

$ terraform -v
Terraform v0.15.3
on darwin_amd64
$ terraform apply -h
Usage: terraform [global options] apply [options] [PLAN]

  Creates or updates infrastructure according to Terraform configuration
  files in the current directory.

  By default, Terraform will generate a new plan and present it for your
  approval before taking any action. You can optionally provide a plan
  file created by a previous call to "terraform plan", in which case
  Terraform will take the actions described in that plan without any
  confirmation prompt.

Options:

  -auto-approve          Skip interactive approval of plan before applying.

  -backup=path           Path to backup the existing state file before
                         modifying. Defaults to the "-state-out" path with
                         ".backup" extension. Set to "-" to disable backup.

  -compact-warnings      If Terraform produces any warnings that are not
                         accompanied by errors, show them in a more compact
                         form that includes only the summary messages.

  -lock=true             Lock the state file when locking is supported.
  -lock-timeout=0s       Duration to retry a state lock.

  -input=true            Ask for input for variables if not directly set.

  -no-color              If specified, output won't contain any color.

  -parallelism=n         Limit the number of parallel resource operations.
                         Defaults to 10.

  -state=path            Path to read and save state (unless state-out
                         is specified). Defaults to "terraform.tfstate".

  -state-out=path        Path to write state to that is different than
                         "-state". This can be used to preserve the old
                         state.

  If you don't provide a saved plan file then this command will also accept
  all of the plan-customization options accepted by the terraform plan command.
  For more information on those options, run:
      terraform plan -help

Are the docs ahead of the releases?

Thanks.
Paul

[DanyC97]

@apparentlymart i'll echo what @pll said, i got confused by the docs. And the situation is a bit complicated because TF docs are not bound to a spefcific release and so i had to bounce here to understand or figure out on which v the new flag is available.

Would appreciate if either docs are aligned or maybe a note is being added to say "is coming soon"