terraform plan -light

[ricardbejarano]

Terraform Version

Terraform v1.8.4
on darwin_amd64

Use Cases

Fixing plan (or rather, refresh) performance on large enough (in resource count) states.

Attempted Solutions

• Increasing parallelism only takes you so far until the providers rate-limit you.
• Disabling refresh works but comes with all sorts of consistency risks.
• Splitting one big state into multiple smaller states comes with all sorts of problems associated with it.

Proposal

Add a terraform plan -light flag such that only resources modified in code are targeted for planning.

This would reduce the scope of the pre-plan refresh down to the set of resources we know changed, which reduces overall plan times without the consistency risk of -refresh=false.

In order for Terraform to know what resources were modified in code, it would store a copy of the code in state every time it successfully applies. This would allow diff'ing "last-applied code" vs. "new code", the result of which is the scope of the next "light" plan.

Basically, -light tells Terraform to autogenerate the -target list from code changes.

More details: https://www.bejarano.io/terraform-plan-light/

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[Tbohunek]

THIS IS BIG! 🔝🚀🔥 Thanks @ricardbejarano!

Make Terraform faster and save CO2? Every Terraform user wants this, whether they know it or not. And I'll do all I can to propagate this idea across my organization and to anyone I bump into, including my mom, who now has a GitHub account!

Adding the benefits that I see:

• Reduce concurrency clashes
  Other plans cause locks, queues, or API depletion; other applies cause me to have to replan again.
• Reduce the likelihood of external changes breaking your change
  Unrelated data.azuread_user fails the whole plan because the user no longer exists.
• Reduce the likelihood of unrelated API problems breaking your change
  Just a week ago, the azurerm_security_center_contact API was down, making TFE apply impossible.
• Reduce Provider API limits exhaustion, especially in hub and spoke topologies
  azurerm and google providers often time out reads with API limits of ~25 reads/sec.
• Reduce CO2 emissions and compute resource waste
  Eating beef is being denounced while Terraform literally wastes CO2 and energy with pointless reads.
• Make VCS-driven TFE workflow feasible again
  VCS-driven TFE workflow cannot apply -target=what.you.want to circumvent the above issues.

If this gets implemented, every single TFE user will be grateful, knowingly or unknowingly.
When released, make them aware through a tooltip with credits. 😉

[jsimoni]

something similar to https://nx.dev/ci/features/affected ?

[igtsekov]

This idea can be extended to init as well.

[ricardbejarano]

BTW I've been doing this for a few months now and it is fantastic, if you want to have this feature before it gets developed:

terraform plan $(terraform plan -refresh=false -json | jq -r '.change.resource.addr|select(.!=null)|"-target="+.')

• .change.resource.addr is where the full resource addresses are in the JSON-formatted output
• select(.!=null) drops all null values (output lines that didn't contain the previous)
• "-target="+. prepends the -target= prefix to the resulting lines without the need to call sed, xargs or anything else

[ricardbejarano]

@jbardin @crw may I ask what's next for this? Shall I write the PR?

[crw]

@ricardbejarano thanks for the offer, however it is unlikely to be reviewed until the maintainers have considered the larger ramifications of such a change. There is interest in exploring this at some point in the future, I will keep you posted.