Option to skip refreshing resource instances whose configuration hasn't changed since the last apply

[ricardbejarano]

OpenTofu Version

OpenTofu v1.7.1
on darwin_amd64

Use Cases

Fixing plan (or rather, refresh) performance on large enough (in resource count) states.

Attempted Solutions

• Increasing parallelism only takes you so far until the providers rate-limit you.
• Disabling refresh works but comes with all sorts of consistency risks.
• Splitting one big state into multiple smaller states comes with all sorts of problems associated with it.

Proposal

Add a tofu plan -light flag such that only resources modified in code are targeted for planning.

This would reduce the scope of the pre-plan refresh down to the set of resources we know changed, which reduces overall plan times without the consistency risk of -refresh=false.

In order for OpenTofu to know what resources were modified in code, it would store a copy of the code in state every time it successfully applies. This would allow diff'ing "last-applied code" vs. "new code", the result of which is the scope of the next "light" plan.

Basically, -light tells OpenTofu to autogenerate the -target list from code changes.

More details: https://www.bejarano.io/terraform-plan-light/

References

No response

Hello and thank you for this issue! The core team regularly reviews new issues and discusses them, but this can take a little time. Please bear with us while we get to your issue. If you're interested, the contribution guide has a section about the decision-making process.

[orbitz]

I think you could avoid keeping a copy of the code in the state but instead store the hash of a normalized representation of the resource. This would make it more robust to refactorings and reduce state size and complexity of the comparison.

[ricardbejarano]

Yup, I'm intentionally vague about the implementation specifics on the state side because I don't really care that much, but here's what I say in the article:

At minimum, a comparable derived from each resource’s code definition, like a checksum of its attribute map that can be compared to the new code’s resource attribute map checksum.

[cam72cam]

We discussed this as part of the core team and are hoping that someone in the community can create a tool that does this automatically. I believe this can be roughly implemented by running tofu plan -refresh=false -json, parsing the json output for changed resources, and building a -target list.

pseudocode:

targets=$(tofu plan -refresh=false -json | jq '.some filter' | sed 'some expr')
echo "Filtering to targets $targets"
tofu apply $targets

Implementing this directly in OpenTofu is a bit tricky due to legacy design decisions that will take time to refactor. In the mean time, a script like the above could help prove out the concept and show that the technique is useful/valid

[orbitz]

@cam72cam Thank you for the idea. We've implemented this in our TACOS and will be testing it out to see what the experience is like. Will report back.

[ricardbejarano]

For a temporary proof of concept, your (@cam72cam's) method is better than what I had tested myself. I had written a quick POC that intercepts (only works with local state) state and it writes the hash of the sorted attribute map to each resource, then removes it before passing state back to tofu.

It is simpler in that it implements my proposal as is and only requires one run of tofu, but for a POC it only works with local state, doesn't support resources in modules, or changes in variables, etc.

Whereas running a -refresh=false plan does it all.

Great idea!

I hope it serves to get traction on this!

[ricardbejarano]

Sharing the one-liner I made to add this to my pipeline:

tofu plan $(tofu plan -refresh=false -json | jq -r '.change.resource.addr|select(.!=null)|"-target="+.')

• .change.resource.addr is where the full resource addresses are in the JSON-formatted output
• select(.!=null) drops all null values (output lines that didn't contain the previous)
• "-target="+. prepends the -target= prefix to the resulting lines without the need to call sed, xargs or anything else

[apparentlymart]

I did some light prototyping today about just the idea of taking a digest of the "desired state" (configuration value) for a resource instance.

I found two interesting things that seem relevant to a potential RFC for this feature:

• Our two existing commonly-used serialization formats for dynamic values -- JSON and MessagePack -- both explicitly serialize null attributes rather than omitting them, with the intention that the serialized object is more explicit about the schema it was created from and so we can detect more potential problems if things become inconsistent.

  However, I think a digest format for this application ought to have the opposite behavior of treating attributes as not present at all if they are currently assigned null, because otherwise if a future provider version adds a new optional attribute to a resource type then that would immediately invalidate every existing checksum, even though no existing configuration would be setting that (previously-invalid) attribute.

  An RFC about this idea ought to be very specific about how we "remember" the desired state associated with each object. If that involves taking a digest of some serialized form of the desired state object, the RFC must say exactly how that serialized form is constructed an in particular specify how null values are to be handled. (As noted above, my current sense is that it should ignore them entirely, but further research might reveal drawbacks to that which I've not yet considered.)

• What we typically think of as the "refresh phase" also includes a non-optional call to the provider to potentially "upgrade" the existing resource data to the latest schema. This must be mandatory because OpenTofu does not have access to any previous versions of the schema and so it cannot decode state data serialized against an earlier incompatible schema version without the provider's help.

  The upgrade process is not an inherent problem in itself -- it affects the prior state, rather than the configuration digest -- but that upgrade process could potentially cause the new data to be a significantly different shape to the data we previously used to calculate the digest, and some existing providers have their "upgrade" step written with the assumption that it will typically be followed by "refresh"¹ and so upgrading without refreshing might leave the prior state worse than it would've been when refreshing, causing the provider to report changes that don't actually need to be made.

  Given that, I think an RFC for this should consider whether and how the upgrade process should interact with the idea of skipping refresh dynamically based on the configuration digest. For example, we could assert that if the prior state's schema version is different than the provider's schema version then it's likely that the upgrade process changed something substantial and so we'd better just conservatively force refreshing anyway in that case. (This would only occur on the first run after upgrading a provider, since that's when a new schema version and thus new upgrade rules can potentially come into play.)

Footnotes

1. Some readers are probably thinking "but I can already do -refresh=false, so surely this is an invalid assumption?"

   You are correct in that assessment: providers shouldn't make this assumption, but in practice several existing resource types in existing providers do, relying on the fact that -refresh=false is a rather obscure option that most folks don't use. If we add something that causes the refresh phase to be skipped more often then that could make the consequences of this faulty assumption more significant. ↩

[ricardbejarano]

@apparentlymart may I ask what the next steps are?
You mention an RFC, I'm open to writing it.

How do we push this?