instances.Expander: A utility for encapsulating "count" and "for_each" concerns

[apparentlymart]

Our existing handling of count and for_each for resources is rather scattered across different parts of Terraform.

This PR represents some initial experimentation towards centralizing the expansion logic for resources and for modules (looking forward to future module count and for_each) to help the different parts of Terraform Core that deal with the repetition constructs to coordinate with each other in a more organized way.

The Expander type includes three different families of methods:

• Set* methods (SetResourceSingle, SetResourceCount, SetResourceForEach, SetModuleCount) are used to record the results of evaluating the count and for_each expressions in the configuration for later use by the Expand* family.

• Expand* methods (ExpandResource and ExpandModule) take the addresses of static (unexpanded) objects in the configuration and return all of the instance addresses that result from the expansions that were recorded by earlier calls to the Set* methods.

  Because modules can nest inside other modules, the Expand* methods fully expand all levels of the tree and so the number of instances for a particular object can grow exponentially for objects in deeper modules.

• Get*RepetitionData methods (GetResourceInstanceRepetitionData and GetModuleInstanceRepetitionData) take the absolute address of a particular module instance or resource instance identified by the Expand* family and return a decision about what values should appear for each.key, each.value, and count.index when evaluating the configuration for the identified instance.

Each family feeds into the next, so they must be called in a particular order for correct operation. The graph walk can take care of ensuring the correct ordering of calls so that data is always available before it is needed.

---

As an initial proof-of-concept I integrated instances.Expander in a rough way to replace some -- but not all -- of the scattered count and for_each logic for resources. This now explicitly registers that every module in the configuration is a singleton (count and for_each aren't supported there yet) and then registers the repetition mode of each resource, which then allows the DynamicExpand logic to just ask the expander what instances it ought to be creating.

---

This Expander thing is also intended to be responsible for deciding what goes in each.key, each.value, and count.index for expressions, but I wasn't able to wire that up just yet because the apply graph (which is pre-expanded with nodes representing resource instances, not resources) isn't interacting with expander quite right yet, and the codepaths that do expression evaluation all run in both the plan and apply phases.

In principle though, if the Expander is being populated properly on all walks, we would be able to get the values representing the current repetition using the GetResourceInstanceRepetitionData method, which takes an absolute resource address and returns an object giving the cty.Value to use for each.key, each.value and count.index.

---

If this were to be taken forward as the basis for module count and for_each, that would require us to do some rework on the way Terraform Core implements the plan walk so that it's clear that the nodes in the main plan graph represent configuration constructs belonging to unexpanded modules only (not to module instances), and add DynamicExpand to all of the nodes representing configuration constructs which calls into the Expander instance in order to find all of the module instances that the configuration construct is instantiated in. That would be a far more disruptive change, so I've not attempted that yet, but the new DynamicExpand implementation for resources is ready to deal with count and for_each on modules once the rest of the plan graph catches up.

---

This does appear to work and pass all the existing context tests without modification, but for the moment it's mainly intended as a conversation starter for whether this particular direction -- DynamicExpand on everything -- feels good for module count and for_each.

[jbardin]

The module expander work is being based off this branch. Since this applies cleanly to master I think we can merge it now to reduce the diff and make rebasing easier.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.