Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upinstances.Expander: A utility for encapsulating "count" and "for_each" concerns #23462
Conversation
This package aims to encapsulate the module/resource repetition problem so that Terraform Core's graph node DynamicExpand implementations can be simpler. This is also a building block on the path towards module repetition, by modelling the recursive expansion of modules and their contents. This will allow the Terraform Core plan graph to have one node per configuration construct, each of which will DynamicExpand into as many sub-nodes as necessary to cover all of the recursive module instantiations. For the moment this is just dead code, because Terraform Core isn't yet updated to use it.
When ModuleInstanceStep values appear alone in debug messages, it's easier to read them in a compact, HCL-like form than as the default struct printing style.
This is not used yet, but in future commits will be used as a "blackboard" to centrally aggregate the information pertaining to expansion of resources and modules (using "count" or "for_each") to help ensure consistent treatment of the expansion process during a graph walk. In practice this only really makes sense for the plan walk, because the apply walk doesn't do any dynamic expansion.
This returns the address of the module that the module instance is an instance of.
This is a minimal integration of instances.Expander used just for resource count and for_each, for now just forcing modules to always be singletons because the rest of Terraform Core isn't ready to deal with expanding module calls yet. This doesn't integrate super cleanly yet because we still have some cleanup work to do in the design of the plan walk, to make it explicit that the nodes in the plan graph represent static configuration objects rather than expanded instances, including for modules. To make this work in the meantime, there is some shimming between addrs.Module and addrs.ModuleInstance to correct for the discontinuities that result from the fact that Terraform currently assumes that modules are always singletons.
We're not far enough along yet to be able to actually use the RepetitionData instances provided by the instances package, but having these types be considered identical will help us to gradually migrate over as we prepare the rest of Terraform to properly populate the Expander.
|
The module expander work is being based off this branch. Since this applies cleanly to master I think we can merge it now to reduce the diff and make rebasing easier. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
apparentlymart commentedNov 22, 2019
•
edited
Our existing handling of
countandfor_eachfor resources is rather scattered across different parts of Terraform.This PR represents some initial experimentation towards centralizing the expansion logic for resources and for modules (looking forward to future module
countandfor_each) to help the different parts of Terraform Core that deal with the repetition constructs to coordinate with each other in a more organized way.The
Expandertype includes three different families of methods:Set*methods (SetResourceSingle,SetResourceCount,SetResourceForEach,SetModuleCount) are used to record the results of evaluating thecountandfor_eachexpressions in the configuration for later use by theExpand*family.Expand*methods (ExpandResourceandExpandModule) take the addresses of static (unexpanded) objects in the configuration and return all of the instance addresses that result from the expansions that were recorded by earlier calls to theSet*methods.Because modules can nest inside other modules, the
Expand*methods fully expand all levels of the tree and so the number of instances for a particular object can grow exponentially for objects in deeper modules.Get*RepetitionDatamethods (GetResourceInstanceRepetitionDataandGetModuleInstanceRepetitionData) take the absolute address of a particular module instance or resource instance identified by theExpand*family and return a decision about what values should appear foreach.key,each.value, andcount.indexwhen evaluating the configuration for the identified instance.Each family feeds into the next, so they must be called in a particular order for correct operation. The graph walk can take care of ensuring the correct ordering of calls so that data is always available before it is needed.
As an initial proof-of-concept I integrated
instances.Expanderin a rough way to replace some -- but not all -- of the scatteredcountandfor_eachlogic for resources. This now explicitly registers that every module in the configuration is a singleton (countandfor_eacharen't supported there yet) and then registers the repetition mode of each resource, which then allows theDynamicExpandlogic to just ask the expander what instances it ought to be creating.This
Expanderthing is also intended to be responsible for deciding what goes ineach.key,each.value, andcount.indexfor expressions, but I wasn't able to wire that up just yet because the apply graph (which is pre-expanded with nodes representing resource instances, not resources) isn't interacting with expander quite right yet, and the codepaths that do expression evaluation all run in both the plan and apply phases.In principle though, if the
Expanderis being populated properly on all walks, we would be able to get the values representing the current repetition using theGetResourceInstanceRepetitionDatamethod, which takes an absolute resource address and returns an object giving thecty.Valueto use foreach.key,each.valueandcount.index.If this were to be taken forward as the basis for module
countandfor_each, that would require us to do some rework on the way Terraform Core implements the plan walk so that it's clear that the nodes in the main plan graph represent configuration constructs belonging to unexpanded modules only (not to module instances), and addDynamicExpandto all of the nodes representing configuration constructs which calls into theExpanderinstance in order to find all of the module instances that the configuration construct is instantiated in. That would be a far more disruptive change, so I've not attempted that yet, but the newDynamicExpandimplementation for resources is ready to deal withcountandfor_eachon modules once the rest of the plan graph catches up.This does appear to work and pass all the existing context tests without modification, but for the moment it's mainly intended as a conversation starter for whether this particular direction --
DynamicExpandon everything -- feels good for modulecountandfor_each.