[RFC] Iteration of modules #853
Conversation
yorinasub17
requested review from
autero1,
brikis98 and
eak12913
as code owners
|
cc @bwhaley |
| `for_each` list to only reference the items you want to destroy, then run `terragrunt destroy` to destroy just those | ||
| items, and then restore the list to the version with the items removed). | ||
|
|
||
| ### Option 3: scaffolding tool that code gens live config using a template |
There was a problem hiding this comment.
@bwhaley : After contemplating about this feature, I am leaning towards this approach for our purposes. The implementation for supporting iteration in Terragrunt has many nuances that suggest it is not a great idea.
Better to wait for the official solution, working around it using code generation. Code generation appears to be the lesser evil of a workaround.
There was a problem hiding this comment.
Great RFC. Really highlights the complexity of for_each natively being built into Terragrunt, and the limited shelf life of such a feature.
I'm reluctantly a +1 on code generation. I'd be tempted to do the generate in the Terraform code (.tf files), simulating for_each on the module by just generating N copies of the module from some template, and committing that. Later, when for_each is officially added to Terraform, remove the generated code, and replace it with one module with the for_each on it.
| - `include` | ||
| - `dependency` | ||
| - Everything else | ||
| - Support `merge_parent` in `remote_state`, which ultimately adjusts how `mergeConfigWithIncludedConfig` merges the |
There was a problem hiding this comment.
Is there a way for us to check the user isn't accidentally overriding the config multiple times? Does remote_state config in Terragrunt become required to use this feature? Or do we read the .terraform/terraform.tfstate file? Seems a bit complicated.
There was a problem hiding this comment.
I think local state will work because I was imagining each iteration will have it's own directory in the cache, but yes this isn't immediately obvious.
|
|
||
| **Cons**: | ||
|
|
||
| - The implementation of the runners can become complex, especially in the dependency detection for the `xxx-all` |
There was a problem hiding this comment.
Yea, there are a lot of complexities here, and many, many ways to get it wrong.
_docs/rfc/for_each_iteration.md
Outdated
|
|
||
| **Cons**: | ||
|
|
||
| - Code generation is complex. When should you run the code generation step? How often? How do you know if there is |
There was a problem hiding this comment.
Do you check in the generated code? If so, you have to maintain it (e.g., 19 sets of files/folders for 19 AWS regions). If it's always generated at runtime, then interacting with just one of the generated things becomes harder.
There was a problem hiding this comment.
Valid points! Adding this in as one of the concerns.
|
@yorinasub17 Thanks for this RFC! Here are my 5 cents... After reading the whole RFC and solving it using cookiecutter before, I like code generation as the proposed solution.
|
Ah that is a good point.
The one thing I am not sure about is if this should be a first class thing in terragrunt. I think this meshes really well with the scaffolding idea (which should be a separate RFC). As in, if we have the |
Thanks for your input @antonbabenko !
Good point, although I would be concerned about accidentally checking in the changes in the code itself instead of the generator. This could happen if one is not aware of the codegen process.
I understand this sentiment and we definitely want to keep terragrunt simple as possible. That said, having codegen baked into terragrunt has some advantages in improving the UX (think |
|
Thanks @yorinasub17 for writing this RFC and thinking through the problem, and thanks to everyone else for your input. Seems like the consensus is that adding iteration to Does it even make sense to include code generation (including examples or docs) here in Since we need the "run this module in all regions" functionality specifically for the AWS Config module as part of a CIS compliance project, perhaps we should just do the code generation in that repo instead, and close this one out. We could have a new PR in that repo discussing exactly how the code generation would work, including addressing some of the cons raised here. For example:
After modifying the template.
Every time the generated code for all regions need to change.
All the generated files should have headers saying "don't touch this, it's generated code." The generated files would be overwritten any time changes are made to the master.
Yes, always check in the generated code, and never edit anything but the master. Obviously there are some deeper complexities that will come up during development, but the point is, code generation probably doesn't belong in terragrunt. |
|
Given all the discussions, I believe we have reached the conclusion that we shouldn't implement anything specific in Terragrunt regarding this specific use case. There may be other use cases for having code generation in Terragrunt, but those should be optimized for the use cases where it makes sense to bake into Terragrunt. I have updated the RFC with the recommendation and conclusions. Going to merge this in now so we have it in the repo for reference. Thanks to everyone who took the time to review this! |
|
Hey since terraform implemented support for |
This proposes an RFC for potentially supporting iteration of modules in Terragrunt. The document contains the background for why we should consider this.