Terragrunt architecture best practices?

[yorinasub17]

I’m getting started with Terragrunt and want to make sure I have this conceptually right. It looks like we can only source a single module from, e.g. my-app/terragrunt.hcl. So if my app needs, say, an ASG, a security group, and a load balancer, I might create modules/my-app/ and source 3 modules from there. Is that right?

Having the terragrunt.hcl reference a fairly specific module that references 3 other re-usable modules seems a bit odd to me, so I’m not sure I’m thinking about this right. Especially since we’re using Gruntwork modules, so each of those three re-usable modules is really a wrapper for a Gruntwork module. The chain of nested modules seems to get long pretty quickly.

[yorinasub17]

The core of Terragrunt architecture is considering how you want to break apart the Terraform state. While Terraform is optimized for working with a single state file, Terragrunt supports breaking apart your Terraform state into multiple component infrastructure through it's many features like dependency.

However, since Terragrunt ultimately relies on Terraform under the hood, it is limited by Terraform's ability to have a single root module, and that's why you still need to define a single Terraform module to act as the root module for Terragrunt to invoke at the folder level.

The way we at Gruntwork think about this is to differentiate between Services and Modules (See our blog post on Modules, Services, and Architectures for more details). The Service module is an opinionated, specific way to deploy components that fit the needs of your organization, which is less reusable, but is designed to be directly deployed by Terragrunt. If you look at our Service module for the ASG (asg-service), you will see that it combines the Gruntwork modules to construct the deployable ASG unit.

Given that, you should be striving for 2 links in your chain: Services and Modules. That is, when extending Gruntwork modules, it is oftentimes much better organizationally to extend them in the Service layer by combining Gruntwork modules with resources and modules to create the deployable unit, rather than extending Gruntwork modules to create more Modules. This should significantly help with the dependency management.

[yorinasub17]

A side note:

When thinking about what should go in the Services, you should also be thinking about the Terraform state file. In essence, each Service Terraform module will translate to a single Terraform state. Given that, it is oftentimes useful to think about splitting your Services based on how you want to split the Terraform state. I previously wrote about this in a Terragrunt issue, which I will copy here since it is still relevant:

When it comes to componentization, the best thing that has worked for us is to base it on the reason why you want to break up your state file in the first place. In general, there are 3 key benefits to breaking up your state file:

• Performance: You keep your modules and deployments smaller, so plan and apply can run faster per iteration. You can also apply the changes across multiple workers, increasing parallelism and concurrency of deployments.
• Reliability: By isolating and componentizing your changes, you run less risk of having a larger blast radius and having unintended consequences. In a world where a small typo could end up destroying things, having the ability to limit the actions is huge in keeping your infrastructure safe.
• Security: You may not want all your developers to have access to touch everything. Breaking up state files allows you to set up fine grained ACLs to split up the permissions needed to do a deployment.

On the other hand, breaking up state files adds a lot of complexity to the dependency management, and you lose the benefits of having reliable planning mechanisms. So you don't want to break it down too much.

Given that, the following litmus test has worked well for us when it comes to breaking things up:

• How long does it take to deploy your changes? A big fat terraform module might take 10-15 minutes just to run plan. This might be ok for something that is deployed infrequently (e.g., once per year), but could add up to a lot of wasted time if it is done multiple times per day. You may want to break that up so that you can deploy in parallel across multiple workers (since only one process can touch the state file at a time).

• How often does the infrastructure change? For example, you might deploy your application and service multiple times per day, touch your kubernetes cluster (like make node adjustments) a few times per month, and update your network architecture once per year. So it is natural to break up your components in to application, eks, and vpc, since the deploy frequency is very different.

• What level of granularity do you need for ACLs? For large organization, you may want only a small subset of trusted developers to access the full stack of the infrastructure. You may even have tiers, where only a handful has access to IAM, some have access to VPC and eks, but everyone has access to k8s services. Breaking up the state file and managing the ACLs via s3 bucket policies can provide this fine grained access control.