Improve support for Terragrunt projects

[alikhajeh1]

Currently our CI/CD integrations ask users to read https://www.infracost.io/docs/iac_tools/terragrunt, which doesn't mention how the user should use Infracost with Terragrunt in those integrations. For example, for GitLab, they could currently do the following.

However, could we detect a Terragrunt project from --path and support this natively without the user having to create a config file? That would make it easier to run Infracost with Terragrunt projects that have many modules, as currently the user has to update the config file.

1. Create an infracost config file, called infracost.yaml, with:

   version: 0.1
   
   projects:
     - path: myproject/module1
       terraform_binary: terragrunt
     - path: myproject/module2
       terraform_binary: terragrunt
   

2. Test the above works locally, by running infracost breakdown --config-file infracost.yaml

3. Add a GitLab CI stage for infracost - the same can be done with other CI/CD integrations by setting the config file param:

     stages:
       - infracost
   
     include:
       # Use a specific version of the template instead of master if locking the template is preferred
       remote: 'https://gitlab.com/infracost/infracost-gitlab-ci/-/raw/master/infracost.yml'
   
     infracost-job:
       only:
         - merge_requests
       stage: infracost
       extends: .infracost
       variables:
         config_file: infracost.yaml
   

[aliscott]

We can detect a Terragrunt project based on the terragrunt.hcl(.json) file or TERRAGRUNT_CONFIG env variable (https://terragrunt.gruntwork.io/docs/getting-started/configuration/).

We need to work out how we handle Terragrunt projects that contain multiple modules. These contain the terragrunt.hcl files in each module subdirectory.

Options for detecting Terragrunt projects with multiple modules:

1. Don't support these (but I think we should).
2. Automatically detect these by walking through all subdirectories and checking if each one is a Terragrunt module (see the Terragrunt logic here). This will be slow for large directories.
3. Manually detect these by adding a flag called something like --terragrunt-run-all. When passed, we assume the path has multiple Terragrunt modules.
4. Add a --type or --iac-type that allows the user to manually override the project type (terraform_dir, terraform_plan_json, terragrunt, etc).

If we do support these and can detect them then we also need to work out how to handle them:

1. Use the terragrunt binary and run terragrunt run-all {init,plan,show}
   • The terragrunt run-all plan -out outpath command stores the the outpath files in each of the child module directories, so we will need some way of finding these (see our current breakdown_all script).
2. Write the logic ourselves to loop through the child modules ourselves and run terragrunt {init,plan,show} for each child module. We could potentially hook into the existing Terragrunt codebase.

cc @alikhajeh1 and @tim775 if you have any thoughts on the options above.

[aliscott]

When supporting the terragrunt run-all commands we need to work out how to handle usage file parameters. Since this will show multiple projects, and each project can have the same resource names, then there's no way of scoping the usage file resources to a specific terragrunt module.

As an initial version we could apply the matching usage file values to all resources that match, but I would like to be able to set different values for my different terragrunt modules.

[aliscott]

Currently for Terragrunt projects with a hierarchy we require each Terragrunt module to be specified in the Infracost config file. Using this method you can pass in a different usage file for each module.

version: 0.1
projects:
  - path: my/terragrunt/project/dev
    terraform_binary: terragrunt
    usage_file: usage-dev.yml
  - path: my/terragrunt/project/prod
    terraform_binary: terragrunt
    usage_file: usage-dev.yml

When we support Terragrunt as first-class, we will allow the root Terragrunt directory to be passed in, like:

version: 0.1
projects:
  - path: my/terragrunt/project

The issue with this is that we now cannot specify a different usage file for each module. Multiple Terragrunt modules may have the same resources, so we have no way of uniquely identifying them in the usage file.

To solve this we could:

Option 1: Update the config file to allow multiple usage files and specify which module they are for

For example:

version: 0.2
projects:
  - path: my/terragrunt/project
    usage_files:
      - match_path: 'dev/base'
        file: infracost-usage-dev-base.yml
      - match_path: 'prod/base'
        file: infracost-usage-prod-base.yml
      - match_path: 'dev/api'
        file: infracost-usage-dev-api.yml
      - match_path: 'prod/api'
        file: infracost-usage-prod-api.yml

Option 2: Update the config file to support multiple projects:

Both below examples are additive only so wouldn't break existing usage files.

Example A:

version: 0.1

# This contains the default resource usage
resource_usage:
  aws_lambda_function.my_lambda_function:
    monthly_requests: 20000
    request_duration_ms: 600
  
# This contains any overrides for the dev/base project 
resource_usage[dev/base]:
  aws_lambda_function.my_lambda_function:
    monthly_requests: 100
    request_duration_ms: 600

Example B:

version: 0.1

# This contains the default
resource_usage:
  aws_lambda_function.my_lambda_function:
    monthly_requests: 20000
    request_duration_ms: 600

# This has any overrides/per-project usage
projects:
  - path: my/terragrunt/project/dev/base
    resource_usage:
      aws_lambda_function.my_lambda_function:
        monthly_requests: 100
        request_duration_ms: 600

Option X: Any other suggestions / comments on the above options?

We will also need to consider how this would work if we support auto-detecting Terraform workspaces or if we support Terraform monorepos (as per #490 and #188).

[alikhajeh1]

@ljhurst, @Ian-T-Price and @mikejk8s just FYI that we shipped improved support for Terragrunt in v0.9.7, see docs here: https://www.infracost.io/docs/iac_tools/terragrunt

We'd love to hear your feedback on it, and also #934 if you use the usage-file.