Terragrunt Github Pipeline with multiple environments and dependencies

[lpossamai]

Hi all 👋

I'm setting up a new Terragrunt repository and have few questions specially regarding the best practices around the pipeline. I'll be using Github Actions.

My current Terragrunt structure looks like this:

.
├── config
│   ├── _envcommon
│   │   ├── README.md
│   │   ├── s3.hcl
│   │   └── s3-logging-bucket.hcl
│   ├── Makefile
├── environments
│   ├── dev
│   │   ├── ap-southeast-2
│   │   │   ├── region.hcl
│   │   │   ├── s3
│   │   │   │   ├── Makefile
│   │   │   │   └── terragrunt.hcl
│   │   │   └── s3-logging-bucket
│   │   │       ├── Makefile
│   │   │       └── terragrunt.hcl
│   │   ├── env.hcl
│   │   └── Makefile
│   ├── prod
│   │   └── env.hcl
│   └── staging
│       └── env.hcl
├── modules
├── README.md
└── terragrunt.hcl

The main reason I'm using Terragrunt, is because after few years mainly using Terraform I ended up learning the hard way that Terraform sucks when there are multiple modules, state files and dependencies between them.

Looks like I cannot use run-all with Terragrunt when there are dependencies between modules. In that case, I'll have to make my CI to enter in each environment/**/ap-southeast-2/**/ folder and run the commands there.

However by doing that, I'll be creating lots and lots of duplicated files in the .github/workflow directory for each of the resources. Github Actions will need to perform the following:

[...]
cd environment/dev/ap-southeast-2/s3
terragrunt plan
[...]

[...]
cd environment/dev/ap-southeast-2/s3-logging-bucket
terragrunt plan
[...]

[...]
cd environment/dev/ap-southeast-2/vpc
terragrunt plan
[...]

etc

Wondering if there are better ways to achieve the same result? Any suggestions are more than welcome.

Cheers! 😄

---

Tracked in ticket #110155

[denis256]

Hello,
I was wondering if was considered terragrunt-infrastructure-live-example / terragrunt-infrastructure-modules-example which has examples for qa / stage deployments.

References:
https://github.com/gruntwork-io/terragrunt-infrastructure-live-example
https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example

[lpossamai]

Hello, I was wondering if was considered terragrunt-infrastructure-live-example / terragrunt-infrastructure-modules-example which has examples for qa / stage deployments.

References: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example

Hi. Yes... I did have a look at those but they do not have a pipeline in place. They only show you the terragrunt structure.

[fraz78]

@lpossamai we have a similar issue when trying to run terragrunt run-all init and terragrunt run-all plan against all of our terragrunt directories. We're using terragrunt in a CICD pipeline via GitHub Actions and after a few iterations, we got around the output dependency issue by introducing the mock outputs, but now we get something like terragrunt Underlying error: invalid character ':' after top-level value when trying to do a run-all plan. Have you managed to find a workaround? In the interim, we've created a Makefile and are now running the int, plan and apply against the first directory then moving onto the second directory. But this is not ideal. Ideally, we want a separate task in GitHub Actions to run a terragrunt run-all init then a separate task to run a terragrunt run-all plan and then finally a separate task to run a terragrunt run-all apply. Any advice?

[lpossamai]

@lpossamai we have a similar issue when trying to run terragrunt run-all init and terragrunt run-all plan against all of our terragrunt directories. We're using terragrunt in a CICD pipeline via GitHub Actions and after a few iterations, we got around the output dependency issue by introducing the mock outputs, but now we get something like terragrunt Underlying error: invalid character ':' after top-level value when trying to do a run-all plan. Have you managed to find a workaround? In the interim, we've created a Makefile and are now running the int, plan and apply against the first directory then moving onto the second directory. But this is not ideal. Ideally, we want a separate task in GitHub Actions to run a terragrunt run-all init then a separate task to run a terragrunt run-all plan and then finally a separate task to run a terragrunt run-all apply. Any advice?

Hey! The only solution I found was to use Atlantis or Terrateam. I've been using Terrateam and the community there is great and really help you out! I strongly suggest you try them :) It's been working great for me!

I even got in touch with antonbabenko to ask for some advise but he also mentioned the two solutions above.

Good luck!