How Gruntwork Pipelines works in a multi-account, multi-region model with terragrunt?

[chambodn]

Hello,

I am trying to understand how the gruntwork pipelines framework will interact with an AWS multi-account, multi-region topology.
We are using terragrunt to manage our infrastructure similar to the one you provide in the infrastructure-live folder of your terraform-aws-service-catalog repository.
We created a new Shared AWS Account to deploy the secrets required to communicate with our VCS. Based on the diagram here, I concluded that the "terraform/terragrunt executors" (i.e., ecs-deploy-runner tasks in ECS) should be deployed in all the AWS Accounts (dev, prod, security, etc...) for all regions (using multi-region modules as defined here).

Note: The VPC Management part, though, as a dependency to this multi-region module and therefore still need to be deployed in all the regions.

This decentralized model sounds great but leads me to some questions.

Considering that we have the live repository with terragrunt to represent our entire infrastructure, I wondered how the CI server would identify which lambda invoker function / ecs-deploy-runner (in which region, which account)? In our case, we plan to use GitLab and the OIDC integration to avoid storing secrets. How can I know in advance which IAM role (allowing me to invoke the lambda) I need to assume to invoke the lambda invoker?

Let's say we have the following structure:

.
└── live
    ├── sandbox-account-backend # <---- Where my microservices are *deployed*
    │   ├── _global
    │   │   └── iam
    │   ├── eu-west-1
    │   │   ├── eks
    │   │   └── mgmt
    │   │       ├── ecs-deploy-runner # <---- My multi-region module for this account
    │   │       └── networking
    │   │           └── vpc
    │   └── eu-west-3
    │       ├── eks
    │       └── mgmt
    │           └── networking # <---- Still needed as we don't have a "multi-region" module for VPC Mgmt 
    │               └── vpc
    ├── sandbox-account-networking
    │   ├── _global
    │   │   ├── dns
    │   │   └── iam
    │   ├── eu-west-1
    │   │   ├── apigateway # <---- Where my microservices are *exposed*
    │   │   └── mgmt
    │   │       ├── ecs-deploy-runner
    │   │       └── networking
    │   │           └── vpc
    │   └── eu-west-3
    │       ├── apigateway
    │       └── mgmt
    │           └── networking
    │               └── vpc
    └── shared
        └── eu-west-1
            ├── _regional
            │   └── ecr-repos
            └── mgmt
                ├── ecs-deploy-runner
                └── networking

Let's consider I create a new PR modifying the EKS cluster in live/sandbox-account-backend/eu-west-1/eks and the associated apigateway in the networking account in live/sandbox-account-networking/eu-west-1/apigateway.

Which ecs-deploy-runner task will be called by GitLab CI? And if we have a cross-account dependency (e.g., apigateway depends on eks), how can we ensure the (cross-account) order will be insured?

---

Tracked in ticket #108582

[yorinasub17]

The primary way to handle this is by routing with a lookup table and using heuristics with the folder structure to guide you on which entry in the table to get.

For example, in the Reference Architecture, we handle this by checking for all the terragrunt folders that changed (in this line), and then for each one, extracting out the first folder in the path and using that to lookup the AWS Account ID that corresponds to the folder by matching the folder name to the entry in the accounts.json file, and then using the ID to construct the IAM Role ARN to assume and assume that (see this function).

We are not very familiar with the GitLab OIDC integration for assuming a role, but presumably you can use similar logic to identify the IAM Role you need to assume in the workflow, and then extract that out as a step output to feed into the OIDC mechanism.

Hope this helps!

[chambodn]

Thanks, @yorinasub17. It helps a lot! However, I am still not quite sure to understand how this will guarantee the ordering. If we use the dependency management capability from terragrunt, do you confirm that even if we have cross-region (and potentially cross-account) dependencies it will still work as expected?

[yorinasub17]

The dependency ordering is not a feature that is supported by the current pipeline, and is a known limitation. See the thread on this discussion for additional details.

In theory, you could hook the pipeline up to use run-all and --terragrunt-include-dir to have a dependency aware pipeline, but the trade off is that you will lose sane plan outputs that you can rely on to determine what will change. So it's a trade off that will be dependent on the team's ability to adapt, and what level of infrastructure risk one is willing to take.