From 60d77578b60a6ad35908bdfdf63d4f00b7e80827 Mon Sep 17 00:00:00 2001 From: Yousif Akbar <11247449+yhakbar@users.noreply.github.com> Date: Fri, 10 Oct 2025 15:19:12 -0400 Subject: [PATCH 1/5] feat: Adding GitLab Azure support --- .../installation/addinggitlabrepo.mdx | 625 +++++++++++++++++- 1 file changed, 623 insertions(+), 2 deletions(-) diff --git a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx index 363746c73..d0737a031 100644 --- a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx +++ b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx @@ -107,6 +107,9 @@ mise ls-remote boilerplate ### Cloud-specific bootstrap instructions + + + The resources that you need provisioned in AWS to start managing resources with Pipelines are: 1. An OpenID Connect (OIDC) provider @@ -203,7 +206,7 @@ boilerplate \ :::note Progress Checklist - + ::: @@ -215,7 +218,7 @@ mise install :::note Progress Checklist - + ::: {/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} @@ -362,6 +365,624 @@ terragrunt run --all --non-interactive --provider-cache apply ::: + + + +The resources that you need provisioned in Azure to start managing resources with Pipelines are: + +1. An Azure Resource Group for OpenTofu state resources + 1. An Azure Storage Account in that resource group for OpenTofu state storage + 1. An Azure Storage Container in that storage account for OpenTofu state storage +2. An Entra ID Application to use for plans + 1. A Flexible Federated Identity Credential for the application to authenticate with your project on any branch + 2. A Service Principal for the application to be used in role assignments + 1. A role assignment for the service principal to access the Azure subscription + 2. A role assignment for the service principal to access the Azure Storage Account +3. An Entra ID Application to use for applies + 1. A Federated Identity Credential for the application to authenticate with your project on the deploy branch + 2. A Service Principal for the application to be used in role assignments + 1. A role assignment for the service principal to access the Azure subscription + +:::tip Don't Panic! + +This may seem like a lot to set up, but the content you need to add to your `infrastructure-live` repository is minimal. The majority of the work will be pulled from a reusable catalog that you'll reference in your `infrastructure-live` repository. + +If you want to peruse the catalog that's used in the bootstrap process, you can take a look at the [terragrunt-scale-catalog](https://github.com/gruntwork-io/terragrunt-scale-catalog) repository. + +::: + +The process that we'll follow to get these resources ready for Pipelines is: + +1. Set up these bootstrap resources by creating some Terragrunt configurations in your `infrastructure-live` repository for bootstrapping Pipelines in a single Azure subscription +2. Use Terragrunt to provision these resources in your Azure subscription +3. Finalizing Terragrunt configurations using the bootstrap resources we just provisioned +4. Pull the bootstrap resources into state, now that we have configured a remote state backend +5. (Optionally) Bootstrap additional Azure subscriptions until all your Azure subscriptions are ready for Pipelines + +{/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} +

Bootstrap your `infrastructure-live` repository

+ +To bootstrap your `infrastructure-live` repository, we'll use Boilerplate to scaffold it with the necessary content for Pipelines to function. + +```bash +boilerplate \ + --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/azure/gitlab/infrastructure-live?ref=v1.0.0' \ + --output-folder . +``` + +:::tip + +You can just reply `y` to all the prompts to include dependencies, and accept defaults unless you want to customize something. + +Alternatively, you could run Boilerplate non-interactively by passing the `--non-interactive` flag. You'll need to supply the relevant values for required variables in that case. + +e.g. + +```bash +boilerplate \ + --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/azure/gitlab/infrastructure-live?ref=v1.0.0' \ + --output-folder . \ + --var 'AccountName=dev' \ + --var 'GitLabGroupName=acme' \ + --var 'GitLabRepoName=infrastructure-live' \ + --var 'GitLabInstanceURL=https://gitlab.com' \ + --var 'SubscriptionName=dev' \ + --var 'AzureTenantID=00000000-0000-0000-0000-000000000000' \ + --var 'AzureSubscriptionID=11111111-1111-1111-1111-111111111111' \ + --var 'AzureLocation=East US' \ + --var 'StateResourceGroupName=pipelines-rg' \ + --var 'StateStorageAccountName=mysa' \ + --var 'StateStorageContainerName=tfstate' \ + --non-interactive +``` + +You can also choose to store these values in a YAML file and pass it to Boilerplate using the `--var-file` flag. + +```yaml title="vars.yml" +AccountName: dev +GitLabGroupName: acme +GitLabRepoName: infrastructure-live +GitLabInstanceURL: https://gitlab.com +SubscriptionName: dev +AzureTenantID: 00000000-0000-0000-0000-000000000000 +AzureSubscriptionID: 11111111-1111-1111-1111-111111111111 +AzureLocation: East US +StateResourceGroupName: pipelines-rg +StateStorageAccountName: my-storage-account +StateStorageContainerName: tfstate +``` + +```bash +boilerplate \ + --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/azure/gitlab/infrastructure-live?ref=v1.0.0' \ + --output-folder . \ + --var-file vars.yml \ + --non-interactive +``` + +::: + +:::note Progress Checklist + +::: + +Next, install Terragrunt and OpenTofu locally (the `.mise.toml` file in the root of the repository after scaffolding should already be set to the versions you want for Terragrunt and OpenTofu): + +```bash +mise install +``` + +:::note Progress Checklist + + + +::: + +{/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} +

Provisioning the resources

+ +Once you've set up the Terragrunt configurations, you can use Terragrunt to provision the resources in your Azure subscription. + +If you haven't already, you'll want to authenticate to Azure using the `az` CLI. + +```bash +az login +``` + +:::note Progress Checklist + + + +::: + + +To dynamically configure the Azure provider with a given tenant ID and subscription ID, ensure that you are exporting the following environment variables if you haven't the values via the `az` CLI: + +- `ARM_TENANT_ID` +- `ARM_SUBSCRIPTION_ID` + +For example: + +```bash +export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000" +export ARM_SUBSCRIPTION_ID="11111111-1111-1111-1111-111111111111" +``` + +:::note Progress Checklist + + + +::: + +First, make sure that everything is set up correctly by running a plan in the subscription directory. + +```bash title="name-of-subscription" +terragrunt run --all --non-interactive --provider-cache plan +``` + +:::tip + +We're using the `--provider-cache` flag here to ensure that we don't re-download the Azure provider on every run to speed up the process by leveraging the [Terragrunt Provider Cache Server](https://terragrunt.gruntwork.io/docs/features/provider-cache-server/). + +::: + +:::note Progress Checklist + + + +::: + +Next, apply the changes to your subscription. + +```bash title="name-of-subscription" +terragrunt run --all --non-interactive --provider-cache --no-stack-generate apply +``` + +:::tip + +We're adding the `--no-stack-generate` flag here, as Terragrunt will already have the requisite stack configurations generated, and we don't want to accidentally overwrite any configurations while we have state stored locally before we pull them into remote state. + +::: + +:::note Progress Checklist + + +::: + +{/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} +

Finalizing Terragrunt configurations

+ +Once you've provisioned the resources in your Azure subscription, you can finalize the Terragrunt configurations using the bootstrap resources we just provisioned. + +First, edit the `root.hcl` file in the root of your `infrastructure-live` repository to leverage the storage account we just provisioned. + +```hcl title="root.hcl" +locals { + sub_hcl = read_terragrunt_config(find_in_parent_folders("sub.hcl")) + + state_resource_group_name = local.sub_hcl.locals.state_resource_group_name + state_storage_account_name = local.sub_hcl.locals.state_storage_account_name + state_storage_container_name = local.sub_hcl.locals.state_storage_container_name +} + +# FIXME: Uncomment the code below when you've successfully bootstrapped Pipelines state. +# +# remote_state { +# backend = "azurerm" +# generate = { +# path = "backend.tf" +# if_exists = "overwrite" +# } +# config = { +# resource_group_name = local.state_resource_group_name +# storage_account_name = local.state_storage_account_name +# container_name = local.state_storage_container_name +# key = "${path_relative_to_include()}/tofu.tfstate" +# } +# } + +generate "provider" { + path = "provider.tf" + if_exists = "overwrite_terragrunt" + contents = < + +::: + +Next, finalize the `.gruntwork/environment-.hcl` file in the root of your `infrastructure-live` repository to reference the IDs for the applications we just provisioned. + +```hcl title=".gruntwork/environment-.hcl" +environment "dev" { + filter { + paths = ["dev/*"] + } + + authentication { + azure_oidc { + tenant_id = "00000000-0000-0000-0000-000000000000" + subscription_id = "11111111-1111-1111-1111-111111111111" + + plan_client_id = "" # FIXME: Fill in the client ID for the plan application after bootstrapping + apply_client_id = "" # FIXME: Fill in the client ID for the apply application after bootstrapping + } + } +} +``` + +You can find the values for the `plan_client_id` and `apply_client_id` by running `terragrunt stack output` in the `bootstrap` directory in `name-of-subscription/bootstrap`. + +```bash +terragrunt stack output +``` + +The relevant bits that you want to extract from the stack output are the following: + +```hcl +bootstrap = { + apply_app = { + client_id = "33333333-3333-3333-3333-333333333333" + } + plan_app = { + client_id = "44444444-4444-4444-4444-444444444444" + } +} +``` + +You can use those values to set the values for `plan_client_id` and `apply_client_id` in the `.gruntwork/environment-.hcl` file. + +:::note Progress Checklist + + + + +::: + +{/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} +

Pulling the resources into state

+ +Once you've provisioned the resources in your Azure subscription, you can pull the resources into state using the storage account we just provisioned. + +```bash title="name-of-subscription" +terragrunt run --all --non-interactive --provider-cache --no-stack-generate -- init -migrate-state -force-copy +``` + +:::tip + +We're adding the `-force-copy` flag here to avoid any issues with OpenTofu waiting for an interactive prompt to copy up local state. + +::: + +:::note Progress Checklist + + + +::: + +{/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} +

Optional: Bootstrapping additional Azure subscriptions

+ +If you have multiple Azure subscriptions, and you want to bootstrap them as well, you can do so by following a similar, but slightly condensed process. + +For each additional subscription you want to bootstrap, you'll use Boilerplate in the root of your `infrastructure-live` repository to scaffold out the necessary content for just that subscription. + +:::tip + +If you are going to bootstrap more Azure subscriptions, you'll probably want to commit your existing changes before proceeding. + +```bash +git add . +git commit -m "Add additional Azure subscriptions [skip ci]" +``` + +::: + +Just like before, you'll use Boilerplate to scaffold out the necessary content for just that subscription. + +```bash +boilerplate \ + --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/azure/gitlab/subscription?ref=v1.0.0' \ + --output-folder . +``` + +:::tip + +Again, you can just reply `y` to all the prompts to include dependencies, and accept defaults unless you want to customize something. + +::: + +Alternatively, you could run Boilerplate non-interactively by passing the `--non-interactive` flag. You'll need to supply the relevant values for required variables in that case. + +e.g. + +```bash +boilerplate \ + --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/azure/gitlab/subscription?ref=v1.0.0' \ + --output-folder . \ + --var 'AccountName=prod' \ + --var 'GitLabGroupName=acme' \ + --var 'GitLabRepoName=infrastructure-live' \ + --var 'GitLabInstanceURL=https://gitlab.com' \ + --var 'SubscriptionName=prod' \ + --var 'AzureTenantID=00000000-0000-0000-0000-000000000000' \ + --var 'AzureSubscriptionID=99999999-9999-9999-9999-999999999999' \ + --var 'AzureLocation=East US' \ + --var 'StateResourceGroupName=pipelines-rg' \ + --var 'StateStorageAccountName=myprodsa' \ + --var 'StateStorageContainerName=tfstate' \ + --non-interactive +``` + +If you prefer to store the values in a YAML file and pass it to Boilerplate using the `--var-file` flag, you can do so like this: + +```yaml title="vars.yml" +AccountName: prod +GitLabGroupName: acme +GitLabRepoName: infrastructure-live +GitLabInstanceURL: https://gitlab.com +SubscriptionName: prod +AzureTenantID: 00000000-0000-0000-0000-000000000000 +AzureSubscriptionID: 99999999-9999-9999-9999-999999999999 +AzureLocation: East US +StateResourceGroupName: pipelines-rg +StateStorageAccountName: myprodsa +StateStorageContainerName: tfstate +``` + +```bash +boilerplate \ + --template-url 'github.com/gruntwork-io/terragrunt-scale-catalog//templates/boilerplate/azure/gitlab/subscription?ref=v1.0.0' \ + --output-folder . \ + --var-file vars.yml \ + --non-interactive +``` + +:::note Progress Checklist + + + +::: + +To avoid issues with the remote state backend not existing yet, you'll want to comment out your remote state backend configurations in your `root.hcl` file before you start the bootstrap process for these new subscriptions. + +```hcl title="root.hcl" +locals { + sub_hcl = read_terragrunt_config(find_in_parent_folders("sub.hcl")) + + state_resource_group_name = local.sub_hcl.locals.state_resource_group_name + state_storage_account_name = local.sub_hcl.locals.state_storage_account_name + state_storage_container_name = local.sub_hcl.locals.state_storage_container_name +} + +# FIXME: Temporarily commented out again, pending successful bootstrap of the new subscription(s). +# +# remote_state { +# backend = "azurerm" +# generate = { +# path = "backend.tf" +# if_exists = "overwrite" +# } +# config = { +# resource_group_name = local.state_resource_group_name +# storage_account_name = local.state_storage_account_name +# container_name = local.state_storage_container_name +# key = "${path_relative_to_include()}/tofu.tfstate" +# } +# } + +generate "provider" { + path = "provider.tf" + if_exists = "overwrite_terragrunt" + contents = < + +::: + +Just like before, you can use Terragrunt to provision the resources in each of these subscriptions. + +For each subscription you want to bootstrap, you'll need to run the following commands: + +```bash +cd /_global/bootstrap +terragrunt run --all --non-interactive --provider-cache plan +terragrunt run --all --non-interactive --provider-cache --no-stack-generate apply +``` + +:::tip + +We're adding the `--no-stack-generate` flag here, as Terragrunt will already have the requisite stack configurations generated, and we don't want to accidentally overwrite any configurations while we have state stored locally before we pull them into remote state. + +::: + +:::note Progress Checklist + + + + +::: + +Next, you can pull the resources into state using the storage account we just provisioned. + +First, edit the `root.hcl` file in the root of your `infrastructure-live` repository to uncomment the remote state backend configurations you commented out earlier. + +```hcl title="root.hcl" +locals { + sub_hcl = read_terragrunt_config(find_in_parent_folders("sub.hcl")) + + state_resource_group_name = local.sub_hcl.locals.state_resource_group_name + state_storage_account_name = local.sub_hcl.locals.state_storage_account_name + state_storage_container_name = local.sub_hcl.locals.state_storage_container_name +} + +remote_state { + backend = "azurerm" + generate = { + path = "backend.tf" + if_exists = "overwrite" + } + config = { + resource_group_name = local.state_resource_group_name + storage_account_name = local.state_storage_account_name + container_name = local.state_storage_container_name + key = "${path_relative_to_include()}/tofu.tfstate" + } +} + +generate "provider" { + path = "provider.tf" + if_exists = "overwrite_terragrunt" + contents = < + +::: + +Next, you can pull the resources into state using the storage account we just provisioned. + +```bash title="name-of-subscription" +terragrunt run --all --non-interactive --provider-cache --no-stack-generate -- init -migrate-state -force-copy +``` + +:::tip + +We're adding the `-force-copy` flag here to avoid any issues with OpenTofu waiting for an interactive prompt to copy up local state. + +::: + +:::note Progress Checklist + + + +::: + +Finally, we can edit each of the `.gruntwork/environment-.hcl` files in the root of your `infrastructure-live` repository to reference the IDs for the applications we just provisioned. + +```hcl title=".gruntwork/environment-.hcl" +environment "prod" { + filter { + paths = ["prod/*"] + } + + authentication { + azure_oidc { + tenant_id = "00000000-0000-0000-0000-000000000000" + subscription_id = "99999999-9999-9999-9999-999999999999" + + plan_client_id = "" # FIXME: Fill in the client ID for the plan application after bootstrapping + apply_client_id = "" # FIXME: Fill in the client ID for the apply application after bootstrapping + } + } +} +``` + +You can find the values for the `plan_client_id` and `apply_client_id` by running `terragrunt stack output` in the `bootstrap` directory in `name-of-subscription/bootstrap`. + +```bash +terragrunt stack output +``` + +The relevant bits that you want to extract from the stack output are the following: + +```hcl +bootstrap = { + apply_app = { + client_id = "55555555-5555-5555-5555-555555555555" + } + plan_app = { + client_id = "66666666-6666-6666-6666-666666666666" + } +} +``` + +You can use those values to set the values for `plan_client_id` and `apply_client_id` in the `.gruntwork/environment-.hcl` file. + +:::note Progress Checklist + + + + +::: + + + + ## Commit and push your changes Commit and push your changes to your repository. From 1e0f8e3163f8dbf0562bda4a933cb91302721b03 Mon Sep 17 00:00:00 2001 From: Yousif Akbar <11247449+yhakbar@users.noreply.github.com> Date: Fri, 10 Oct 2025 17:29:39 -0400 Subject: [PATCH 2/5] Update docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx Co-authored-by: Oreoluwa Agunbiade <21035422+oredavids@users.noreply.github.com> --- docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx index d0737a031..3963e573e 100644 --- a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx +++ b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx @@ -373,7 +373,7 @@ The resources that you need provisioned in Azure to start managing resources wit 1. An Azure Resource Group for OpenTofu state resources 1. An Azure Storage Account in that resource group for OpenTofu state storage 1. An Azure Storage Container in that storage account for OpenTofu state storage -2. An Entra ID Application to use for plans +2. An Entra ID Application to use for Pipelines plans 1. A Flexible Federated Identity Credential for the application to authenticate with your project on any branch 2. A Service Principal for the application to be used in role assignments 1. A role assignment for the service principal to access the Azure subscription From 758280d902c692ce2cf44617ef67bbd89e89da4e Mon Sep 17 00:00:00 2001 From: Yousif Akbar <11247449+yhakbar@users.noreply.github.com> Date: Fri, 10 Oct 2025 17:29:45 -0400 Subject: [PATCH 3/5] Update docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx Co-authored-by: Oreoluwa Agunbiade <21035422+oredavids@users.noreply.github.com> --- docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx index 3963e573e..00373ef31 100644 --- a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx +++ b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx @@ -378,7 +378,7 @@ The resources that you need provisioned in Azure to start managing resources wit 2. A Service Principal for the application to be used in role assignments 1. A role assignment for the service principal to access the Azure subscription 2. A role assignment for the service principal to access the Azure Storage Account -3. An Entra ID Application to use for applies +3. An Entra ID Application to use for Pipelines applies 1. A Federated Identity Credential for the application to authenticate with your project on the deploy branch 2. A Service Principal for the application to be used in role assignments 1. A role assignment for the service principal to access the Azure subscription From df16d0f787156045a287300ff818589b00cd9a49 Mon Sep 17 00:00:00 2001 From: Yousif Akbar <11247449+yhakbar@users.noreply.github.com> Date: Fri, 10 Oct 2025 17:29:52 -0400 Subject: [PATCH 4/5] Update docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx Co-authored-by: Oreoluwa Agunbiade <21035422+oredavids@users.noreply.github.com> --- docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx index 00373ef31..a650cc198 100644 --- a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx +++ b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx @@ -396,7 +396,7 @@ The process that we'll follow to get these resources ready for Pipelines is: 1. Set up these bootstrap resources by creating some Terragrunt configurations in your `infrastructure-live` repository for bootstrapping Pipelines in a single Azure subscription 2. Use Terragrunt to provision these resources in your Azure subscription 3. Finalizing Terragrunt configurations using the bootstrap resources we just provisioned -4. Pull the bootstrap resources into state, now that we have configured a remote state backend +4. Pull the bootstrapped resources into state, now that we have configured a remote state backend 5. (Optionally) Bootstrap additional Azure subscriptions until all your Azure subscriptions are ready for Pipelines {/* We're using an h3 tag here instead of a markdown heading to avoid adding content to the ToC that won't work when switching between tabs */} From cfba2b86620c28cdf426e7bc6d07695039bd60db Mon Sep 17 00:00:00 2001 From: Yousif Akbar <11247449+yhakbar@users.noreply.github.com> Date: Fri, 10 Oct 2025 17:31:14 -0400 Subject: [PATCH 5/5] Update docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx Co-authored-by: Oreoluwa Agunbiade <21035422+oredavids@users.noreply.github.com> --- docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx index a650cc198..ad1673729 100644 --- a/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx +++ b/docs/2.0/docs/pipelines/installation/addinggitlabrepo.mdx @@ -496,7 +496,7 @@ az login ::: -To dynamically configure the Azure provider with a given tenant ID and subscription ID, ensure that you are exporting the following environment variables if you haven't the values via the `az` CLI: +To dynamically configure the Azure provider with a given tenant ID and subscription ID, ensure that you are exporting the following environment variables if you haven't set the values via the `az` CLI: - `ARM_TENANT_ID` - `ARM_SUBSCRIPTION_ID`