IO (Digital Citizenship) infrastructure Terraform modules

The repository contains the Terraform modules used to provision and maintain the IO infrastructure. This should be considered a library of modules to be called from the Terraform live infrastructure repository.

What is IO?

More informations about the IO can be found on the Digital Transformation Team website

What tools should I know to work in this repo?

NOTE: The project is currently still using Terraform 0.11.x and -consequentially- compatible Terragrunt versions (0.18.x)

Repository directories and files structure

The "live" Terraform component in this repository have been structured in a hierarchical way, following the best-practices described by Terragrunt.

Environment
    |_ Deployment area
        |_ Module

The root folder contains one or more deployment environments, for example dev or prod. There's an extra -service- environment, called infra, in which the io.italia.it DNS zone and the resources used by Terraform itself live. While the infra resource group and the resources needed by Terraform (such as a storage account and a dedicated Azure keyvault) are pre-created with a bash az-init.sh script, Terraform manages the DNS zone and its records.

Each environment contains one or more deployment geographical areas, for example westeurope.

Each deployment area contains one or more live components that have a one to one correspondence with a Terragrunt module. The Terragrunt modules are possibly versioned and maintained in a separate repository.

Variable values -passed to the modules in input- and the main Terragrunt configuration files are stored in terraform.tfvars files located in each live component folder.

Modules can also optionally inherit shared variables from higher level folders. These variables may be stored at each level of the hierarchy in the vars.tfvars files. For example, the westeurope folder under each environment contains the variable location = "westeurope", that is inherited by all the underlying live components.

There's a naming convention around folders and files. First of all, they should all be lowercase. The first part of the folder name usually describes what the component does (for example, if it's a kubernetes cluster, a function or a virtual network); the second part reflects the unique name of the specific resource on Azure. Underscores are used to separate words; dashes separate words referring to the Azure resource name, which uses by convention dashes as well. For example, if a Kubernetes cluster on Azure is named k8s-01, the corresponding live TF component folder could be named as kubernetes_cluster_k8s-01.

Prerequisites

The following softwares need to be installed on your machine before moving forward:

Python 3 - used by the Azure CLI
Azure CLI
Terraform v0.11.14
Terragrunt v0.18.2

NOTE: it's strongly suggested to install both the tools manually copying their binary in the local path.

NOTE: Terraform 0.12 and related Terragrunt versions are still not supported. Make sure to use compatible versions, and possibly to use the ones indicated above. Brew or other automated tools try to install the latest version available, which is not alwasy necessarily compatible.

Additional requirements:

Ask to another Administrator to invite you on the Azure subscription you've to run the scripts aginst, and to assign you a Contributor role.
Ask to another Administrator to set (at least) a Reader access policy on the Keyvauly io-infra-keyvault in the resource group io-infra-rg

Authentication and environment variables export

Before being able to apply Terragrunt configurations, users need to authenticate on Azure and export some environment variables, needed by Terraform to operate.

To authenticate on Azure, use the Azure CLI you've downloaded, running:

az login

A web page will show up. When the authentication process is complete, you'll be able to go back to the Terminal.

Copy the .env.example file located in the utils directory to a .env file (keep it in the same location) and fill in the values required. You can get the missing values from the Azure GUI (portal.azure.com) or from a colleague. Alternatively, you can directly ask to another Administrator for a copy of the final .env file.

Finally, source the az-export.sh script in the utils folder (while you are in the utils folder):

source az-export.sh

You can now apply the Terragrunt configurations.

More info about the utils scripts can be found in the README located in the utils folder.

Terragrunt usage overview

Each live component applied along the hierarchy will also inherit the variable values from the top vars.tfvars and terraform.tfvars files. Variables re-declared at lower level of the hierarchy can also override the same variables, declared at higher levels of the hierarchy.

To provision a specific component, go in the live component folder and run

terragrunt apply

If you have not committed your Terragrunt modules yet, and you'd like to test the scripts using a local copy of the modules run:

terragrunt apply --terragrunt-source PATH_TO_YOUR_THE_LOCAL_MODULE_DIR

HINT: Substitute apply with destroy or plan to destroy a resource or preview the effects of its creation.

Sometimes, you may want to provision the entire environment. To do so, go into the environment folder (for example dev or dev/westeurope) and run

terragrunt apply-all

Terragrunt will try to apply all the components under the folder where the Terragrunt command is applied, understanding what module should be applied first and what should be applied later, depending on the dependencies defined in the components' terraform.tfvars files.

NOTE: Substitute apply-all with destroy-all or plan-all to destroy a set of resources or preview the effects of their creation.

How to build a new infrastructure from scratch

The paragraph describes the action to perform and the order in which they should be applied, in order to bring up a new IO infrastructure from scratch.

Notes about secrets and certificates creation/import

While some of the secrets and certificates needed by some components below are automatically generated and uploaded during the creation of other resources, some secrets and certificates need to be uploaded manually. This may not necessarily be an easy operation as it would seems. Here are few tips that may help:

The secrets-copier.sh utility script automates copy operation of secrets between two keyvaults
Certificates should be generally uploaded manually in the certificates section of the keyvault. Uploading certificates in PEM format may be error prone, since the private key and the public certificate should be manually combined in a unique file. We found out that the best way to upload certificates is to previously convert them in pfx format with openssl. For example, openssl pkcs12 -inkey your_private.key -in your_public.crt -export -out your_pfx.pfx
Some secrets contain the keyword certificate, even if they do not represent a private/public key-pair, but only a public certificate. This is the reason why they have been uploaded as secrets and not certificates. This is for example the case of k8s-pagopa-proxy-test-secrets-pagopa-ca-chain-certs, which only represents the PagoPA public certificate. Do not try to upload it as certificate. It simply won't work!

Step by step

The commands assume you're trying to provision the dev environment and that you've already initialized once the Azure infrastructure with the az-init.sh script and that you already have a working .env file.

cd io-infrastructure-live

# Export env vars
cd utils
source az-export.sh

# Optionally set TERRAGRUNT_SOURCE_UPDATE to true
# to download the latest Terraform module each time
# an apply is executed
export TERRAGRUNT_SOURCE_UPDATE=true

# Deploy the components of the dev environment, under the West Europe Azure DC location
cd ../dev/westeurope

# Resource group
cd resource_group && terragrunt apply

# Networking: vnet and subnets
cd ../vnet_common && terragrunt apply
cd ../subnet_redis && terragrunt apply
cd ../subnet_function_app_app && terragrunt apply
cd ../subnet_function_app_admin && terragrunt apply
cd ../subnet_function_app_services && terragrunt apply
cd ../subnet_function_app_public && terragrunt apply
cd ../subnet_apim && terragrunt apply
cd ../subnet_ag-frontend && terragrunt apply
cd ../subnet_ag-to-k8s-01 && terragrunt apply
cd ../subnet_k8s-01 && terragrunt apply

# Private DNS zone
cd ../dns_zone_private_common && terragrunt apply

# Keyvault
cd ../key_vault && terragrunt apply

# CosmosDB: account, sql-database, sql-containers
cd ../cosmosdb_account_01 && terragrunt apply
cd ../cosmosdb_sql_database_db-01 && terragrunt apply
cd ../cosmosdb_container_message-status && terragrunt apply
cd ../cosmosdb_container_messages && terragrunt apply
cd ../cosmosdb_container_notification-status && terragrunt apply
cd ../cosmosdb_container_notifications && terragrunt apply
cd ../cosmosdb_container_profiles && terragrunt apply
cd ../cosmosdb_container_sender-services && terragrunt apply
cd ../cosmosdb_container_services && terragrunt apply

# Eventhub
cd ../eventhub_apim && terragrunt apply