Skip to content

A Terraform module for provisioning and installing Terraform Enterprise on Azure virtual machines as described in HashiCorp Validated Designs

License

Notifications You must be signed in to change notification settings

hashicorp/terraform-azurerm-terraform-enterprise-hvd

Terraform Enterprise HVD on Azure VM

Terraform module aligned with HashiCorp Validated Designs (HVD) to deploy Terraform Enterprise (TFE) on Microsoft Azure using Azure Virtual Machines with a container runtime. This module defaults to deploying TFE in the active-active operational mode, but external is also supported. Docker and Podman are the supported container runtimes.

TFE architecture

Prerequisites

General

  • TFE license file (e.g. terraform.hclic).
  • Terraform CLI >= 1.9 installed on clients/workstations that will be used to deploy TFE
  • General understanding of how to use Terraform (Community Edition)
  • General understanding of how to use Azure cloud
  • git CLI and Visual Studio Code editor installed on workstations are strongly recommended
  • Azure subscription that TFE will be deployed in with admin-like permissions to provision these resources in via Terraform CLI
  • Azure blob storage account for AzureRM remote state backend that will be used to manage the Terraform state of this TFE deployment (out-of-band from the TFE application) via Terraform CLI (Community Edition)

Networking

  • Azure VNet ID.
  • Load balancer subnet ID (if load balancer is to be internal)
  • Load balancer static IP address (if load balancer is to be internal)
  • VM subnet ID with service endpoints enabled for Microsoft.KeyVault, Microsoft.Sql, and Microsoft.Storage
  • VM subnet requires access to the egress endpoints listed here
  • Database subnet ID with service delegation configured for Microsoft.DBforPostgreSQL/flexibleServers for join action (Microsoft.Network/virtualNetworks/subnets/join/action)
  • Redis cache subnet ID
  • Ability to create private endpoints on the database and redis cache subnets
  • Chosen fully qualified domain name (FQDN) for your TFE instance (e.g. tfe.azure.example.com)

Network security group (NSG)/firewall rules

  • Allow TCP/443 ingress to load balancer subnet (if TFE load balancer is to be internal) or VM subnet (if TFE load balancer is to be external) from CIDR ranges of TFE users/clients, your VCS, and other systems (such as CI/CD) that will need to access TFE
  • (Optional) Allow TCP/9091 (HTTPS) and/or TCP/9090 (HTTP) ingress to VM subnet from monitoring/observability tool CIDR range (for scraping TFE metrics endpoints)
  • Allow TCP/5432 ingress to database subnet from VM subnet (for PostgreSQL traffic)
  • Allow TCP/6380 ingress to Redis cache subnetfrom VM subnet (for Redis TLS traffic)
  • Allow TCP/8201 between VMs on VM subnet (for TFE embedded Vault internal cluster traffic between TFE nodes when tfe_operational_mode is active-active)

TLS certificates

  • TLS certificate (e.g. cert.pem) and private key (e.g. privkey.pem) that matches your chosen fully qualified domain name (FQDN) for TFE
    • TLS certificate and private key must be in PEM format
    • Private key must not be password protected
  • TLS certificate authority (CA) bundle (e.g. ca_bundle.pem) corresponding with the CA that issues your TFE TLS certificates
    • CA bundle must be in PEM format
    • You may include additional certificate chains corresponding to external systems that TFE will make outbound connections to (e.g. your self-hosted VCS, if its certificate was issued by a different CA than your TFE certificate)

Key Vault secrets

Azure Key Vault containing the following TFE bootstrap secrets:

  • TFE license - raw contents of TFE license file (i.e. the string value of cat terraform.hclic)
  • TFE encryption password - used for TFE's embedded Vault; randomly generate this yourself, this is kept completely internal to the TFE system
  • TFE database password - used to create PostgreSQL Flexible Server; randomly generate this yourself (avoid the $ character as Azure PostgreSQL Flexible Server does not like it), fetched from within the module via data source and applied to the PostgreSQL Flexible Server resource
  • TFE TLS certificate - base64-encoded string of certificate file in PEM format
  • TFE TLS private key - base64-encoded string of private key file in PEM format
  • TFE custom CA bundle - base64-encoded string of custom CA bundle file in PEM format

📝 Note: See the TFE TLS Certificate Rotation doc for instructions on how to base64-encode the certificates with proper formatting before storing them as Key Vault secrets.

Compute

  • Supported operating systems:

    • RHEL 8.x, 9.x
    • Ubuntu 22.x, 24.x
  • One of the following mechanisms for shell access to TFE Azure Linux VMs within the Virtual Machine Scale Set (VMSS):

    • SSH key pair (public key would be a module input)
    • Username/password (password would be a module input)
  • Bastion host (if VM subnet is not reachable from clients/workstations)

Log forwarding (optional)

One of the following logging destinations for the TFE container logs:

  • Azure Log Analytics Workspace
  • A custom Fluent Bit configuration to forward logs to a custom destination

Usage

  1. Create/configure/validate the applicable prerequisites.

  2. Within the examples/main directory is a ready-made Terraform configuration which contains example scenarios on how to call and deploy this module. To get started, choose the example scenario that most closely matches your requirements. You can customize your deployment later by adding additional module inputs as you see fit (see the Deployment-Customizations for more details).

  3. Copy all of the Terraform files from your example scenario of choice into a new destination directory to create your Terraform configuration that will manage your TFE deployment. This is a common directory structure for managing multiple TFE deployments:

    .
    └── environments
        ├── production
        │   ├── backend.tf
        │   ├── main.tf
        │   ├── outputs.tf
        │   ├── terraform.tfvars
        │   └── variables.tf
        └── sandbox
            ├── backend.tf
            ├── main.tf
            ├── outputs.tf
            ├── terraform.tfvars
            └── variables.tf
    

    📝 Note: In this example, the user will have two separate TFE deployments; one for their sandbox environment, and one for their production environment. This is recommended, but not required.

  4. (Optional) Uncomment and update the AzureRM Remote State backend configuration provided in the backend.tf file with your own custom values. While this step is highly recommended so that your state file managing your TFE deployment is not stored on your local disk and others can safely collaborate, it is technically not required to use a remote backend config for your TFE deployment.

  5. Populate your own custom values into the terraform.tfvars.example file that was provided (in particular, the values enclosed in the <> characters). Then, remove the .example file extension such that the file is now named terraform.tfvars. If you would like to further customize your deployment beyond what is in your chosen example scenario, see the deployment customizations doc for more details on common customizations.

  6. Navigate to the directory of your newly created Terraform configuration for your TFE deployment, and run terraform init, terraform plan, and terraform apply.

  7. After your terraform apply finishes successfully, you can monitor the installation progress by connecting to your TFE VM shell (via SSH or other preferred method) and observing the cloud-init (custom_data) script logs:

    Connecting to the Azure VM

    ssh -i /path/to/vm_ssh_private_key tfeadmin@<vm-private-ip>

    Viewing the logs

    View the higher-level logs:

    tail -f /var/log/tfe-cloud-init.log

    View the lower-level logs:

    journalctl -xu cloud-final -f

    📝 Note: the -f argument is to follow the logs as they append in real-time, and is optional. You may remove the -f for a static view.

    Successful install log message

    The log files should display the following message after the cloud-init (custom_data) script finishes successfully:

    [INFO] TFE custom_data script finished successfully!
  8. After the cloud-init (custom_data) script finishes successfully, while still connected to the TFE VM shell, you can check the health status of TFE:

    cd /etc/tfe
    sudo docker compose exec tfe tfe-health-check-status
  9. Follow the steps to create the TFE initial admin user.

Docs

Below are links to various docs related to the customization and management of your TFE deployment:

Module support

This open source software is maintained by the HashiCorp Technical Field Organization, independently of our enterprise products. While our Support Engineering team provides dedicated support for our enterprise offerings, this open source software is not included.

  • For help using this open source software, please engage your account team.
  • To report bugs/issues with this open source software, please open them directly against this code repository using the GitHub issues feature.

Please note that there is no official Service Level Agreement (SLA) for support of this software as a HashiCorp customer. This software falls under the definition of Community Software/Versions in your Agreement. We appreciate your understanding and collaboration in improving our open source projects.

Requirements

Name Version
terraform >= 1.9
azurerm ~> 3.116

Providers

Name Version
azurerm ~> 3.116

Resources

Name Type
azurerm_dns_a_record.tfe resource
azurerm_key_vault_access_policy.postgres_cmk resource
azurerm_key_vault_access_policy.storage_account_cmk resource
azurerm_key_vault_access_policy.tfe_kv_reader resource
azurerm_lb.tfe resource
azurerm_lb_backend_address_pool.tfe_servers resource
azurerm_lb_probe.tfe resource
azurerm_lb_rule.tfe resource
azurerm_linux_virtual_machine_scale_set.tfe resource
azurerm_postgresql_flexible_server.tfe resource
azurerm_postgresql_flexible_server_configuration.tfe resource
azurerm_postgresql_flexible_server_database.tfe resource
azurerm_private_dns_a_record.blob_storage resource
azurerm_private_dns_a_record.redis resource
azurerm_private_dns_a_record.tfe resource
azurerm_private_dns_zone.blob_storage resource
azurerm_private_dns_zone.postgres resource
azurerm_private_dns_zone.redis resource
azurerm_private_dns_zone_virtual_network_link.blob_storage resource
azurerm_private_dns_zone_virtual_network_link.postgres resource
azurerm_private_dns_zone_virtual_network_link.redis resource
azurerm_private_endpoint.blob_storage resource
azurerm_private_endpoint.redis resource
azurerm_public_ip.tfe_lb resource
azurerm_redis_cache.tfe resource
azurerm_resource_group.tfe resource
azurerm_role_assignment.tfe_kv_reader resource
azurerm_role_assignment.tfe_sa_owner resource
azurerm_role_assignment.tfe_vmss_disk_encryption_set_reader resource
azurerm_storage_account.tfe resource
azurerm_storage_account_network_rules.tfe resource
azurerm_storage_container.tfe resource
azurerm_user_assigned_identity.postgres resource
azurerm_user_assigned_identity.storage_account resource
azurerm_user_assigned_identity.tfe resource
azurerm_client_config.current data source
azurerm_disk_encryption_set.vmss data source
azurerm_dns_zone.tfe data source
azurerm_image.custom data source
azurerm_key_vault.bootstrap data source
azurerm_key_vault_secret.tfe_database_password data source
azurerm_log_analytics_workspace.logging data source
azurerm_platform_image.latest_os_image data source
azurerm_private_dns_zone.tfe data source
azurerm_storage_account.tfe data source
azurerm_storage_container.tfe data source

Inputs

Name Description Type Default Required
bootstrap_keyvault_name Name of the 'bootstrap' Key Vault to use for bootstrapping TFE deployment. string n/a yes
bootstrap_keyvault_rg_name Name of the Resource Group where the 'bootstrap' Key Vault resides. string n/a yes
container_runtime Value of container runtime to use for TFE deployment. For Redhat, the default is podman, but optionally docker can be used. For Ubuntu, the default is docker. string n/a yes
db_subnet_id Subnet ID for PostgreSQL database. string n/a yes
friendly_name_prefix Friendly name prefix used for uniquely naming all Azure resources for this deployment. Most commonly set to either an environment (e.g. 'sandbox', 'prod'), a team name, or a project name. string n/a yes
location Azure region for this TFE deployment. string n/a yes
resource_group_name Name of resource group for this TFE deployment. Must be an existing resource group if create_resource_group is false. string n/a yes
tfe_database_password_keyvault_secret_name Name of the secret in the Key Vault that contains the TFE database password. string n/a yes
tfe_encryption_password_keyvault_secret_id ID of Key Vault secret containing TFE encryption password. string n/a yes
tfe_fqdn Fully qualified domain name of TFE instance. This name should resolve to the load balancer IP address and will be what clients use to access TFE. string n/a yes
tfe_license_keyvault_secret_id ID of Key Vault secret containing TFE license. string n/a yes
tfe_tls_ca_bundle_keyvault_secret_id ID of Key Vault secret containing TFE TLS custom CA bundle. string n/a yes
tfe_tls_cert_keyvault_secret_id ID of Key Vault secret containing TFE TLS certificate. string n/a yes
tfe_tls_privkey_keyvault_secret_id ID of Key Vault secret containing TFE TLS private key. string n/a yes
vm_subnet_id Subnet ID for Virtual Machine Scaleset (VMSS). string n/a yes
vnet_id ID of VNet where TFE will be deployed. string n/a yes
availability_zones List of Azure availability zones to spread TFE resources across. set(string)
[
"1",
"2",
"3"
]
no
common_tags Map of common tags for taggable Azure resources. map(string) {} no
create_blob_storage_private_endpoint Boolean to create a private endpoint and private DNS zone for TFE Storage Account. bool true no
create_lb Boolean to create an Azure Load Balancer for TFE. bool true no
create_postgres_private_endpoint Boolean to create a private endpoint and private DNS zone for PostgreSQL Flexible Server. bool true no
create_redis_private_endpoint Boolean to create a private DNS zone and private endpoint for Redis cache. bool true no
create_resource_group Boolean to create a new resource group for this TFE deployment. bool true no
create_tfe_private_dns_record Boolean to create a DNS record for TFE in a private Azure DNS zone. A private_dns_zone_name must also be provided when true. bool false no
create_tfe_public_dns_record Boolean to create a DNS record for TFE in a public Azure DNS zone. A public_dns_zone_name must also be provided when true. bool false no
custom_fluent_bit_config Custom Fluent Bit configuration for log forwarding. Only valid if log_fwd_destination_type is custom. string null no
docker_version Version of Docker to install on TFE VMSS. string "24.0.9" no
is_govcloud_region Boolean indicating whether this TFE deployment is in an Azure Government Cloud region. bool false no
is_secondary_region Boolean indicating whether this TFE deployment is for 'primary' region or 'secondary' region. bool false no
lb_is_internal Boolean to create an internal or external Azure Load Balancer for TFE. bool true no
lb_private_ip Private IP address for internal Azure Load Balancer. Only valid when lb_is_internal is true. string null no
lb_subnet_id Subnet ID for Azure load balancer. string null no
log_analytics_workspace_name Name existing Azure Log Analytics Workspace for log forwarding destination. Only valid if log_fwd_destination_type is log_analytics. string null no
log_analytics_workspace_rg_name Name of Resource Group where Log Analytics Workspace exists. string null no
log_fwd_destination_type Type of log forwarding destination. Valid values are 'log_analytics' or 'custom'. string "log_analytics" no
postgres_administrator_login Username for administrator login of PostreSQL database. string "tfeadmin" no
postgres_backup_retention_days Number of days to retain backups of PostgreSQL Flexible Server. number 35 no
postgres_cmk_keyvault_id ID of the Key Vault containing the customer-managed key (CMK) for encrypting the PostgreSQL Flexible Server database. string null no
postgres_cmk_keyvault_key_id ID of the Key Vault key to use for customer-managed key (CMK) encryption of PostgreSQL Flexible Server database. string null no
postgres_create_mode Determines if the PostgreSQL Flexible Server is being created as a new server or as a replica. string "Default" no
postgres_enable_high_availability Boolean to enable ZoneRedundant high availability with PostgreSQL database. bool false no
postgres_geo_backup_keyvault_key_id ID of the Key Vault key to use for customer-managed key (CMK) encryption of PostgreSQL Flexible Server geo-redundant backups. This key must be in the same region as the geo-redundant backup. string null no
postgres_geo_backup_user_assigned_identity_id ID of the User-Assigned Identity to use for customer-managed key (CMK) encryption of PostgreSQL Flexible Server geo-redundant backups. This identity must have 'Get', 'WrapKey', and 'UnwrapKey' permissions to the Key Vault. string null no
postgres_geo_redundant_backup_enabled Boolean to enable PostreSQL geo-redundant backup configuration in paired Azure region. bool true no
postgres_maintenance_window Map of maintenance window settings for PostgreSQL Flexible Server. map(number)
{
"day_of_week": 0,
"start_hour": 0,
"start_minute": 0
}
no
postgres_primary_availability_zone Number for the availability zone for the primary PostgreSQL Flexible Server instance to reside in. number 1 no
postgres_secondary_availability_zone Number for the availability zone for the standby PostgreSQL Flexible Server instance to reside in. number 2 no
postgres_sku PostgreSQL database SKU. string "GP_Standard_D4ds_v4" no
postgres_source_server_id ID of the source PostgreSQL Flexible Server to replicate from. Only valid when is_secondary_region is true and postgres_create_mode is Replica. string null no
postgres_storage_mb Storage capacity of PostgreSQL Flexible Server (unit is megabytes). number 65536 no
postgres_version PostgreSQL database version. number 15 no
private_dns_zone_name Name of existing private Azure DNS zone to create DNS record in. Required when create_tfe_private_dns_record is true. string null no
private_dns_zone_rg_name Name of Resource Group where private_dns_zone_name resides. Required when create_tfe_private_dns_record is true. string null no
public_dns_zone_name Name of existing public Azure DNS zone to create DNS record in. Required when create_tfe_public_dns_record is true. string null no
public_dns_zone_rg_name Name of Resource Group where public_dns_zone_name resides. Required when public_dns_zone_name is not null. string null no
redis_capacity The size of the Redis cache to deploy. Valid values for a SKU family of C (Basic/Standard) are 0, 1, 2, 3, 4, 5, 6, and for P (Premium) family are 1, 2, 3, 4. number 1 no
redis_family The SKU family/pricing group to use. Valid values are C (for Basic/Standard SKU family) and P (for Premium). string "P" no
redis_min_tls_version Minimum TLS version to use with Redis cache. string "1.2" no
redis_non_ssl_port_enabled Boolean to enable non-SSL port 6379 for Redis cache. bool false no
redis_sku_name Which SKU of Redis to use. Options are 'Basic', 'Standard', or 'Premium'. string "Premium" no
redis_subnet_id Subnet ID for Redis cache. string null no
redis_version Redis cache version. Only the major version is needed. number 6 no
secondary_vm_subnet_id VM subnet ID of existing TFE virtual machine scaleset (VMSS) in secondary region. Used to allow TFE VMs in secondary region access to TFE storage account in primary region. string null no
storage_account_blob_change_feed_enabled Boolean to enable blob change feed for the TFE Storage Account. bool false no
storage_account_blob_versioning_enabled Boolean to enable blob versioning for the TFE Storage Account. bool false no
storage_account_cmk_keyvault_id ID of the Key Vault containing the customer-managed key (CMK) for encrypting the TFE Storage Account. string null no
storage_account_cmk_keyvault_key_id ID of the customer-managed key (CMK) within Key Vault for encrypting the TFE Storage Account. string null no
storage_account_ip_allow List of IP addresses allowed to access TFE Storage Account. Set this to the IP address that you are running Terraform from to deploy this module to avoid a 403 error from Azure when creating the storage container. list(string) [] no
storage_account_public_network_access_enabled Boolean to enable public network access to Azure Blob Storage Account. Needs to be true for initial deployment. Optionally set to false after initial deployment. bool true no
storage_account_replication_type Type of replication to use for TFE Storage Account. string "GRS" no
tfe_capacity_concurrency Number of concurrent runs TFE can handle. number 10 no
tfe_capacity_cpu Number of CPU cores for TFE. number 0 no
tfe_capacity_memory Amount of memory in MB for TFE. number 2048 no
tfe_database_name PostgreSQL database name for TFE. string "tfe" no
tfe_database_parameters PostgreSQL server parameters for the connection URI. Used to configure the PostgreSQL connection. string "sslmode=require" no
tfe_hairpin_addressing Boolean to enable hairpin addressing for layer 4 load balancer with loopback prevention. Must be true when lb_is_internal is true. bool true no
tfe_http_port HTTP port for TFE application containers to listen on. number 8080 no
tfe_https_port HTTPS port for TFE application containers to listen on. number 8443 no
tfe_image_name Name of the TFE container image. Only change this if you are hosting the TFE container image in your own custom repository. string "hashicorp/terraform-enterprise" no
tfe_image_repository_password Password for container registry where TFE container image is hosted. Only set this if you are hosting the TFE container image in your own custom repository. string null no
tfe_image_repository_url Repository for the TFE image. Only change this if you are hosting the TFE container image in your own custom repository. string "images.releases.hashicorp.com" no
tfe_image_repository_username Username for container registry where TFE container image is hosted. Only change this if you are hosting the TFE container image in your own custom repository. string "terraform" no
tfe_image_tag Tag for the TFE container image. This represents the version of TFE to deploy. string "v202502-1" no
tfe_license_reporting_opt_out Boolean to opt out of license reporting. bool false no
tfe_log_forwarding_enabled Boolean to enable TFE log forwarding feature. bool false no
tfe_metrics_enable Boolean to enable metrics. bool false no
tfe_metrics_http_port HTTP port for TFE metrics endpoint. number 9090 no
tfe_metrics_https_port HTTPS port for TFE metrics endpoint. number 9091 no
tfe_object_storage_azure_use_msi Boolean to use a User-Assigned Identity (MSI) for TFE blob storage account authentication rather than a storage account key. bool true no
tfe_operational_mode Operational mode for TFE. Valid values are active-active or external. string "active-active" no
tfe_primary_resource_group_name Name of existing resource group of TFE deployment in primary region. Only set when is_secondary_region is true. string null no
tfe_primary_storage_account_name Name of existing TFE storage account in primary region. Only set when is_secondary_region is true. string null no
tfe_primary_storage_container_name Name of existing TFE storage container (within TFE storage account) in primary region. Only set when is_secondary_region is true. string null no
tfe_redis_use_auth Boolean to enable authentication to the Redis cache. bool true no
tfe_redis_use_tls Boolean to enable TLS for the Redis cache. bool true no
tfe_run_pipeline_docker_network Docker network where the containers that execute Terraform runs will be created. The network must already exist, it will not be created automatically. Leave as null to use the default network. string null no
tfe_run_pipeline_image Name of the Docker image to use for the run pipeline driver. string null no
tfe_tls_enforce Boolean to enforce TLS, Strict-Transport-Security headers, and secure cookies within TFE. bool false no
tfe_vault_disable_mlock Boolean to disable mlock for internal Vault. bool false no
vm_admin_username Admin username for VMs in VMSS. string "tfeadmin" no
vm_custom_image_name Name of custom VM image to use for VMSS. If not using a custom image, leave this blank. string null no
vm_custom_image_rg_name Name of Resource Group where vm_custom_image_name image resides. Only valid if vm_custom_image_name is not null. string null no
vm_disk_encryption_set_name Name of Disk Encryption Set to use for VMSS. string null no
vm_disk_encryption_set_rg Name of Resource Group where the Disk Encryption Set to use for VMSS exists. string null no
vm_enable_boot_diagnostics Boolean to enable boot diagnostics for VMSS. bool false no
vm_os_image The OS image to use for the VM. Options are: redhat8, redhat9, ubuntu2204, ubuntu2404. string "redhat9" no
vm_sku SKU for VM size for the VMSS. string "Standard_D4s_v4" no
vm_ssh_public_key SSH public key for VMs in VMSS. string null no
vmss_instance_count Number of VM instances to run in the Virtual Machine Scaleset (VMSS). number 1 no

Outputs

Name Description
tfe_database_host FQDN and port of PostgreSQL Flexible Server.
tfe_database_name Name of PostgreSQL Flexible Server database.
tfe_object_storage_azure_account_name Name of primary TFE Azure Storage Account.
tfe_object_storage_azure_container_name Name of TFE Azure Storage Container.
url URL of TFE application based on tfe_fqdn input.

About

A Terraform module for provisioning and installing Terraform Enterprise on Azure virtual machines as described in HashiCorp Validated Designs

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •