Azure Cloud Starter Project

Create and manage the basic resources needed for a new Azure project
using Terraform and Github Actions.

Getting Started

You will need to install the Azure CLI and log-in. I would also reccommend you install a secrets or password manager to hold the credentials we will create. I'll be using a free Bitwarden account along with their bitwarden-cli to manage my secrets for this demonstration.

• Install Azure CLI

  brew update && \
  brew install azure-cli

• Install Bitwarden CLI

  brew install bitwarden-cli

• Login

  # Authorize the Azure CLI
  az login

---

IAM Setup

You need to create a service account to represent your digital self and use that to run terraform locally. In production, a unique service account should be created for running and applying the terraform jobs,and it should create smaller accounts at instantiation to run the infra it provisions.

'Owner' level access is required because we need to create role assignments. This may potentially be scoped down to 'User Access Administrator' + 'Contributor'

1. Create your service principle and then add the resulting data to KeePassXC / Bitwarden for now. You will need it again multiple times.

   SUBSCRIPTION=$(az account show --query id --output tsv)
   SP_NAME="myserviceaccount"
   
   az ad sp create-for-rbac --sdk-auth \
     --display-name="${SP_NAME}" \
     --role="Owner" \
     --scopes="/subscriptions/$SUBSCRIPTION"

2. Setup the Azure Active Directory Permissions for your Service Principle. This is required in order to set AD roles in terraform.

   • Login to https://portal.azure.com/

   • Navigate to Azure Active Directory

   • Select Roles and administrators from the left-side menu

   • Click Application administrator

   • Click Add Assignments

   • Search for your service accounts name

   • Repeat for Application Developer Role.

Creating the State Bucket

• Before we start you should login to Azure again, but now as the service principle we created. We will use this account to create the terraform state bucket. That way, only the service-principle will have access to it.

  az login --service-principal \
     --username $(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientId") |.value') \
     --password $(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientSecret") |.value') \
     --tenant $(bw get item admin-robot |jq -r '.fields[] |select(.name=="tenantId") |.value')

• Now Create the Terraform state bucket

  export SUBSCRIPTION=$(az account show --query id --output tsv)    
  export KIND="StorageV2"
  export LOCATION="westeurope"
  export RG_NAME="example-tf-state"
  export STORAGE_NAME="examplertfstatebucket"
  export STORAGE_SKU="Standard_RAGRS"
  export CONTAINER_NAME="exampletfstate"
  
  az group create \
    -l="${LOCATION}" \
    -n="${RG_NAME}"
  
  az storage account create \
    --name="${STORAGE_NAME}" \
    --resource-group="${RG_NAME}" \
    --location="${LOCATION}" \
    --sku="${STORAGE_SKU}" \
    --kind="${KIND}"
  
  az storage account encryption-scope create \
    --account-name="${STORAGE_NAME}"  \
    --key-source Microsoft.Storage \
    --name="tfencryption"\
    --resource-group="${RG_NAME}" \
    --subscription="${SUBSCRIPTION}"
  
  az storage container create \
      --name="${CONTAINER_NAME}" \
      --account-name="${STORAGE_NAME}" \
      --resource-group="${RG_NAME}" \
      --default-encryption-scope="tfencryption" \
      --prevent-encryption-scope-override="true" \
      --auth-mode="login" \
      --fail-on-exist \
      --public-access="off"

Initializing your Terraform Project

• Add your state-bucket details to the providers.tf file if you made any customisations

  backend "azurerm" {
     resource_group_name  = "example-tf-state"
     storage_account_name = "examplertfstatebucket"
     container_name       = "exampletfstate"
     key                  = "example.terraform.tfstate"
  }

• Add your ip-address and personal azure account's clientID to the environment-base.tf file under the Firewall header.

  # Firewall
  allowed_ips = ["192.168.50.1"]
  admin_users = ["clientId-goes-here"]

• Initialize the terraform project

  docker run --platform linux/amd64 -it \
     -e ARM_CLIENT_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientId") |.value') \
     -e ARM_CLIENT_SECRET=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientSecret") |.value') \
     -e ARM_SUBSCRIPTION_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="subscriptionId") |.value') \
     -e ARM_TENANT_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="tenantId") |.value') \
     -v $(pwd):/terraform -w /terraform \
     hashicorp/terraform init

• Plan / Apply resources

  docker run --platform linux/amd64 -it \
     -e ARM_CLIENT_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientId") |.value') \
     -e ARM_CLIENT_SECRET=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientSecret") |.value') \
     -e ARM_SUBSCRIPTION_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="subscriptionId") |.value') \
     -e ARM_TENANT_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="tenantId") |.value') \
     -v $(pwd):/terraform -w /terraform \
     hashicorp/terraform apply

• Destroy Resources

  docker run --platform linux/amd64 -it \
     -e ARM_CLIENT_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientId") |.value') \
     -e ARM_CLIENT_SECRET=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="clientSecret") |.value') \
     -e ARM_SUBSCRIPTION_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="subscriptionId") |.value') \
     -e ARM_TENANT_ID=$(bw get item admin-robot |jq -r '.fields[] |select(.name=="tenantId") |.value') \
     -v $(pwd):/terraform -w /terraform \
     hashicorp/terraform destroy

• Cleanup

  # ToDo

Instance Types and Limits

This project is tested using NVadsA10v5, NCasT4_v3, and NVv3 instances which utilize Nvidia A10, T4 and M60 GPUs. If you do not need a GPU, you are advised to consider Hetzner or Equinix who have better prices on CPU-only instances. Buildstar Online the following quota limits in west-europe:

• 72 vCores for NVadsA10v5
• 64 vCores for NCasT4_v3
• 48 vCores for NVv3

All prices are for Spot/hourly in the closest datacenter to Amsterdam, NL

NVadsA10v5

• NVadsA10v5 series virtual machines are powered by NVIDIA A10 GPUs and AMD EPYC 74F3V(Milan) CPUs with a base frequency of 3.2 GHz, peak frequency of all cores of 4.0 GHz.

  Instance Size	GPUs	GPU RAM	vCPUs	RAM (GiB)	Disk Size (GB)	Network (Gbps)	Spot Price
  Standard_NV6ads_A10_v5	1/6	4	6	55	180	5	€0.1362
  Standard_NV12ads_A10_v5	1/3	8	12	110	360	10	€0.2724
  Standard_NV18ads_A10_v5	1/2	12	18	220	720	20	€0.4802
  Standard_NV36ads_A10_v5	1	24	36	440	1440	40	€0.9604
  Standard_NV72ads_A10_v5	2	48	72	880	2880	80	€1.9568

NCasT4_v3

• NCasT4_v3 series virtual machines are powered by Nvidia Tesla T4 GPUs and AMD EPYC 7V12(Rome) CPUs.

  Instance Size	GPUs	GPU RAM	vCPUs	RAM (GiB)	Disk Size (GB)	Network (Gbps)	Spot Price
  Standard_NC4as_T4_v3	1	16	4	28	180	8	€0.1275
  Standard_NC8as_T4_v3	1	16	8	56	360	8	€0.1821
  Standard_NC16as_T4_v3	1	16	16	110	360	8	€0.2916
  Standard_NC64as_T4_v3	4	64	64	440	2880	32	€1.0539

NVv3

• The NVv3 series virtual machines are powered by NVIDIA Tesla M60 GPUs and NVIDIA GRID technology with Intel E5-2690 v4 (Broadwell) CPUs and Intel Hyper-Threading Technology.

  Instance Size	GPUs	GPU RAM	vCPUs	RAM (GiB)	Disk Size (GB)	Network (Gbps)	Spot Price
  Standard_NV12s_v3	1	8	12	112	320	6	€0.1317
  Standard_NV24s_v3	2	16	24	224	640	12	€0.2632
  Standard_NV48s_v3	4	32	48	448	1280	24	€0.5264

Requirements

Name	Version
azuread	~>2.36.0
azurerm	~>3.47.0
random	~>3.4.3
time	~>0.9.1
tls	~>4.0.4

Providers

Name	Version
azurerm	~>3.47.0

Modules

Name	Source	Version
environment-base	github.com/cloudymax/modules-azure-tf-base	n/a
scale-set	github.com/cloudymax/modules-azure-tf-scale-set	n/a

Resources

Name	Type
azurerm_client_config.current	data source

Inputs

Name	Description	Type	Default	Required
account_replication_type	n/a	string	"LRS"	no
account_tier	n/a	string	"Standard"	no
admin_identity	n/a	string	"bradley"	no
allowed_ips	n/a	list(string)	n/a	yes
cr_sku	n/a	string	"Basic"	no
environment	n/a	string	"dmeo"	no
eviction_policy	n/a	string	"Deallocate"	no
github_username	n/a	string	n/a	yes
hostname	n/a	string	"azurespot"	no
kv_sku_name	n/a	string	"standard"	no
location	n/a	string	"westeurope"	no
log_storage_tier	n/a	string	"Hot"	no
max_bid_price	n/a	string	n/a	yes
overprovision	n/a	bool	false	no
priority	n/a	string	"Spot"	no
resource_group	n/a	string	"demo-rg"	no
scale_in_force_deletion_enabled	n/a	bool	true	no
scale_in_rule	n/a	string	"NewestVM"	no
scale_set_name	n/a	string	"scale-set"	no
spot_restore_enabled	n/a	bool	true	no
spot_restore_timeout	n/a	string	"PT1H30M"	no
ultra_ssd_enabled	n/a	bool	false	no
user_data_path	n/a	string	"./NVadsA10v5.yaml"	no
username	n/a	string	n/a	yes
vm_instances	n/a	number	1	no
vm_network_interface	n/a	string	"vm-nic"	no
vm_os_disk_size_gb	n/a	number	64	no
vm_sku	n/a	string	n/a	yes

Outputs

No outputs.