Terraform Remote State Manager(tfremote)

tf is a python package for managing terraform remote state for: Google(Gcloud), AWS, and Azure. It sets a defined structure for all cloud providers by removing the overheard of configuring and managing the path in storage buckets.

It works with:

👉 Google Storage Bucket

👉 AWS S3

👉 Azure Storage

❗️ Note Best practice is to make sure buckets are versioned.

Install package

pip install tfremote --upgrade

Environment setup

• Install Python 3.6+

• Using virtualenv is strongly recommended:

python3 -m venv <venv name>

• Terraform 0.14.0 and above (download: https://www.terraform.io/downloads.html)

Default log level is WARNING, to change:

export TF_LOG_LEVEL to any of these: 'CRITICAL', 'ERROR', 'WARNING', 'INFO', 'DEBUG'

❗️ Important - Two variables are required for using tf package (used set creat path in remote storage):

• teamid
• prjid

Required variables can be defined using:

• As inline variables e.g.: -var='teamid=demo-team' -var='prjid=demo-project'
• Inside .tfvars file e.g.: -var-file=<tfvars file location>

Two optional variables:

workspace and state_key can be defined using:

• -w=<workspace_name>. If no workspace is provided default workspace is used.

• s=<state_key name>. If no key is provided terraform is used.

Path created in S3 backend: /<teamid>/<prjid>/<workspace>/<state-key>.tfstate

For more information refer to Terraform documentation

Setup environment variables

Workspace list file location TF_WORKSPACE_FILE_LOCATION

export TF_WORKSPACE_FILE_LOCATION=<workspace yml file location>

Reference file: link

AWS

❗️ Important - s3 bucket for remote state should reside in us-west-2

Set these env variables:

export TF_AWS_BUCKET=<your_remote_state_bucket_name>
export TF_AWS_BUCKET_REGION=us-west-2

One of below environment variable is required:

export TF_AWS_PROFILE=<aws profile to use>

or

export AWS_ACCESS_KEY_ID=<aws access key>
export AWS_SECRET_ACCESS_KEY=<aws secret access key>

Azure

To create storage for remote state there is handy script.

Run scripts/remote_state.sh (fill in the required information)

Set below env variables:

export TF_AZURE_STORAGE_ACCOUNT=<remote state storage account name>
export TF_AZURE_CONTAINER=<remote state container>
export ARM_ACCESS_KEY=<storage account access key>

GCP(gcloud)

https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform

Set below env variables:

export TF_GCLOUD_BUCKET=<remote state storage bucket name>
export TF_GCLOUD_CREDENTIALS=json credentials file path>

Usage

For GCP(gcloud):

tf plan -c=gcloud -var=teamid=demo-team -var=prjid=demo-app -w=demo-workspace

The structure in Google Storage Bucket:

For AWS:

tf plan -c=aws -var=teamid=demo-team -var=prjid=demo-app -w=demo-workspace

The structure in AWS S3:

If you need to specify state_key in S3, specify -s=tryme-key

For Azure:

tf plan -c=azure -var=teamid=demo-team -var=prjid=demo-app -w=demo-workspace

The structure in Azure Storage:

For more available options:

tf --help
usage: tf [-h] [-var] [-var-file] [-c] [-w] [-wp] [-s] [-no-color] [-json] [-out] [-f] [-nf] [-v]

Terraform remote state wrapper package
--------------------------------------
Usage: Set below env variables to begin (more information: https://github.com/tomarv2/tfremote):
TF_WORKSPACE_FILE_LOCATION
aws: TF_AWS_BUCKET, TF_AWS_BUCKET_REGION=us-west-2, TF_AWS_PROFILE or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
azure: TF_AZURE_STORAGE_ACCOUNT, TF_AZURE_CONTAINER, ARM_ACCESS_KEY
gcloud: TF_GCLOUD_BUCKET, TF_GCLOUD_CREDENTIALS

optional arguments:
  -h, --help  show this help message and exit
  -var        Set Terraform configuration variable. This flag can be set multiple times
  -var-file   Set Terraform configuration variables from a file. This flag can be set multiple times
  -c          Specify cloud provider (default: 'aws'). Supported values: gcloud, aws, or azure
  -w          Specify existing workspace name(default: 'default')
  -wp         Overwrite workspace directory path structure
  -s          File name in remote state (default: 'terraform.tfstate')
  -no-color   Disables terminal formatting sequences in the output
  -json       Enables the machine readable JSON UI output
  -out        Writes the generated plan to the given filename in an opaque file format
  -f          Enable FIPS endpoints (default: True)
  -nf         Disable FIPS endpoints
  -v          show program's version number and exit