From 2fbe5fc4cb11df0e92e4e35e5a66856bc02e1bac Mon Sep 17 00:00:00 2001 From: Andy Stevens Date: Thu, 29 Nov 2018 10:11:18 -0500 Subject: [PATCH 1/4] [New] Secrets Management with Terraform High level overview of how to secure both a .tf file and a .tfstate file. Will be updated to include a link to the Terraform backends guide once it is written. --- .../index.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 docs/applications/configuration-management/secrets-management-with-terraform/index.md diff --git a/docs/applications/configuration-management/secrets-management-with-terraform/index.md b/docs/applications/configuration-management/secrets-management-with-terraform/index.md new file mode 100644 index 00000000000..2b2403ed700 --- /dev/null +++ b/docs/applications/configuration-management/secrets-management-with-terraform/index.md @@ -0,0 +1,117 @@ +--- +author: + name: Linode + email: docs@linode.com +description: 'How to Manage Secrets with Terraform' +keywords: ['terraform','secrets','secrets management','backend','hcl'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +published: 2018-11-26 +modified: 2018-11-26 +modified_by: + name: Linode +title: "Secrets Management with Terraform" +contributor: + name: Linode +external_resources: +- '[Terraform Variables](https://www.terraform.io/docs/configuration/variables.html)' +- '[Terraform Backend Configuration](https://www.terraform.io/docs/backends/config.html)' +- '[Terraform Backend Types](https://www.terraform.io/docs/backends/types/index.html)' +- '[Terraform State Storing and Locking](https://www.terraform.io/docs/backends/state.html)' +- '[GitHub Discussion on Storing Sensitive Information with Terraform](https://github.com/hashicorp/terraform/issues/516)' + +--- + +Terraform is an Infrastructure as Code (IaC) tool that allows you to write declarative code to manage your infrastructure. However, in order to implement IaC with Terraform it is necessary to supply secrets such as server passwords and API tokens within your code. This guide will discuss the known methods for securing those secrets within Terraform. + +## Keeping Secrets Out of .tf Files + +In Terraform, `.tf` files contain the declarative code used to create, manage, and destroy infrastructure. This code is often committed to a version control system like Git, using a platform like GitHub, and shared within a team. Because it is easy for this information to become public-facing, it is important that you make sure your committed code is free of secrets. There are currently a few ways to keep your secrets out of your `.tf` files. + +### Variable Files + +Terraform employs *variable files* to help separate variable values from `.tf` files. When Terraform runs a command like `plan` or `apply`, it looks through the working directory for a file named `terraform.tfvars`, or for files with the `.auto.tfvars` extension, and populates the variable values they contain wherever they are [interpolated](https://www.terraform.io/docs/configuration/interpolation.html). This means that you can provide variable definitions in a `.tf` file and the values for those variables in a `.tfvars` file. For example, you might have a `linode-infrastructure.tf` file with a Provider block that requires an API access token: + +{{< file "linode-infrastructure.tf" >}} +variable "token" { + description = "Your API Access Token" +} + +provider "linode" { + token = "${var.token}" +} +{{< /file >}} + +You would then provide a `terraform.tfvars` file with the following value: + +{{< file "terraform.tfvars" >}} +token = 'a1b2c3d4e5f6g7h8i9j0' +{{< /file >}} + +The `terraform.tfvars` file would then be added added to a `.gitignore` file and kept out of version control, allowing you to safely commit the `linode-infrastructure.tf` file. For ease of use with large `terraform.tfvars` files, it might be beneficial to include an example file (`terraform.tfvars.example`) in your Git repository with all of the variable names defined, so that it is easier for a team member to recreate this file locally. + +Additionally, you can supply any file with a `.tfvars` extension to Terraform by using the `-var-file` option. + + terraform apply -var-file=myvars.tfvars + +Supplying multiple `.tfvars` file is another way to further separate secret variables and non-secret variables. + +### Environmental Variables + +Terraform allows for the use of environmental variables. These variables have the prefix `TF_VAR_` and are supplied at the command line. Using the above example of an API access token, you could export the variable like so: + + export TF_VAR_token=a1b2c3d4e5f6g7h8i9j0 + +You could then interpolate the value as normal: + + ${var.token} + +You could also include the variable when running `terraform plan` or `terraform apply`: + + TF_VAR_token=a1b2c3d4e5f6g7h8i9j0 terraform apply + +This method does commit the environmental variable to your shell's history, so take care when using this method. + +### Supply Variables at Prompt + +If Terraform does not find a value for a defined variable in a `.tf` file, in a `.tfvars` file, or in an environmental variable, it will prompt you for a value before running an action. + +{{< output >}} +$ terraform plan +var.token + Your API Access Token + + Enter a value: +{{< /output >}} + +This method is a bit easier to use than supplying environmental variables, and has the added benefit of using the description you set up when defining your variable to ease deployment. + +## How to Manage Your State File + +While it is relatively easy to keep secrets out of `.tf` files using any of the above methods, there is another file you need to be aware of when managing secrets, and that is the `terraform.tfstate` file. This *state file* contains a JSON object that holds your managed infrastructure's current state, i.e., a snapshot of the various attributes of your infrastructure at the time it was last modified. It is generated on `terraform apply` and is a necessary part of the Terraform process, as it matches the declarative code of the `.tf` file to real world infrastructure. + +However, as of the writing of this guide, **sensitive information used to generate your Terraform state can be stored as plain text in the `terraform.tfstate` file**. For example, if you are working with the Linode Provider and have supplied a root password for your Linode Instance, that root password will be stored as plain text in the state file. The following are some strategies for storing and sharing your state files. + +### Remote Backends + +Terraform [Backends](https://www.terraform.io/docs/backends/index.html) allow the user to securely store their state in a remote location, such as a key/value store like [Consul](https://www.consul.io/), or an S3 compatible bucket storage like [Minio](https://www.minio.io/). This allows the Terraform state to be read from the remote store, and because the state only ever exists locally in memory, there is no worry about storing secrets in plain text. Some Backends, like Consul, also allow for state locking, so that if one user is applying a state another user will be unable to make any changes. + +Using a Backend is the preferred way to share a Terraform state file. + + +### Use a Dummy Password + +It is possible to supply a dummy password to Terraform and later change that password manually to a more secure one. For instance, if you were to create a Linode Instance with a dummy root password, you could later change that password from the command line or in the Linode Manager. Note, however, that any attempt to change the password in a `.tf` file will result in the creation of new resources on `terraform apply`. + +### Encrypting Secrets + +There are some third-party tools that allow you to encrypt your secrets. + +[git-crypt](https://github.com/AGWA/git-crypt) allows you to encrypt files when they are committed to a Git repository, and decrypt files when they are checked out. With `git-crypt` you could encrypt your `terraform.tfstate` file and securely commit it to version control. Note that for this option, you must initialize `git-crypt` in a repository before committing your state file or the state file will not be eligible for encryption. + +Another option is [Terrahelp](https://github.com/opencredo/terrahelp). Terrahelp allows you to encrypt and decrypt a whole state file, or just the variables you have include in your `terraform.tfvars` file. + +### Privatize Version Control + +If you are unwilling or unable to use the above options to help manage your state file, and if you are using a platform like GitHub or GitLab to share your state files, then at the very least the repository should be private. + + From d2a30292cf92d003df30038e4968f4a3c091cec9 Mon Sep 17 00:00:00 2001 From: Andy Stevens Date: Thu, 29 Nov 2018 10:17:34 -0500 Subject: [PATCH 2/4] Spelling fix added terrahelp to vale --- ci/vale/dictionary.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/ci/vale/dictionary.txt b/ci/vale/dictionary.txt index 8d59b486f5b..00fa84d74b8 100644 --- a/ci/vale/dictionary.txt +++ b/ci/vale/dictionary.txt @@ -1255,6 +1255,7 @@ tcp tcpwrappers teamspeak tensorflow +terrahelp terraria testdb testfile From 2d7d12d0f746d1e0169b93ba042ff7d6ffc63e3b Mon Sep 17 00:00:00 2001 From: nmelehan Date: Tue, 11 Dec 2018 18:02:48 +0000 Subject: [PATCH 3/4] Copy edit --- .../index.md | 121 ++++++++++++------ 1 file changed, 84 insertions(+), 37 deletions(-) diff --git a/docs/applications/configuration-management/secrets-management-with-terraform/index.md b/docs/applications/configuration-management/secrets-management-with-terraform/index.md index 2b2403ed700..065ca374742 100644 --- a/docs/applications/configuration-management/secrets-management-with-terraform/index.md +++ b/docs/applications/configuration-management/secrets-management-with-terraform/index.md @@ -13,27 +13,28 @@ title: "Secrets Management with Terraform" contributor: name: Linode external_resources: -- '[Terraform Variables](https://www.terraform.io/docs/configuration/variables.html)' +- '[Terraform Input Variable Configuration](https://www.terraform.io/docs/configuration/variables.html)' - '[Terraform Backend Configuration](https://www.terraform.io/docs/backends/config.html)' - '[Terraform Backend Types](https://www.terraform.io/docs/backends/types/index.html)' -- '[Terraform State Storing and Locking](https://www.terraform.io/docs/backends/state.html)' -- '[GitHub Discussion on Storing Sensitive Information with Terraform](https://github.com/hashicorp/terraform/issues/516)' - +- '[Terraform State Storage and Locking](https://www.terraform.io/docs/backends/state.html)' +- '[GitHub Discussion - Storing Sensitive Values in State Files](https://github.com/hashicorp/terraform/issues/516)' --- -Terraform is an Infrastructure as Code (IaC) tool that allows you to write declarative code to manage your infrastructure. However, in order to implement IaC with Terraform it is necessary to supply secrets such as server passwords and API tokens within your code. This guide will discuss the known methods for securing those secrets within Terraform. +Terraform is an Infrastructure as Code (IaC) tool that allows you to write declarative code to manage your infrastructure. In order to implement IaC with Terraform it is necessary to supply secrets, such as server passwords and API tokens, within your code. This guide will discuss methods for securing those secrets within Terraform. ## Keeping Secrets Out of .tf Files -In Terraform, `.tf` files contain the declarative code used to create, manage, and destroy infrastructure. This code is often committed to a version control system like Git, using a platform like GitHub, and shared within a team. Because it is easy for this information to become public-facing, it is important that you make sure your committed code is free of secrets. There are currently a few ways to keep your secrets out of your `.tf` files. +In Terraform, `.tf` files contain the declarative code used to create, manage, and destroy infrastructure. This code is often committed to a version control system like Git, using a platform like GitHub, and shared within a team. Because it is easy for this information to become public-facing, it is important that you make sure your committed code is free of secrets. + +### Input Variables -### Variable Files +Terraform configurations in `.tf` files can accept values from [*input variables*](https://www.terraform.io/docs/configuration/variables.html). These variables are included in your configuration using Terraform's [interpolation syntax](https://www.terraform.io/docs/configuration/interpolation.html). -Terraform employs *variable files* to help separate variable values from `.tf` files. When Terraform runs a command like `plan` or `apply`, it looks through the working directory for a file named `terraform.tfvars`, or for files with the `.auto.tfvars` extension, and populates the variable values they contain wherever they are [interpolated](https://www.terraform.io/docs/configuration/interpolation.html). This means that you can provide variable definitions in a `.tf` file and the values for those variables in a `.tfvars` file. For example, you might have a `linode-infrastructure.tf` file with a Provider block that requires an API access token: +For example, you might have a `linode-infrastructure.tf` file with a provider block that requires an API access token. The `token` variable definition is declared inside your `.tf` file and is then interpolated inside the provider declaration with the `"${var.token}"` syntax: {{< file "linode-infrastructure.tf" >}} variable "token" { - description = "Your API Access Token" + description = "Your API access token" } provider "linode" { @@ -41,77 +42,123 @@ provider "linode" { } {{< /file >}} -You would then provide a `terraform.tfvars` file with the following value: +Variable definitions are written in `.tf` files. In this example, it's the same file as your provider configuration, but the definition could have been in a separate `.tf` file too. + +{{< note >}} +Your variable definitions can have default values assigned to them. Here's an example that encodes Linode's Newark data center as the default value for a `region` variable: + +{{< file "variables.tf" >}} +variable "region" { + description = "The region to deploy Linode instances in" + default = "us-east" +} +{{< /file >}} + +You could later use this variable when declaring your Linode instances. +{{< /note >}} + +### Assigning Variable Values in a File + +The values assigned to your variables (aside from default values) are not included in the variable definitions in your `.tf` files. Instead, the values are stored in separate files with the `.tfvars` extension. When Terraform runs a command like `plan` or `apply`, it automatically looks through the working directory for a file named `terraform.tfvars`, or for files with the `.auto.tfvars` extension. + +Here's an example `terraform.tfvars` which supplies a value for the `token` variable from the previous example: {{< file "terraform.tfvars" >}} -token = 'a1b2c3d4e5f6g7h8i9j0' +token = 'your-token-value' {{< /file >}} -The `terraform.tfvars` file would then be added added to a `.gitignore` file and kept out of version control, allowing you to safely commit the `linode-infrastructure.tf` file. For ease of use with large `terraform.tfvars` files, it might be beneficial to include an example file (`terraform.tfvars.example`) in your Git repository with all of the variable names defined, so that it is easier for a team member to recreate this file locally. +You would then add the `terraform.tfvars` file to your `.gitignore` file and keep it out of version control. This strategy allows you to safely commit the `linode-infrastructure.tf` file. + +For ease of use with large `terraform.tfvars` files, it might be beneficial to include an example `terraform.tfvars.example` in your Git repository with all of the variable names recorded (but none of the values entered). Team members could then copy this example into their local repository's `terraform.tfvars` and enter the appropriate values. -Additionally, you can supply any file with a `.tfvars` extension to Terraform by using the `-var-file` option. +{{< note >}} +Variable value files with names that don't match `terraform.tfvars` or `*.auto.tfvars` can be specified with the `-var-file` option: terraform apply -var-file=myvars.tfvars -Supplying multiple `.tfvars` file is another way to further separate secret variables and non-secret variables. +Supplying multiple `.tfvars` files is another way to further separate secret variables and non-secret variables; e.g.: -### Environmental Variables + terraform apply \ + -var-file=non-secret-variables.tfvars \ + -var-file=secret-variables.tfvars -Terraform allows for the use of environmental variables. These variables have the prefix `TF_VAR_` and are supplied at the command line. Using the above example of an API access token, you could export the variable like so: +{{< /note >}} - export TF_VAR_token=a1b2c3d4e5f6g7h8i9j0 +### Assigning Values in Environment Variables -You could then interpolate the value as normal: +Terraform allows you to keep input variable values in environment variables. These variables have the prefix `TF_VAR_` and are supplied at the command line. Using the above example of an API access token, you could export the variable and use it like so: - ${var.token} + export TF_VAR_token=your-token-value + terraform apply -You could also include the variable when running `terraform plan` or `terraform apply`: +You could also include the variable on the same line when running `terraform plan` or `terraform apply`: - TF_VAR_token=a1b2c3d4e5f6g7h8i9j0 terraform apply + TF_VAR_token=your-token-value terraform apply -This method does commit the environmental variable to your shell's history, so take care when using this method. +{{< caution >}} +This method commits the environment variable to your shell's history, so take care when using this method. +{{< /caution >}} + +### Assigning Values in Command-Line Flags + +Variable values can be set with the `-var` option: + + terraform apply -var 'token=your-token-value' + +{{< caution >}} +This method commits the command-line variable to your shell's history, so take care when using this method. +{{< /caution >}} ### Supply Variables at Prompt -If Terraform does not find a value for a defined variable in a `.tf` file, in a `.tfvars` file, or in an environmental variable, it will prompt you for a value before running an action. +If Terraform does not find a default value for a defined variable; or a value from a `.tfvars` file, environment variable, or CLI flag; it will prompt you for a value before running an action: {{< output >}} $ terraform plan var.token - Your API Access Token + Your API access token Enter a value: {{< /output >}} -This method is a bit easier to use than supplying environmental variables, and has the added benefit of using the description you set up when defining your variable to ease deployment. +This method is a bit easier to use than supplying environment variables, and has the added benefit of displaying the description you set up when defining your variable. ## How to Manage Your State File -While it is relatively easy to keep secrets out of `.tf` files using any of the above methods, there is another file you need to be aware of when managing secrets, and that is the `terraform.tfstate` file. This *state file* contains a JSON object that holds your managed infrastructure's current state, i.e., a snapshot of the various attributes of your infrastructure at the time it was last modified. It is generated on `terraform apply` and is a necessary part of the Terraform process, as it matches the declarative code of the `.tf` file to real world infrastructure. +While it is relatively easy to keep secrets out of `.tf` files using any of the above methods, there is another file you need to be aware of when managing secrets, and that is the `terraform.tfstate` file. + +This *state file* contains a JSON object that holds your managed infrastructure's current state. This state is a snapshot of the various attributes of your infrastructure at the time it was last modified. It is generated on `terraform apply` and is a necessary part of the Terraform process, as it maps the declarative code of your `.tf` files to your real world infrastructure. -However, as of the writing of this guide, **sensitive information used to generate your Terraform state can be stored as plain text in the `terraform.tfstate` file**. For example, if you are working with the Linode Provider and have supplied a root password for your Linode Instance, that root password will be stored as plain text in the state file. The following are some strategies for storing and sharing your state files. +As of the writing of this guide, **sensitive information used to generate your Terraform state can be stored as plain text in the `terraform.tfstate` file.** For example, if you are working with the Linode provider and have supplied a root password for your Linode instance, that root password will be stored as plain text in the state file. **Avoid checking your `terraform.tfstate` file into your version control repository.** Instead, the following are some strategies for storing and sharing your state files. ### Remote Backends -Terraform [Backends](https://www.terraform.io/docs/backends/index.html) allow the user to securely store their state in a remote location, such as a key/value store like [Consul](https://www.consul.io/), or an S3 compatible bucket storage like [Minio](https://www.minio.io/). This allows the Terraform state to be read from the remote store, and because the state only ever exists locally in memory, there is no worry about storing secrets in plain text. Some Backends, like Consul, also allow for state locking, so that if one user is applying a state another user will be unable to make any changes. +Terraform [*backends*](https://www.terraform.io/docs/backends/index.html) allow the user to securely store their state in a remote location, such as a key/value store like [Consul](https://www.consul.io/), or an S3 compatible bucket storage like [Minio](https://www.minio.io/). This allows the Terraform state to be read from the remote store, and because the state only ever exists locally in memory, there is no worry about storing secrets in plain text. -Using a Backend is the preferred way to share a Terraform state file. +Some backends, like Consul, also allow for state locking. If one user is applying a state, another user will be unable to make any changes. +Using a Terraform backend is the preferred way to share a Terraform state file. -### Use a Dummy Password +### Encrypting Secrets -It is possible to supply a dummy password to Terraform and later change that password manually to a more secure one. For instance, if you were to create a Linode Instance with a dummy root password, you could later change that password from the command line or in the Linode Manager. Note, however, that any attempt to change the password in a `.tf` file will result in the creation of new resources on `terraform apply`. +Third-party tools exist that allow you to encrypt your secrets. If you encrypt the secrets in your `terraform.tfstate` (or your `.tfvars` files), you can check them into version control securely: -### Encrypting Secrets +- [git-crypt](https://github.com/AGWA/git-crypt) allows you to encrypt files when they are committed to a Git repository. git-crypt also decrypts files when they are checked out. -There are some third-party tools that allow you to encrypt your secrets. + {{< note >}} +You must initialize git-crypt in a repository before committing your state file or variable value files, or they will not be eligible for encryption. +{{< /note >}} -[git-crypt](https://github.com/AGWA/git-crypt) allows you to encrypt files when they are committed to a Git repository, and decrypt files when they are checked out. With `git-crypt` you could encrypt your `terraform.tfstate` file and securely commit it to version control. Note that for this option, you must initialize `git-crypt` in a repository before committing your state file or the state file will not be eligible for encryption. +- [Terrahelp](https://github.com/opencredo/terrahelp) allows you to encrypt and decrypt a whole state file, or just the variables you have include in your `terraform.tfvars` file. -Another option is [Terrahelp](https://github.com/opencredo/terrahelp). Terrahelp allows you to encrypt and decrypt a whole state file, or just the variables you have include in your `terraform.tfvars` file. +### Use a Dummy Password -### Privatize Version Control +It is possible to supply a dummy password to Terraform and later change that password manually to a more secure one. For instance, if you were to create a Linode instance with a dummy root password, you could later change that password from the command line or in the Linode Manager. -If you are unwilling or unable to use the above options to help manage your state file, and if you are using a platform like GitHub or GitLab to share your state files, then at the very least the repository should be private. +{{< note >}} +Any attempt to change the password in a `.tf` file will result in the creation of new resources on `terraform apply`. +{{< /note >}} +### Privatize Version Control +If you are unwilling or unable to use the above options to help manage your state file, and if you are using a platform like GitHub or GitLab to share your state files, then at minimum the repository should be private. From fa2ee99a547c3d15f1fb89d9aa1e82936229e361 Mon Sep 17 00:00:00 2001 From: nmelehan Date: Tue, 11 Dec 2018 18:50:35 +0000 Subject: [PATCH 4/4] Remove trailing whitespace --- .../secrets-management-with-terraform/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/applications/configuration-management/secrets-management-with-terraform/index.md b/docs/applications/configuration-management/secrets-management-with-terraform/index.md index 065ca374742..0e38bb7622c 100644 --- a/docs/applications/configuration-management/secrets-management-with-terraform/index.md +++ b/docs/applications/configuration-management/secrets-management-with-terraform/index.md @@ -125,7 +125,7 @@ This method is a bit easier to use than supplying environment variables, and has ## How to Manage Your State File -While it is relatively easy to keep secrets out of `.tf` files using any of the above methods, there is another file you need to be aware of when managing secrets, and that is the `terraform.tfstate` file. +While it is relatively easy to keep secrets out of `.tf` files using any of the above methods, there is another file you need to be aware of when managing secrets, and that is the `terraform.tfstate` file. This *state file* contains a JSON object that holds your managed infrastructure's current state. This state is a snapshot of the various attributes of your infrastructure at the time it was last modified. It is generated on `terraform apply` and is a necessary part of the Terraform process, as it maps the declarative code of your `.tf` files to your real world infrastructure.