standalone client-side remote state encryption

[StephanHCB]

Proposal

See this comment on 9556

Also see the discussions on #9556 for the case for full client-side state encryption.

Summary:

This PR implements client side encryption of the entire state for all remote state storage backends except the enhanced backends.

What gets sent to the remote state storage just looks like this:

{"crypted":"e93e3e7ad343405525...dda4fc061"}

The idea is that even the company that operates the remote state storage cannot read it. Of course, one should still configure all other protection mechanisms on the remote storage, this is just one layer of security, but I think it's a very important one.

Features:

• encryption algorithms are pluggable, I have implemented AES256 in CFB stream mode with a fresh random initialization vector generated each time and SHA256 hashing prior to encryption to check for payload integrity
• supports key rotation by giving an optional second key to try for decryption
• supports encryption of existing non-encrypted state
• supports decryption of existing encrypted state
• has unit tests that demonstrate how it works, as well as near complete code coverage
• includes documentation as an experimental feature

The change to existing code is minimal, see states/remote/states.go, I have basically inserted the encryption/decryption at the point at which Client.Get / Client.Put are invoked. As the result of the encryption is again json, this should hopefully work with all Clients with zero changes.

Successfully tested with the azure backend.

Limitations

I have marked this feature experimental in the documentation, I do not have the resources to test it with all remote backends.

I assume it will not work with enhanced backends because these need access to the state.

.

I invite, and very much welcome, feedback from tf maintainers as to anything I might have overlooked or should do differently.

[maludwig]

This looks fantastic, great code coverage, great documentation. The security of the system is extremely well thought out and well-designed. I have only minor cosmetic suggestions.

[maludwig]

Cosmetic suggestion: This function is fairly long, which is OK, but I would suggest splitting it into more functions to improve readability. Perhaps "extractCiphertext", "extractIV", "decryptPayload", and "verifyIntegrity".

[StephanHCB]

Thank you for the suggestion. I have played around with this a bit, and in the end I only extracted the encoding and decoding to/from json. Further extractions didn't really improve readability much.

[maludwig]

Cosmetic suggestion:
I would also encode the encryption algorithm into the payload, denoting that the payload was encrypted with AES256, and include a version number, for revisions to the algorithm. Future payloads may be encrypted with other ciphers, but the same key, or perhaps the same key derivation scheme, but a different, future symmetric cipher.
I would use base64 encoding for the binary data, just to increase the density, but continue to allow it to be easily encoded as JSON.

[StephanHCB]

I am feeling slightly uneasy retrieving the decryption algorithm from the (potentially manipulated) remote state. It's probably safe, but I'd rather not. If key / algorithm rotation is desired or necessary, this can be done using the configuration pulled in from the environment variables.

[maludwig]

Cosmetic suggestion, this is very deeply nested. I would consider extracting a function here for readability.

[StephanHCB]

Thanks! Done.

[maludwig]

Cosmetic, I would also capitalize "cfb", if you're capitalizing aes and sha.

[StephanHCB]

Done!

[maludwig]

I would also include, perhaps, a description of how to securely generate a key, since this doesn't use a key derivation algorithm. Perhaps a bash and powershell snippet, such as:

Bash

# Generate random bits with openssl
openssl rand -hex 32

# Generate random bits with /dev/urandom
tr -dc a-f0-9 </dev/urandom | head -c 64 ; echo ''

Powershell

$AESKey = New-Object Byte[] 32
[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($AESKey)
($AESKey|ForEach-Object ToString x2) -join ''

[StephanHCB]

Too easy to google imho.

[maludwig]

It would be nice, but I think, obviously, beyond the scope of this Pull Request, to create a CLI tool to help with these operations. Something like:

terraform state encrypt --implementation 'client-side/AES256-cfb/SHA256' --key 'a0a1a2a3a4a5a6a7a8a9b0b1b2b3b4b5b6b7b8b9c0c1c2c3c4c5c6c7c8c9d0d1'
> Encrypting state, implementation 'client-side/AES256-cfb/SHA256', key: a0a1a2a3a4a5a6a7a8a9b0b1b2b3b4b5b6b7b8b9c0c1c2c3c4c5c6c7c8c9d0d1
> Done

terraform state encrypt
> Encryption Implementations:
> 1. (Recommended) Client-side, AES256, CFB mode, SHA256 integrity check
> 2. Client-side, DES, ECB mode, no integrity check
> 3. Client-side, Pig Latin
>Which encryption implementation would you like to use (default: 1)?: 1
> Key Generation:
> 1. (Recommended) Generate a new completely random key
> 2. Generate a new key from a password PBKDF2
> 3. Use 4 (From random dice roll) https://xkcd.com/221/
> Which key generation algorithm would you like to use (default: 1)? 3
>
> Encrypting state, implementation 'client-side/AES256-cfb/SHA256', key: 0000000000000000000000000000000000000000000000000000000000000004
> Done

[StephanHCB]

If my PR gets accepted, I'm likely to build a simple encrypt / decrypt cli for downloaded state files/blobs. It is more or less required so state surgery is still possible.

[frittentheke]

@StephanHCB I honestly don't think yet another tool would be the best approach, but rather give terraform state pull (https://www.terraform.io/docs/cli/commands/state/pull.html) the ability to dump the encrypted remote state locally and unencrypted - in short extend the existing terraform state handling commands to deal with encrypted remote state.

[StephanHCB]

@StephanHCB I honestly don't think yet another tool would be the best approach, but rather give terraform state pull (https://www.terraform.io/docs/cli/commands/state/pull.html) the ability to dump the encrypted remote state locally and unencrypted - in short extend the existing terraform state handling commands to deal with encrypted remote state.

@frittentheke those commands already work

It's just that here are a number of edge case limitations, most of which also affect terraform state pull with non-encrypted state, only with encryption you'll be completely locked out instead of being able to just externally download your state and manually merge changes with a previous version to recover.

Or consider bulk operations on a large number of state blobs, which naturally occur if you use terragrunt. It's really convenient to be able to just download all storage blobs from an Azure Storage container as a directory structure, then process them locally on the command line.

[scaswell-hirez]

Y'all (Hashicorp) have been sitting on this since May of 2021 and it solves an open issue that is 7 years old! It solves the basic use case and the community has been asking for this. Please expedite this for inclusion in the next release.

[hashicorp-cla]

All committers have signed the CLA.

[StephanHCB]

[...] All committers have signed the CLA.

Huh, I signed that when I first made the original draft PR.

In any case, I am absolutely thrilled to see progress on this PR.

[binlab]

@StephanHCB fantastic work, many thanks! @StephanHCB @maludwig @crw do you have any idea when it might be merged into mainstream? Thanks!

[Frankkkkk]

Hi @apparentlymart @jbardin any chance that you could comment on this MR please ? :-)

[dhancocksqbe]

@StephanHCB fantastic work, many thanks! @StephanHCB @maludwig @crw do you have any idea when it might be merged into mainstream? Thanks!

Any view on when this might make live?

[maludwig]

I'm sadly just, like, a normal human and not super related to terraform. I simply reviewed the excellent code and gave it a thumbs up with some very minor suggestions. It's not up to me.

[siepkes]

I can understand Hashicorp being busy with other things and not wanting to maintain this if it doesn't align with their roadmap. It's a security sensitive thing so reviewing it, maintaining it, etc. is a lot of work.

A way to make it easier to review and maintain is to off-load the crypto heavy-lifting to Google's Tink library. Which describes itself as a "open source library that provides cryptographic APIs that are secure, easy to use correctly, and hard(er) to misuse.". Because there is a lot that one can botch when using low-level crypto primitves to do encryption.

Additionally perhaps it's an idea to add a flag to Terraform like --enable-unsupported-features (and print a "no warranty" warning when it is used) and put this feature behind it?

[Frankkkkk]

Furthermore, if backends were a pluggable option (c.f. #5877) we could still maintain encryption as a non official tf plugin.

[archmangler]

Has there been any progress on this? Basically CMK encryption for terraform backend state is mandated for enterprise security compliance in many enterprises these days. We have one such case.

If there is any workaround that would allow terraform to decrypt state files stored in a CMK-encrypted azure storage account that would be useful as well!

Currently I'm getting this:

+ terraform fmt
+ terraform init
Error loading state: blobs.Client#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="KeyVaultEncryptionKeyNotFound" Message="The key vault key is not found to unwrap the encryption key.\nRequestId:6ed9e67a-401e-0051-7a28-320ead000000\nTime:2023-01-27T08:19:48.7696249Z"

With this backend config:

  backend "azurerm" {
    resource_group_name  = "lol-caf-ea-nprd-rg-001"
    storage_account_name = "lolcafeanprdst001"
    container_name       = "lol-caf-ea-nprd-stct-001"
    key                  = "lol-azure-caf-dnsrecords-subnets.tfstate"
  }

[archmangler]

Some light at the end of the tunnel: OpenTF promises to have this feature as a default.

[caretak3r]

Are there any practices using this that would help a user or an organization avoid losing access to the backend state completely? Seems like if you lose the key to encrypt state, you also lose ability to manage anything using tf moving forward? And aren't there already practices to redact sensitive data from state?

If anything, tightly restricting access to the state (i.e in S3) and logging accesses should suffice no?

[siepkes]

Are there any practices using this that would help a user or an organization avoid losing access to the backend state completely? Seems like if you lose the key to encrypt state, you also lose ability to manage anything using tf moving forward? And aren't there already practices to redact sensitive data from state?

I think that's out of scope for Terraform and is something which requires similar precautions for all apps which use encrypted data. For example using WAL-G with PostgreSQL and uploading your backups to S3 has a similar problem.

Having said that there are good solutions for mitigating this issue; For example you could print out the key on a piece of paper (as base64 and as a QR code), laminate the piece of paper and put it in a safe somewhere off site.

USB sticks are typically not well suited for long term data storage since they store data by using a electric charge. This charge will fade over time and you might loose your data. An M-Disc is well suited for long term storage but is more hassle then using a piece of laminated paper.

[StephanHCB]

It's now been 2.5 years. I'm closing this stale PR.

[dominic-p]

Is client side encryption explicitly a "won't fix" then? Or, is it just this specific implementation? It seems like it would be a generally useful addition to Terraform from my perspective.

[crw]

@dominic-p as the PR was closed by the original contributor (understandably), that is not a HashiCorp comment on whether or not the concept of client-side encryption is a "wont fix." These issues and PRs are frequently re-examined and discussed, though I have no new information to relay at this time.

[dominic-p]

Thanks for the clarification. I completely missed who closed the PR. :)

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.