Storing sensitive values in state files

[seanherron]

#309 was the first change in Terraform that I could find that moved to store sensitive values in state files, in this case the password value for Amazon RDS. This was a bit of a surprise for me, as previously I've been sharing our state files publicly. I can't do that now, and feel pretty nervous about the idea of storing state files in version control at all (and definitely can't put them on github or anything).

If Terraform is going to store secrets, then some sort of field-level encryption should be built in as well. In the meantime, I'm going to change things around to use https://github.com/AGWA/git-crypt on sensitive files in my repos.

[bitglue]

See #874. I changed the RDS provider to store an SHA1 hash of the password.

That said, I'm not sure I'd agree that it's Terraform's responsibility to protect data in the state file. Things other than passwords can be sensitive: for example if I had a security group restricting SSH access to a particular set of hosts, I wouldn't want the world to know which IP they need to spoof to gain access. The state file can be protected orthogonally: you can not put it on github, you can put it in a private repo, you can use git-crypt, etc.

[kubek2k]

related #689

[dentarg]

Just want to give my opinion on this topic.

I do think Terraform should address this issue. I think it will increase the usefulness and ease of use of Terraform.

Some examples from other projects: Ansible has vaults, and on Travis CI you can encrypt informaton in the .travis.yml file.

[ketzacoatl]

Ansible vaults is a feature I often want in other devops tools. Protecting these details is not as easy as protecting the state file.. what about using consul or Atlas as a remote/backend store?

+1 on this

[dayer4b]

I just want to point out that, according to official documentation, storing the state file in version control is a best practice:

https://www.terraform.io/intro/getting-started/build.html

Terraform also put some state into the terraform.tfstate file by default. This state file is extremely
important; it maps various resource metadata to actual resource IDs so that Terraform knows what
it is managing. This file must be saved and distributed to anyone who might run Terraform. We
recommend simply putting it into version control, since it generally isn't too large.

(emphasis added)

Which means we really shouldn't have to worry about secrets popping up in there...

[hobbeswalsh]

👍 on this idea -- it would be enough for our case to allow configuration of server-side encryption for S3 buckets. Any thoughts on implementing that?

[apparentlymart]

At the risk of adding scope to this discussion, I think another way to think of this is that Terraform's current architecture is based on a faulty assumption: Terraform assumes that all provider configuration is sensitive and that all resource configuration isn't sensitive. That is wrong in both directions:

• Several resources now take passwords as inputs or produce secret values as outputs. In this issue we see the RDS password as one example. The potential Vault provider discussed in Vault provider #2221 is another example.
• Several provider arguments are explicitly not sensitive, such as the AWS region name, and excluding them from the Terraform state results in Terraform having an incomplete picture of the world: it can see that there is an EC2 instance with the id i-12345 but it can't see what region that instance is in without help of the configuration. Changing the region on the AWS provider causes Terraform to lose track of all of the existing resources, because as far as the AWS provider is concerned they've all been apparently deleted.

So all of this is to say that I think overloading the provider/resource separation as a secret/non-secret separation is not the best design. Instead, it'd be nice to have a mechanism on both sides to distinguish between things that should live in the state and things that should not, so that e.g. generated secrets can be passed into provisioners but not retained in the state, and that the state can encode that a particular instance belongs to a particular AWS region and respond in a better way when the region changes.

There are of course a number of tricky cases in making this situation, which I'd love to explore some more. Here are some to start:

• If you don't retain something in the state then it's not safe to interpolate it anywhere because future runs will assume they can interpolate attributes from existing resources in the state.
• Some provider config changes effectively switch all resources to an entirely new "namespace", and thus effectively force every attached resource to be destroyed and recreated in the new region. The AWS region is one example, since AWS resources are region-specific. But that's not so simple for other arguments: the AWS access_key might change what Terraform has permission to interact with, but it doesn't change the id namespace that resources live within.