New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storing sensitive values in state files #516

Open
seanherron opened this Issue Oct 28, 2014 · 91 comments

Comments

Projects
None yet
@seanherron
Contributor

seanherron commented Oct 28, 2014

#309 was the first change in Terraform that I could find that moved to store sensitive values in state files, in this case the password value for Amazon RDS. This was a bit of a surprise for me, as previously I've been sharing our state files publicly. I can't do that now, and feel pretty nervous about the idea of storing state files in version control at all (and definitely can't put them on github or anything).

If Terraform is going to store secrets, then some sort of field-level encryption should be built in as well. In the meantime, I'm going to change things around to use https://github.com/AGWA/git-crypt on sensitive files in my repos.

@bitglue

This comment has been minimized.

Show comment
Hide comment
@bitglue

bitglue Jan 28, 2015

See #874. I changed the RDS provider to store an SHA1 hash of the password.

That said, I'm not sure I'd agree that it's Terraform's responsibility to protect data in the state file. Things other than passwords can be sensitive: for example if I had a security group restricting SSH access to a particular set of hosts, I wouldn't want the world to know which IP they need to spoof to gain access. The state file can be protected orthogonally: you can not put it on github, you can put it in a private repo, you can use git-crypt, etc.

bitglue commented Jan 28, 2015

See #874. I changed the RDS provider to store an SHA1 hash of the password.

That said, I'm not sure I'd agree that it's Terraform's responsibility to protect data in the state file. Things other than passwords can be sensitive: for example if I had a security group restricting SSH access to a particular set of hosts, I wouldn't want the world to know which IP they need to spoof to gain access. The state file can be protected orthogonally: you can not put it on github, you can put it in a private repo, you can use git-crypt, etc.

@kubek2k

This comment has been minimized.

Show comment
Hide comment
@kubek2k

kubek2k Jan 28, 2015

Contributor

related #689

Contributor

kubek2k commented Jan 28, 2015

related #689

@dentarg

This comment has been minimized.

Show comment
Hide comment
@dentarg

dentarg Mar 17, 2015

Just want to give my opinion on this topic.

I do think Terraform should address this issue. I think it will increase the usefulness and ease of use of Terraform.

Some examples from other projects: Ansible has vaults, and on Travis CI you can encrypt informaton in the .travis.yml file.

dentarg commented Mar 17, 2015

Just want to give my opinion on this topic.

I do think Terraform should address this issue. I think it will increase the usefulness and ease of use of Terraform.

Some examples from other projects: Ansible has vaults, and on Travis CI you can encrypt informaton in the .travis.yml file.

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Mar 28, 2015

Contributor

Ansible vaults is a feature I often want in other devops tools. Protecting these details is not as easy as protecting the state file.. what about using consul or Atlas as a remote/backend store?

+1 on this

Contributor

ketzacoatl commented Mar 28, 2015

Ansible vaults is a feature I often want in other devops tools. Protecting these details is not as easy as protecting the state file.. what about using consul or Atlas as a remote/backend store?

+1 on this

@dayer4b

This comment has been minimized.

Show comment
Hide comment
@dayer4b

dayer4b May 28, 2015

Contributor

I just want to point out that, according to official documentation, storing the state file in version control is a best practice:

https://www.terraform.io/intro/getting-started/build.html

Terraform also put some state into the terraform.tfstate file by default. This state file is extremely
important; it maps various resource metadata to actual resource IDs so that Terraform knows what
it is managing. This file must be saved and distributed to anyone who might run Terraform. We
recommend simply putting it into version control
, since it generally isn't too large.

(emphasis added)

Which means we really shouldn't have to worry about secrets popping up in there...

Contributor

dayer4b commented May 28, 2015

I just want to point out that, according to official documentation, storing the state file in version control is a best practice:

https://www.terraform.io/intro/getting-started/build.html

Terraform also put some state into the terraform.tfstate file by default. This state file is extremely
important; it maps various resource metadata to actual resource IDs so that Terraform knows what
it is managing. This file must be saved and distributed to anyone who might run Terraform. We
recommend simply putting it into version control
, since it generally isn't too large.

(emphasis added)

Which means we really shouldn't have to worry about secrets popping up in there...

@hobbeswalsh

This comment has been minimized.

Show comment
Hide comment
@hobbeswalsh

hobbeswalsh Jun 18, 2015

👍 on this idea -- it would be enough for our case to allow configuration of server-side encryption for S3 buckets. Any thoughts on implementing that?

hobbeswalsh commented Jun 18, 2015

👍 on this idea -- it would be enough for our case to allow configuration of server-side encryption for S3 buckets. Any thoughts on implementing that?

@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Sep 11, 2015

Contributor

At the risk of adding scope to this discussion, I think another way to think of this is that Terraform's current architecture is based on a faulty assumption: Terraform assumes that all provider configuration is sensitive and that all resource configuration isn't sensitive. That is wrong in both directions:

  • Several resources now take passwords as inputs or produce secret values as outputs. In this issue we see the RDS password as one example. The potential Vault provider discussed in #2221 is another example.
  • Several provider arguments are explicitly not sensitive, such as the AWS region name, and excluding them from the Terraform state results in Terraform having an incomplete picture of the world: it can see that there is an EC2 instance with the id i-12345 but it can't see what region that instance is in without help of the configuration. Changing the region on the AWS provider causes Terraform to lose track of all of the existing resources, because as far as the AWS provider is concerned they've all been apparently deleted.

So all of this is to say that I think overloading the provider/resource separation as a secret/non-secret separation is not the best design. Instead, it'd be nice to have a mechanism on both sides to distinguish between things that should live in the state and things that should not, so that e.g. generated secrets can be passed into provisioners but not retained in the state, and that the state can encode that a particular instance belongs to a particular AWS region and respond in a better way when the region changes.

There are of course a number of tricky cases in making this situation, which I'd love to explore some more. Here are some to start:

  • If you don't retain something in the state then it's not safe to interpolate it anywhere because future runs will assume they can interpolate attributes from existing resources in the state.
  • Some provider config changes effectively switch all resources to an entirely new "namespace", and thus effectively force every attached resource to be destroyed and recreated in the new region. The AWS region is one example, since AWS resources are region-specific. But that's not so simple for other arguments: the AWS access_key might change what Terraform has permission to interact with, but it doesn't change the id namespace that resources live within.
Contributor

apparentlymart commented Sep 11, 2015

At the risk of adding scope to this discussion, I think another way to think of this is that Terraform's current architecture is based on a faulty assumption: Terraform assumes that all provider configuration is sensitive and that all resource configuration isn't sensitive. That is wrong in both directions:

  • Several resources now take passwords as inputs or produce secret values as outputs. In this issue we see the RDS password as one example. The potential Vault provider discussed in #2221 is another example.
  • Several provider arguments are explicitly not sensitive, such as the AWS region name, and excluding them from the Terraform state results in Terraform having an incomplete picture of the world: it can see that there is an EC2 instance with the id i-12345 but it can't see what region that instance is in without help of the configuration. Changing the region on the AWS provider causes Terraform to lose track of all of the existing resources, because as far as the AWS provider is concerned they've all been apparently deleted.

So all of this is to say that I think overloading the provider/resource separation as a secret/non-secret separation is not the best design. Instead, it'd be nice to have a mechanism on both sides to distinguish between things that should live in the state and things that should not, so that e.g. generated secrets can be passed into provisioners but not retained in the state, and that the state can encode that a particular instance belongs to a particular AWS region and respond in a better way when the region changes.

There are of course a number of tricky cases in making this situation, which I'd love to explore some more. Here are some to start:

  • If you don't retain something in the state then it's not safe to interpolate it anywhere because future runs will assume they can interpolate attributes from existing resources in the state.
  • Some provider config changes effectively switch all resources to an entirely new "namespace", and thus effectively force every attached resource to be destroyed and recreated in the new region. The AWS region is one example, since AWS resources are region-specific. But that's not so simple for other arguments: the AWS access_key might change what Terraform has permission to interact with, but it doesn't change the id namespace that resources live within.
@little-arhat

This comment has been minimized.

Show comment
Hide comment
@little-arhat

little-arhat Sep 18, 2015

Hi, any progress on that? Terraform 0.6.3 still stores raw passwords in the state file. Also, as a related issue, if you do not want to keep passwords in configuration, you can create variable without default value. But, this will force you to pass this variable every time you run plan/apply, even if you're not going to change resource that has this password.

I think, it would be nice to separate sensitive stuff from other attributes, so it will:

  • be stored as sha1 or smth in state file
  • not require value if it already has one.

So, for configuration like:

variable db {
    password {}
}

resource ... {
    password = "${var.db.password}"
}

terraform will require variable for the first run, when it doesn't have anything, but will not require on subsequent runs.

To change such value one need to provide different value for password.

little-arhat commented Sep 18, 2015

Hi, any progress on that? Terraform 0.6.3 still stores raw passwords in the state file. Also, as a related issue, if you do not want to keep passwords in configuration, you can create variable without default value. But, this will force you to pass this variable every time you run plan/apply, even if you're not going to change resource that has this password.

I think, it would be nice to separate sensitive stuff from other attributes, so it will:

  • be stored as sha1 or smth in state file
  • not require value if it already has one.

So, for configuration like:

variable db {
    password {}
}

resource ... {
    password = "${var.db.password}"
}

terraform will require variable for the first run, when it doesn't have anything, but will not require on subsequent runs.

To change such value one need to provide different value for password.

@EvanKrall

This comment has been minimized.

Show comment
Hide comment
@EvanKrall

EvanKrall Oct 16, 2015

Contributor

Maybe there's a simple solution: store the state in Vault?

Contributor

EvanKrall commented Oct 16, 2015

Maybe there's a simple solution: store the state in Vault?

@mwarkentin

This comment has been minimized.

Show comment
Hide comment
@mwarkentin

mwarkentin Oct 19, 2015

Contributor

A good solution for this would be useful for us as well - we're manually configuring certain things to keep them out of the tfstate file in the meantime.

Contributor

mwarkentin commented Oct 19, 2015

A good solution for this would be useful for us as well - we're manually configuring certain things to keep them out of the tfstate file in the meantime.

@ascendantlogic

This comment has been minimized.

Show comment
Hide comment
@ascendantlogic

ascendantlogic Oct 30, 2015

So as I slowly cobble together another clean-sheet infra with Terraform I see this problem still exists, and this issue is almost exactly 1 year old. What is the thinking in regards to solving this? the ability to mark specific attributes within a resource as sensitive and storing SHA1 or SHA2 hashes of their values in the state for comparison? I see this comment on a related ticket, does that mean that using Vault will be the prescribed way? I get that it promotes product synergy but I'd really like a quick-n-dirty hashing solution as a fallback option if I'm honest.

ascendantlogic commented Oct 30, 2015

So as I slowly cobble together another clean-sheet infra with Terraform I see this problem still exists, and this issue is almost exactly 1 year old. What is the thinking in regards to solving this? the ability to mark specific attributes within a resource as sensitive and storing SHA1 or SHA2 hashes of their values in the state for comparison? I see this comment on a related ticket, does that mean that using Vault will be the prescribed way? I get that it promotes product synergy but I'd really like a quick-n-dirty hashing solution as a fallback option if I'm honest.

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Oct 30, 2015

Contributor

Moving secrets to vault, and using consul-template or integration with other custom solutions you have for CM certainly helps for a lot of cases, but completely avoiding secrets in TF or ending up in TF state is not always reasonable.

Contributor

ketzacoatl commented Oct 30, 2015

Moving secrets to vault, and using consul-template or integration with other custom solutions you have for CM certainly helps for a lot of cases, but completely avoiding secrets in TF or ending up in TF state is not always reasonable.

@ascendantlogic

This comment has been minimized.

Show comment
Hide comment
@ascendantlogic

ascendantlogic Oct 30, 2015

Sure, in this particular case I don't want to manually manage RDS but I don't want the PW in the state in cleartext, regardless of where I keep it. I'm sure this is a somewhat common issue. Maybe an overarching ability to mark arbitrary attributes as sensitive is shooting for the moon but a good start would be anything that is obviously sensitive, such as passwords.

ascendantlogic commented Oct 30, 2015

Sure, in this particular case I don't want to manually manage RDS but I don't want the PW in the state in cleartext, regardless of where I keep it. I'm sure this is a somewhat common issue. Maybe an overarching ability to mark arbitrary attributes as sensitive is shooting for the moon but a good start would be anything that is obviously sensitive, such as passwords.

@jfuechsl

This comment has been minimized.

Show comment
Hide comment
@jfuechsl

jfuechsl Nov 26, 2015

Would it be feasible to open up state handling to plugins?
The standard could be to store it in files, like it is currently done.
Other options could be Vault, S3, Atlas, etc.

That way this issue can be dealt with appropriately based on the use-case.

jfuechsl commented Nov 26, 2015

Would it be feasible to open up state handling to plugins?
The standard could be to store it in files, like it is currently done.
Other options could be Vault, S3, Atlas, etc.

That way this issue can be dealt with appropriately based on the use-case.

@brikis98

This comment has been minimized.

Show comment
Hide comment
@brikis98

brikis98 Dec 12, 2015

Contributor

I just got tripped up by this as well, as the docs explicitly tell you to store .tfstate files in version control, which is problematic if passwords and other secrets end up in the .tfstate files. At the bare minimum, the docs should be updated with a massive warning about this. Beyond that, there seem to be a few options:

  1. Offer some way to mark variables as secret and either ensure they never get stored in .tfstate files or store them in a hashed form.
  2. Encrypt the entire .tfstate file.
  3. Remove the recommendation to store .tfstate files in version control and only recommend them to be stored in secure, preferably encrypted storage.
Contributor

brikis98 commented Dec 12, 2015

I just got tripped up by this as well, as the docs explicitly tell you to store .tfstate files in version control, which is problematic if passwords and other secrets end up in the .tfstate files. At the bare minimum, the docs should be updated with a massive warning about this. Beyond that, there seem to be a few options:

  1. Offer some way to mark variables as secret and either ensure they never get stored in .tfstate files or store them in a hashed form.
  2. Encrypt the entire .tfstate file.
  3. Remove the recommendation to store .tfstate files in version control and only recommend them to be stored in secure, preferably encrypted storage.
@ejoubaud

This comment has been minimized.

Show comment
Hide comment
@ejoubaud

ejoubaud Jan 5, 2016

One thing to consider around this is output. When you create a resource with secrets (key pair, access keys, db password, etc.), you likely want to show the secret in question at least once (possibly in the stdout of the first run, as output do)

Currently output are also stored in plain text in the .tfstate, and can be retrieved later with terraform output.

One possible solution would be a mechanism to only show the secrets once, then not store them at all and not show them again (like AWS does), possibly using only-once output as I suggested in #4437

ejoubaud commented Jan 5, 2016

One thing to consider around this is output. When you create a resource with secrets (key pair, access keys, db password, etc.), you likely want to show the secret in question at least once (possibly in the stdout of the first run, as output do)

Currently output are also stored in plain text in the .tfstate, and can be retrieved later with terraform output.

One possible solution would be a mechanism to only show the secrets once, then not store them at all and not show them again (like AWS does), possibly using only-once output as I suggested in #4437

@revett

This comment has been minimized.

Show comment
Hide comment
@revett

revett commented Jan 14, 2016

+1

@sstarcher

This comment has been minimized.

Show comment
Hide comment
@sstarcher

sstarcher commented Jan 19, 2016

+1

@ascendantlogic

This comment has been minimized.

Show comment
Hide comment
@ascendantlogic

ascendantlogic Jan 19, 2016

To get around this for now in my production RDS I just created the instance with a password of changeme1234 and then went to the console and manually changed the PW.

ascendantlogic commented Jan 19, 2016

To get around this for now in my production RDS I just created the instance with a password of changeme1234 and then went to the console and manually changed the PW.

@gtmtech

This comment has been minimized.

Show comment
Hide comment
@gtmtech

gtmtech Jan 22, 2016

+1

I notice also that Redshift is imminently going to be supported on terraform, and the same mistakes are being made all over again:

master_password - (Required) Password for the master DB user. Note that this may show up in logs, and it will be stored in the state file

"Applications should not transmit or store passwords in unencrypted form"

Page 77 - ISO27001 :

https://books.google.co.uk/books?id=Ur1lviHCd-4C&pg=PA77&dq=no+password+unencrypted+disk+iso27001&hl=en&sa=X&ved=0ahUKEwibuMrywb3KAhUCVxQKHR7yCNwQ6AEIPTAB#v=onepage&q=no%20password%20unencrypted%20disk%20iso27001&f=false

Can this please not be done - it instantly means for lots of us (for compliance reasons) we cant use it - the whole "no password should be written down on any disk unencrypted" thing is a killer.

This has been in progress for over a year - is there any attempt to solve this? I would have thought the simplest approach would be to hash the password, store the hash.

gtmtech commented Jan 22, 2016

+1

I notice also that Redshift is imminently going to be supported on terraform, and the same mistakes are being made all over again:

master_password - (Required) Password for the master DB user. Note that this may show up in logs, and it will be stored in the state file

"Applications should not transmit or store passwords in unencrypted form"

Page 77 - ISO27001 :

https://books.google.co.uk/books?id=Ur1lviHCd-4C&pg=PA77&dq=no+password+unencrypted+disk+iso27001&hl=en&sa=X&ved=0ahUKEwibuMrywb3KAhUCVxQKHR7yCNwQ6AEIPTAB#v=onepage&q=no%20password%20unencrypted%20disk%20iso27001&f=false

Can this please not be done - it instantly means for lots of us (for compliance reasons) we cant use it - the whole "no password should be written down on any disk unencrypted" thing is a killer.

This has been in progress for over a year - is there any attempt to solve this? I would have thought the simplest approach would be to hash the password, store the hash.

@johnrengelman

This comment has been minimized.

Show comment
Hide comment
@johnrengelman

johnrengelman Jan 22, 2016

Contributor

@gtmtech a hash isn't cryptographically safe either because it can be reversed.
The right solution here is something that can store the value securely, doing anything else IMO would be a waste of energy.

Contributor

johnrengelman commented Jan 22, 2016

@gtmtech a hash isn't cryptographically safe either because it can be reversed.
The right solution here is something that can store the value securely, doing anything else IMO would be a waste of energy.

@ketzacoatl

This comment has been minimized.

Show comment
Hide comment
@ketzacoatl

ketzacoatl Jan 22, 2016

Contributor

@gtmtech, if this is a blocker, can you put in a goof password on first run, and then manually update it, as @ascendantlogic notes above? While not "clean", and while it "gives you something to do", that seems like a reasonable middle ground, no?

Contributor

ketzacoatl commented Jan 22, 2016

@gtmtech, if this is a blocker, can you put in a goof password on first run, and then manually update it, as @ascendantlogic notes above? While not "clean", and while it "gives you something to do", that seems like a reasonable middle ground, no?

@ascendantlogic

This comment has been minimized.

Show comment
Hide comment
@ascendantlogic

ascendantlogic Jan 22, 2016

@johnrengelman forgive me if I am misunderstanding, but I thought hashes were, by definition, one way. Or at least any reasonable use of one to add some level of protection to secrets necessitates the use of a one-way?

ascendantlogic commented Jan 22, 2016

@johnrengelman forgive me if I am misunderstanding, but I thought hashes were, by definition, one way. Or at least any reasonable use of one to add some level of protection to secrets necessitates the use of a one-way?

@gtmtech

This comment has been minimized.

Show comment
Hide comment
@gtmtech

gtmtech Jan 22, 2016

So problems with the comments above are:

  • A goof password will mean every time we terraform plan it will advise us it wants to rewire the password - not a great experience when we're always after clean terraform states, but you're right its probably the only workaround we have right now so thanks for the suggestion...
  • A hash is one-way, and yes if you use SHA1, or even SHA256 you deserve everything you get, but there are other hash functions that are pretty secure - they are one way, and in ANY case, it would fulfil compliance and allow people to use the resource.

Maybe there's a way of not storing the password at all, and not caring about it? When I worked on configuration management tools before, I did this approach sometimes - it was never stored in state, and the value was always set to be ignored, so the diff algorithm just completely ignored it, but it was used on first create only. This isnt ideal either as you cant manage passwords with TF, but its better than unencrypted, readable master database passwords everywhere.

Its a real shame to not be able to use terraform for something as dumb as plaintext readable master passwords, its such a great tool otherwise!

gtmtech commented Jan 22, 2016

So problems with the comments above are:

  • A goof password will mean every time we terraform plan it will advise us it wants to rewire the password - not a great experience when we're always after clean terraform states, but you're right its probably the only workaround we have right now so thanks for the suggestion...
  • A hash is one-way, and yes if you use SHA1, or even SHA256 you deserve everything you get, but there are other hash functions that are pretty secure - they are one way, and in ANY case, it would fulfil compliance and allow people to use the resource.

Maybe there's a way of not storing the password at all, and not caring about it? When I worked on configuration management tools before, I did this approach sometimes - it was never stored in state, and the value was always set to be ignored, so the diff algorithm just completely ignored it, but it was used on first create only. This isnt ideal either as you cant manage passwords with TF, but its better than unencrypted, readable master database passwords everywhere.

Its a real shame to not be able to use terraform for something as dumb as plaintext readable master passwords, its such a great tool otherwise!

@johnrengelman

This comment has been minimized.

Show comment
Hide comment
@johnrengelman

johnrengelman Jan 22, 2016

Contributor

@ascendantlogic ah yeah, sorry, don't know where my brain was this morning. I'm blaming that fact that I hadn't had coffee yet.

Contributor

johnrengelman commented Jan 22, 2016

@ascendantlogic ah yeah, sorry, don't know where my brain was this morning. I'm blaming that fact that I hadn't had coffee yet.

@gtmtech

This comment has been minimized.

Show comment
Hide comment
@gtmtech

gtmtech Jan 22, 2016

I would recommend bcrypt or even scrypt for the password hashing.

In the meantime, a goof password with ignore_changes=["master_password"] will suffice so long as I can get terraform to NOT store it in the state file.

A meta_parameter to accomplish not storing a particular atribute would be an alternative to having to do the hashing work - either one could accomplish the end goal

gtmtech commented Jan 22, 2016

I would recommend bcrypt or even scrypt for the password hashing.

In the meantime, a goof password with ignore_changes=["master_password"] will suffice so long as I can get terraform to NOT store it in the state file.

A meta_parameter to accomplish not storing a particular atribute would be an alternative to having to do the hashing work - either one could accomplish the end goal

@sstarcher

This comment has been minimized.

Show comment
Hide comment
@sstarcher

sstarcher Jan 22, 2016

@gtmtech A goof password should not cause terraform states to complain every time. As long as the password in the tf files and the password in the state file are the same terraform should be happy.

I created a RDS database using a password in the TF files and after creation went into the TF file and changed it to XXXXX. I also went into the state file and change it to XXXXX. terraform plan and terraform apply are happy.

sstarcher commented Jan 22, 2016

@gtmtech A goof password should not cause terraform states to complain every time. As long as the password in the tf files and the password in the state file are the same terraform should be happy.

I created a RDS database using a password in the TF files and after creation went into the TF file and changed it to XXXXX. I also went into the state file and change it to XXXXX. terraform plan and terraform apply are happy.

@gtmtech

This comment has been minimized.

Show comment
Hide comment
@gtmtech

gtmtech Jan 22, 2016

@sstarcher interesting, what happens when you terraform refresh, does it not override your XXXXXing out of the password?

gtmtech commented Jan 22, 2016

@sstarcher interesting, what happens when you terraform refresh, does it not override your XXXXXing out of the password?

@mingfang

This comment has been minimized.

Show comment
Hide comment
@mingfang

mingfang Dec 31, 2016

One option is to implement something like Ansible Vault. terraform.tfstate will be encrypted and can be in git. A password is required each time the file is accessed.

mingfang commented Dec 31, 2016

One option is to implement something like Ansible Vault. terraform.tfstate will be encrypted and can be in git. A password is required each time the file is accessed.

@hardboiled

This comment has been minimized.

Show comment
Hide comment
@hardboiled

hardboiled Jan 30, 2017

Is it possible to do something similar to remote state, except for key/values? Basically, I'd like to push a configuration file to an s3 bucket that supports only encrypted files. I don't care about the format as long it supports key/value pairs. Then, in .tf files it might look something like this:

data "remote_config" "credentials" {
  type = "map" /* this could be implicit, meaning flat */
                        /* key/value maps could be the only data-type supported */
  backend = "s3"
  config {
        bucket = "tf-my-bucket"
        key = "credentials.json"
        region = "us-east-2"
    }
}

resource "aws_db_instance" "rds_example" {
   ...
   rds details etc ...
   ...
   username = "#{data.remote_config.credentials["rds_example_username"]}"
   password = "#{data.remote_config.credentials["rds_example_password"]}"
   ...
}

Note as an alternative someone else was loading passwords from a local file. I'd rather it be on s3 though ;).

hardboiled commented Jan 30, 2017

Is it possible to do something similar to remote state, except for key/values? Basically, I'd like to push a configuration file to an s3 bucket that supports only encrypted files. I don't care about the format as long it supports key/value pairs. Then, in .tf files it might look something like this:

data "remote_config" "credentials" {
  type = "map" /* this could be implicit, meaning flat */
                        /* key/value maps could be the only data-type supported */
  backend = "s3"
  config {
        bucket = "tf-my-bucket"
        key = "credentials.json"
        region = "us-east-2"
    }
}

resource "aws_db_instance" "rds_example" {
   ...
   rds details etc ...
   ...
   username = "#{data.remote_config.credentials["rds_example_username"]}"
   password = "#{data.remote_config.credentials["rds_example_password"]}"
   ...
}

Note as an alternative someone else was loading passwords from a local file. I'd rather it be on s3 though ;).

@Fodoj

This comment has been minimized.

Show comment
Hide comment
@Fodoj

Fodoj Jan 30, 2017

Contributor

@hardboiled It should be easy to implement with external data source https://www.terraform.io/docs/providers/external/data_source.html It is focused precisely on key-value objects.

Contributor

Fodoj commented Jan 30, 2017

@hardboiled It should be easy to implement with external data source https://www.terraform.io/docs/providers/external/data_source.html It is focused precisely on key-value objects.

@hardboiled

This comment has been minimized.

Show comment
Hide comment
@hardboiled

hardboiled Jan 30, 2017

@Fodoj Actually, while I don't know if this will be my final solution, I instead opted to handcraft a tfstate file and configure it as a remote. Seems to work exactly like people here would want, meaning supporting version control and encryption (since s3 supports those things), with the added bonus of supporting all terraform types, since I'm just manually typing a state file in with outputs :).

state file

{
    "version": 3,
    "terraform_version": "0.8.4",
    "serial": 1,
    "lineage": "",
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "tf-your-s3-bucket",
            "encrypt": "true",
            "key": "credentials/production/terraform.tfstate",
            "region": "us-east-2"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {
                "production_rds_main": {
                    "sensitive": false,
                    "type": "map",
                    "value": {
                        "username":"your_org_admin",
                        "password":"your_sensitive_password"
                    }
                }
            },
            "resources": {},
            "depends_on": []
        }
    ]
}

remote configuration in my repo

data "terraform_remote_state" "credentials" {
    backend = "s3"
    config {
        bucket = "tf-your-s3-bucket",
        key = "credentials/production/terraform.tfstate",
        region = "us-east-2"
    }
}

referencing the remote in terraform

resource "aws_db_instance" "production_rds_main" {
   ...
   rds details etc ...
   ...
   username = "${data.terraform_remote_state.credentials.production_rds_main.username}"
   password = "${data.terraform_remote_state.credentials.production_rds_main.password}"
   ...
}

hardboiled commented Jan 30, 2017

@Fodoj Actually, while I don't know if this will be my final solution, I instead opted to handcraft a tfstate file and configure it as a remote. Seems to work exactly like people here would want, meaning supporting version control and encryption (since s3 supports those things), with the added bonus of supporting all terraform types, since I'm just manually typing a state file in with outputs :).

state file

{
    "version": 3,
    "terraform_version": "0.8.4",
    "serial": 1,
    "lineage": "",
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "tf-your-s3-bucket",
            "encrypt": "true",
            "key": "credentials/production/terraform.tfstate",
            "region": "us-east-2"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {
                "production_rds_main": {
                    "sensitive": false,
                    "type": "map",
                    "value": {
                        "username":"your_org_admin",
                        "password":"your_sensitive_password"
                    }
                }
            },
            "resources": {},
            "depends_on": []
        }
    ]
}

remote configuration in my repo

data "terraform_remote_state" "credentials" {
    backend = "s3"
    config {
        bucket = "tf-your-s3-bucket",
        key = "credentials/production/terraform.tfstate",
        region = "us-east-2"
    }
}

referencing the remote in terraform

resource "aws_db_instance" "production_rds_main" {
   ...
   rds details etc ...
   ...
   username = "${data.terraform_remote_state.credentials.production_rds_main.username}"
   password = "${data.terraform_remote_state.credentials.production_rds_main.password}"
   ...
}
@Fodoj

This comment has been minimized.

Show comment
Hide comment
@Fodoj

Fodoj Jan 30, 2017

Contributor

That's a really good trick! 👏

Contributor

Fodoj commented Jan 30, 2017

That's a really good trick! 👏

@tmichel

This comment has been minimized.

Show comment
Hide comment
@tmichel

tmichel Jan 30, 2017

We faced similar issues and ended up using credstash to store secrets. We also wrote a simple terraform provider to integrate with it: https://github.com/sspinc/terraform-provider-credstash

This solves the problem of secrets in variable files but secrets still leak into the state file in plain text.

tmichel commented Jan 30, 2017

We faced similar issues and ended up using credstash to store secrets. We also wrote a simple terraform provider to integrate with it: https://github.com/sspinc/terraform-provider-credstash

This solves the problem of secrets in variable files but secrets still leak into the state file in plain text.

@hardboiled

This comment has been minimized.

Show comment
Hide comment
@hardboiled

hardboiled Jan 30, 2017

@tmichel Yes, you're right. I checked, and even though I used remote state to pull in the passwords, they were still stored as part of the local state file. However, all my state files, regardless of whether they are my tfstate files or the "config" file I used above as a workaround, are stored on a versioned s3 bucket with encryption. Thus, I don't think this changes the security around my original suggestion.

hardboiled commented Jan 30, 2017

@tmichel Yes, you're right. I checked, and even though I used remote state to pull in the passwords, they were still stored as part of the local state file. However, all my state files, regardless of whether they are my tfstate files or the "config" file I used above as a workaround, are stored on a versioned s3 bucket with encryption. Thus, I don't think this changes the security around my original suggestion.

@coen-hyde

This comment has been minimized.

Show comment
Hide comment
@coen-hyde

coen-hyde Jan 30, 2017

As @tmichel pointed out since secrets leak into the state file anyway we use self referential outputs via a wrapper script with state stored on s3 with encryption. That is the first time we run some infrastructure with a secret we are prompted for it. Subsequent runs pull the secret from the outputs and set the corresponding env variable with the secret.

coen-hyde commented Jan 30, 2017

As @tmichel pointed out since secrets leak into the state file anyway we use self referential outputs via a wrapper script with state stored on s3 with encryption. That is the first time we run some infrastructure with a secret we are prompted for it. Subsequent runs pull the secret from the outputs and set the corresponding env variable with the secret.

SergK added a commit to SergK/terraform that referenced this issue Mar 19, 2017

[docs][tfstate]Align document to the general recommendation
Terraform also puts some state into the `terraform.tfstate` file
by default. This state file is extremely important...
This will mean that any potential secrets
stored in the state file, will not be checked into version control

Related issue #516
@tgjamin

This comment has been minimized.

Show comment
Hide comment
@tgjamin

tgjamin Mar 23, 2017

Update: We have started using git-crypt https://github.com/AGWA/git-crypt for all of our .tfstate and .tfstate.backup files, using our GPG keys to unlock them. This way we can keep all of our state in the git repo we have our other terraform files in, without using remote state storage.

We are also using https://github.com/sspinc/terraform-provider-credstash as we already used credstash for credentials, and this way they are hidden from our terraform files (namely password field of AWS RDS instances)

tgjamin commented Mar 23, 2017

Update: We have started using git-crypt https://github.com/AGWA/git-crypt for all of our .tfstate and .tfstate.backup files, using our GPG keys to unlock them. This way we can keep all of our state in the git repo we have our other terraform files in, without using remote state storage.

We are also using https://github.com/sspinc/terraform-provider-credstash as we already used credstash for credentials, and this way they are hidden from our terraform files (namely password field of AWS RDS instances)

@odupuy

This comment has been minimized.

Show comment
Hide comment
@odupuy

odupuy May 15, 2017

We have the original password of RDS in Vault. Some options with S3 are to use acl=private, encrypt=true and possibly your own KMS key but it's not great still.
As well, you could use S3 to retrieve the state, then you change the configuration to use the local file system (terraform remote config -backend=local -backend-config="path=.terraform/terraform.state"), run your terraform apply, then obfuscate some values in the state file and manually push the state to S3 in the same location.
This being said, it would be nice having a simple solution of some sort.

odupuy commented May 15, 2017

We have the original password of RDS in Vault. Some options with S3 are to use acl=private, encrypt=true and possibly your own KMS key but it's not great still.
As well, you could use S3 to retrieve the state, then you change the configuration to use the local file system (terraform remote config -backend=local -backend-config="path=.terraform/terraform.state"), run your terraform apply, then obfuscate some values in the state file and manually push the state to S3 in the same location.
This being said, it would be nice having a simple solution of some sort.

@dzhus

This comment has been minimized.

Show comment
Hide comment
@dzhus

dzhus May 15, 2017

How is this better than using git-crypt?

dzhus commented May 15, 2017

How is this better than using git-crypt?

@odupuy

This comment has been minimized.

Show comment
Hide comment
@odupuy

odupuy May 15, 2017

The local policy is to not have any sensitive values in our Git repositories, even encrypted. We use Vault for this as it has a fine-grained control.

odupuy commented May 15, 2017

The local policy is to not have any sensitive values in our Git repositories, even encrypted. We use Vault for this as it has a fine-grained control.

@swoodford

This comment has been minimized.

Show comment
Hide comment
@swoodford

swoodford May 24, 2017

+1 on this issue, it's a big problem to have IAM secret keys and RDS database passwords in plaintext tfstate files.

swoodford commented May 24, 2017

+1 on this issue, it's a big problem to have IAM secret keys and RDS database passwords in plaintext tfstate files.

@netflash

This comment has been minimized.

Show comment
Hide comment
@netflash

netflash May 25, 2017

@swoodford you can use pgp encryption for IAM keys, https://www.terraform.io/docs/providers/aws/r/iam_access_key.html#pgp_key.
But nothing like that for RDS passwords :-(

netflash commented May 25, 2017

@swoodford you can use pgp encryption for IAM keys, https://www.terraform.io/docs/providers/aws/r/iam_access_key.html#pgp_key.
But nothing like that for RDS passwords :-(

@mwarkentin

This comment has been minimized.

Show comment
Hide comment
@mwarkentin

mwarkentin May 25, 2017

Contributor

For RDS passwords, I think you can just set a dummy value in your terraform config and then modify it manually on RDS. Terraform has no way to check that a password hasn't changed, so it seems to work just fine after that without issue.

Contributor

mwarkentin commented May 25, 2017

For RDS passwords, I think you can just set a dummy value in your terraform config and then modify it manually on RDS. Terraform has no way to check that a password hasn't changed, so it seems to work just fine after that without issue.

@swoodford

This comment has been minimized.

Show comment
Hide comment
@swoodford

swoodford May 25, 2017

thanks @netflash that's great to know it's one option available!

I ended up writing my own script to redact the secret keys and ses smtp password from the state files: https://github.com/swoodford/aws/blob/master/terraform-redact-iam-secrets.sh

It may be helpful to others that want to totally remove them from tf state and git.

swoodford commented May 25, 2017

thanks @netflash that's great to know it's one option available!

I ended up writing my own script to redact the secret keys and ses smtp password from the state files: https://github.com/swoodford/aws/blob/master/terraform-redact-iam-secrets.sh

It may be helpful to others that want to totally remove them from tf state and git.

@sarneaud

This comment has been minimized.

Show comment
Hide comment
@sarneaud

sarneaud Aug 8, 2017

This won't solve every problem raised in this thread, but it works for a lot of use cases:

I've been experimenting with adding a new no_store key to the lifecycle block, which would be a list of attributes to not store in the state file. It would have to imply ignore_changes. In theory, ignore_changes could be changed to behave this way, too, but that wouldn't give anyone a migration path for dealing with any breakages. ignore_changes could be deprecated.

I'd like some feedback before I put the work into making a production-ready PR, though.

sarneaud commented Aug 8, 2017

This won't solve every problem raised in this thread, but it works for a lot of use cases:

I've been experimenting with adding a new no_store key to the lifecycle block, which would be a list of attributes to not store in the state file. It would have to imply ignore_changes. In theory, ignore_changes could be changed to behave this way, too, but that wouldn't give anyone a migration path for dealing with any breakages. ignore_changes could be deprecated.

I'd like some feedback before I put the work into making a production-ready PR, though.

sarneaud added a commit to govau/terraform that referenced this issue Aug 14, 2017

Add a no_store meta-parameter
This partially addresses #516.

no_store is like ignore_changes but stronger.  A no_store attribute
isn't stored in the state file at all.

Existing end users and providers shouldn't be relying on ignore_changes
attributes being stored in the state file, so in principle this could be
implemented without creating a new parameter.  However, that wouldn't
leave any migration path for anything affected by a significant change
in behaviour.  The new parameter has a name that better reflects its
more general usage.

This PR deprecates ignore_changes.  no_store has been implemented with
the assumption that the ignore_changes code will eventually be deleted.

Because "not storing in state" is a functional requirement and not just
an implementation detail, a check for this was added to the end-to-end
test.
@aymericbeaumet

This comment has been minimized.

Show comment
Hide comment
@aymericbeaumet

aymericbeaumet Oct 2, 2017

We use https://github.com/StackExchange/blackbox to commit the sensitive data next to our sources, give it a try that may match your needs.

aymericbeaumet commented Oct 2, 2017

We use https://github.com/StackExchange/blackbox to commit the sensitive data next to our sources, give it a try that may match your needs.

@rayterrill

This comment has been minimized.

Show comment
Hide comment
@rayterrill

rayterrill Nov 30, 2017

Running into this same issue with the windows_opt_config.domain_user_password in the vsphere_virtual_machine resource in the VMware vSphere Provider. I would love some way to say "don't store this to the state file" - a no_store parameter sounds perfect.

rayterrill commented Nov 30, 2017

Running into this same issue with the windows_opt_config.domain_user_password in the vsphere_virtual_machine resource in the VMware vSphere Provider. I would love some way to say "don't store this to the state file" - a no_store parameter sounds perfect.

@daveadams

This comment has been minimized.

Show comment
Hide comment
@daveadams

daveadams Apr 5, 2018

Contributor

FWIW, this feels like a great opportunity for some Terraform and Vault integration. I'd love to see the ability to specify various encryption providers up front and then within a resource to specify certain fields to encrypt.

provider "transit" {
  alias = "tfstate"
  vault_addr = "https://myvault.example.com"
  key_name = "tfstate"
}

resource "aws_iam_access_key" {
  user = "myuser"
  encrypted_state {
    provider = "transit.tfstate"
    attributes = ["secret"]
  }
}

This would avoid the weird hacky per-resource workaround like the aws_iam_access_key resource has, and would provide a solution for any field in any resource that could be pluggable with any number of encryption providers. KMS, GPG, etc.

Contributor

daveadams commented Apr 5, 2018

FWIW, this feels like a great opportunity for some Terraform and Vault integration. I'd love to see the ability to specify various encryption providers up front and then within a resource to specify certain fields to encrypt.

provider "transit" {
  alias = "tfstate"
  vault_addr = "https://myvault.example.com"
  key_name = "tfstate"
}

resource "aws_iam_access_key" {
  user = "myuser"
  encrypted_state {
    provider = "transit.tfstate"
    attributes = ["secret"]
  }
}

This would avoid the weird hacky per-resource workaround like the aws_iam_access_key resource has, and would provide a solution for any field in any resource that could be pluggable with any number of encryption providers. KMS, GPG, etc.

@korfuri

This comment has been minimized.

Show comment
Hide comment
@korfuri

korfuri Jun 28, 2018

Contributor

I think the approach of solving this in Terraform core is required, because the sensitive values are not just stored in the state file: they also end up in plan files. This is significant if your workflow involves storing plans, which you're definitely doing if you're using Terraform Enterprise.

With support for sensitive values in TF core, sensitive values should not be stored in the plan file at all. Instead, a step to generate/load the sensitive value should be added to the plan.

Contributor

korfuri commented Jun 28, 2018

I think the approach of solving this in Terraform core is required, because the sensitive values are not just stored in the state file: they also end up in plan files. This is significant if your workflow involves storing plans, which you're definitely doing if you're using Terraform Enterprise.

With support for sensitive values in TF core, sensitive values should not be stored in the plan file at all. Instead, a step to generate/load the sensitive value should be added to the plan.

@marrik96

This comment has been minimized.

Show comment
Hide comment
@marrik96

marrik96 Aug 25, 2018

+1 on Terraform core handling secrets better. Not ready for enterprise IMO until then.

marrik96 commented Aug 25, 2018

+1 on Terraform core handling secrets better. Not ready for enterprise IMO until then.

@gtmtech

This comment has been minimized.

Show comment
Hide comment
@gtmtech

gtmtech Aug 28, 2018

@apparentlymart Are there any plans to solve this at the moment? Because terraform has no secure data support, other products are being compromised (e.g. the vault provider for hashicorp vault via terraform-providers/terraform-provider-vault#160 ).

gtmtech commented Aug 28, 2018

@apparentlymart Are there any plans to solve this at the moment? Because terraform has no secure data support, other products are being compromised (e.g. the vault provider for hashicorp vault via terraform-providers/terraform-provider-vault#160 ).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment