provider/aws: New data provider to decrypt KMS secrets

[borgstrom]

This is a continuation of #7886

It adds a new data source for AWS that allows you to use the KMS service to encrypt "secrets" and then use them in your resource definitions.

data "aws_kms_secret" "db" {
    secret {
        name = "master_password"
        payload = "AQECAHgaPa0J8WadplGCqqVAr4HNvDaFSQ+NaiwIBhmm6qDSFwAAAGIwYAYJKoZIhvcNAQcGoFMwUQIBADBMBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI+LoLdvYv8l41OhAAIBEIAfx49FFJCLeYrkfMfAw6XlnxP23MmDBdqP8dPp28OoAQ=="

        context {
            foo = "bar"
        }
    }
}

resource "aws_rds_cluster" "rds" {
    master_username = "root"
    master_password = "${data.aws_kms_secret.db.master_password}"
    ...
}

This includes docs and acceptance tests.

#505 ❯❯❯ AWS_PROFILE=nwdev make testacc TEST=./builtin/providers/aws TESTARGS='-run=KmsSecret'                                                                                      
==> Checking that code complies with gofmt requirements...
go generate $(go list ./... | grep -v /terraform/vendor/)
2017/01/04 11:21:19 Generated command/internal_plugin_list.go
TF_ACC=1 go test ./builtin/providers/aws -v -run=KmsSecret -timeout 120m
=== RUN   TestAccAWSKmsSecretDataSource_basic
--- PASS: TestAccAWSKmsSecretDataSource_basic (45.24s)
PASS
ok      github.com/hashicorp/terraform/builtin/providers/aws    45.269s

I've already confirmed with AWS that these tests will only result in a fraction of $0.01 during each run (See: #7886 (comment))

[stack72]

ok @borgstrom so @jen20 and I have discussed this - we are going to merge this. We will, however, follow up with a note to the top of the documentation to suggest that when using this, decrypted secrets may be stored in state

Does this sound ok to you?

Thanks

Paul

[stack72]

% make testacc TEST=./builtin/providers/aws TESTARGS='-run=KmsSecret'                                                                     ✭
==> Checking that code complies with gofmt requirements...
go generate $(go list ./... | grep -v /terraform/vendor/)
2017/01/29 20:59:21 Generated command/internal_plugin_list.go
TF_ACC=1 go test ./builtin/providers/aws -v -run=KmsSecret -timeout 120m
=== RUN   TestAccAWSKmsSecretDataSource_basic
--- PASS: TestAccAWSKmsSecretDataSource_basic (68.39s)
PASS
ok  	github.com/hashicorp/terraform/builtin/providers/aws	68.411s

[borgstrom]

Thanks, @stack72. Glad to see this merged!

Please feel free to add admonition to the docs however you feel makes the most sense.

[mrafieee]

nice feature, but if that data is sensitive and we need a key to encrypt it how we can leave it decrypted in state files which are accessible in S3 buckets?!!!

in my case I can not use it since it should be consumed as environment variable and I do not want it be shown to everyone who has read access.

[borgstrom]

@mrafieee see the discussion in #7886, and the note in the docs https://www.terraform.io/docs/providers/aws/d/kms_secret.html

Encrypted state is something that many people want. #9556

Like others in #7886 our solution was to use KMS in our workflow to encrypt and decrypt the state ourselves before/after we run terraform and then only store the encrypted state publicly. While it added a little bit of overhead to our workflow, it's been an easy adoption.

[AndrewFarley]

Not to dredge this back up... but since we're using the KMS backends, why does the plaintext need to be in the state file? Can't terraform on-boot go lookup in KMS what the plaintext is and only use it in-memory? Can't it keep the plaintext of a KMS-related secret out of state? :\

[apparentlymart]

The AWS provider is no longer developed in this repository, so any further discussion about it should happen in the AWS provider repository where the AWS provider maintainers can see it. Thanks!