AWS Secrets Manager - retrieve json secret as a map

[zarnovican]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

AWS Secrets Manager can store secrets as a plain string or json with flat string to string mapping.

It would be nice to have native support for retrieving secrets stored as json. Currently Terraform returns only string output with no easy way to convert it to map. Ideally, there could be something like secret_map attribute on aws_secretsmanager_secret_version data source, returning secret as a map.

Terraform probably should attempt to parse the secret string only when secret_map attribute is accessed.

The motivation is to conserve the number of secrets (AWS cost), Terraform objects, API calls, etc.

New or Affected Resource(s)

• data.aws_secretsmanager_secret_version

Potential Terraform Configuration

data "aws_secretsmanager_secret_version" "myapp" {
  secret_id = "${data.aws_secretsmanager_secret.myapp.id}"
}

resource "aws_opsworks_application" "myapp" {

  environment = {
    key    = "SOME_SECRET"
    value  = "${data.aws_secretsmanager_secret_version.myapp.secret_map["SOME_SECRET"]}"
    secure = true
  }

  environment = {
    key    = "OTHER_SECRET"
    value  = "${data.aws_secretsmanager_secret_version.myapp.secret_map["OTHER_SECRET"]}"
    secure = true
  }

}

References

• Feature Request - Support Secrets Manager Key/Value in addition to String #4611 - this ticket provides a workaround for storing secrets as json.
• https://www.terraform.io/docs/providers/aws/d/secretsmanager_secret_version.html#attributes-reference

[ozbillwang]

This seems for me is a bug to set wrong schema type for output secret_string

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/data_source_aws_secretsmanager_secret_version.go#L23

		"secret_string": {
			Type:      schema.TypeString,
			Computed:  true,
			Sensitive: true,
		},

Which should be defined to schema.TypeMap

https://www.terraform.io/docs/extend/schemas/schema-types.html#typemap

The original output is ( d.Set("secret_string", output.SecretString))

2018/07/05 21:04:46 [DEBUG] Get Secret Value, its output is : {
   ARN: "arn:aws:secretsmanager:us-east-2:xxxx:secret:team_dev-Xhzkt6",
   CreatedDate: 2018-07-05 06:50:07 +0000 UTC,
   Name: "team_dev",
   SecretString: "{\"password\":\"test\"}",
   VersionId: "6b65bfe4-7908-474b-9ae6-xxxx",
   VersionStages: ["AWSCURRENT"]
 }

Similar codes in aws/structure.go can be reference

// Takes JSON in a string. Decodes JSON into
// an array of ecs.ContainerDefinition compatible objects
func expandEcsContainerDefinitions(rawDefinitions string) ([]*ecs.ContainerDefinition, error) {
        var definitions []*ecs.ContainerDefinition

        err := json.Unmarshal([]byte(rawDefinitions), &definitions)
        if err != nil {
                return nil, fmt.Errorf("Error decoding JSON: %s", err)
        }

        return definitions, nil
}

[bflad]

Secrets Manager allows two modes of operation with respect to non-binary secrets:

• A secret value being a single plain value (e.g. mypassword)
• A secret value being a JSON mapping. Currently the API only supports string keys and string values, but this may not necessarily be true forever. (e.g. {"password": "mypassword", "username": "username"})

The Secrets Manager API only returns a SecretString (Go type string) attribute to retrieve the value, which handles both cases by returning just the string value in the first case and a JSON string representing the value in the second case.

The existing secret_string (of Terraform type schema.TypeString) attribute is a direct return of the SecretString (of Go type string) attribute from the Secrets Manager API and therefore is the most correct from a Terraform resource implementation standpoint as it supports all use cases.

The second case of a JSON mapping being returned in its "raw" string format is understandably harder to work with given the current native abilities of Terraform core (as of 0.11.7) and its configuration language (HCL) though, since we do not provide an easy built-in function (e.g. jsondecode()) to convert the JSON string into a HCL map for interpolation.

Thanks to some of the stricter typing that will be available with the enhancements with Terraform's configuration language coming in the next major version of Terraform (preview blog post available here), the implementation of a jsondecde() built-in function will be provided. This should make something like the following possible:

# Potential Terraform 0.12 configuration - details may change during implementation
data "aws_secretsmanager_secret_version" "map_example" {
  secret_id = data.aws_secretsmanager_secret.map_example.id
}

output "map_value" {
  value = jsondecode(data.aws_secretsmanager_secret_version.map_example.secret_string)["example"]
}

Further tracking of the new jsondecode() built-in function can be found here: hashicorp/terraform#10363

While I cannot give personal experience in this recommendation, it is probably worth mentioning that there is a community jsondecode() implementation available in the meantime, if you are comfortable working with custom providers: https://github.com/EvilSuperstars/terraform-provider-jsondecode).

Given the above, I believe we should wait until Terraform 0.12 is released with jsondecode() as this will be the preferred and straightforward solution, rather than implementing a confusing and temporary workaround (e.g. a second schema.TypeMap attribute) or changing the behavior of the existing attribute.

[ozbillwang]

Thanks, it is clear explanation.

I understood the secret string can be three types, string, map and binary, then PR #5087 will not be suitable. I will close it.

This issue should be closed as well to not misleading the reader who need spend time on it.

I will try the plugin terraform-provider-jsondecode you mentioned, if it doesn't work as expect, I will continuous to use mine, until v0.12 get released. Because our project use secret map only and need this feature now.

[udondan]

Here's a solution for Terraform v0.11 without any dependencies:

data "external" "helper" {
  program = ["echo", "${replace(data.aws_secretsmanager_secret_version.map_example.secret_string, "\\\"", "\"")}"]
}

data.external.helper.result is the full map stored in secret_string and elements can be accessed as data.external.helper.result["example"]

Or throw it into a module and return the whole map as output:

output "parsed" {
  value = "${data.external.helper.result}"
}

[bflad]

Hi Everyone 👋

We recently released our first beta of Terraform 0.12, which includes the jsondecode() function. This enables the functionality desired here without any changes to the Terraform AWS Provider itself once we officially release a Terraform 0.12 compatible version (version 2.2.0, hopefully 🤞).

By example, if I create a secret with a JSON string:

$ aws --region us-east-2 secretsmanager create-secret --name bflad-testing --secret-string '{"key1": "value1", "key2": "value2"}'
{
    "ARN": "arn:aws:secretsmanager:us-east-2:123456789012:secret:bflad-testing-a8Wfqj",
    "Name": "bflad-testing",
    "VersionId": "ae6f71c3-e99f-4d35-b4f1-7d2037df0976"
}

It can be parsed with Terraform 0.12 and a Terraform 0.12-compatible version of the Terraform AWS Provider (I used a development snapshot of the provider from the blog post in this case) via this example configuration:

terraform {
  required_version = "0.12.0"
}

provider "aws" {
  region = "us-east-2"
}

data "aws_secretsmanager_secret" "example" {
  name = "bflad-testing"
}

data "aws_secretsmanager_secret_version" "example" {
  secret_id = data.aws_secretsmanager_secret.example.id
}

output "secret_key1_value" {
  value = jsondecode(data.aws_secretsmanager_secret_version.example.secret_string)["key1"]
}

output "secret_key2_value" {
  value = jsondecode(data.aws_secretsmanager_secret_version.example.secret_string)["key2"]
}

Its terraform apply output:

$ terraform0.12-beta1 apply
data.aws_secretsmanager_secret.example: Refreshing state...
data.aws_secretsmanager_secret_version.example: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

secret_key1_value = value1
secret_key2_value = value2

I will keep this issue open to add a jsondecode() example to the resource and data source documentation and it will be closed when we release our Terraform 0.12-compatible provider. 👍

[bflad]

Hi again 👋

Coinciding with the timing of the second beta release of Terraform 0.12, version 2.7.0 of the Terraform AWS Provider was released today and is compatible with terraform init.

Given the below configuration:

terraform {
  required_version = "0.12.0"
}

provider "aws" {
  region  = "us-east-2"
  version = "2.7.0"
}

data "aws_secretsmanager_secret" "example" {
  name = "bflad-testing"
}

data "aws_secretsmanager_secret_version" "example" {
  secret_id = data.aws_secretsmanager_secret.example.id
}

output "secret_key1_value" {
  value = jsondecode(data.aws_secretsmanager_secret_version.example.secret_string)["key1"]
}

output "secret_key2_value" {
  value = jsondecode(data.aws_secretsmanager_secret_version.example.secret_string)["key2"]
}

And running with Terraform 0.12.0-beta2:

$ aws --region us-east-2 secretsmanager create-secret --name bflad-testing --secret-string '{"key1": "value1", "key2": "value2"}'
{
    "ARN": "arn:aws:secretsmanager:us-east-2:--OMITTED--:secret:bflad-testing-nENYtI",
    "Name": "bflad-testing",
    "VersionId": "1e22208f-10ab-4ed6-bade-7b043abac110"
}

$ terraform0.12-beta2 init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws (terraform-providers/aws)" (2.7.0)...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform0.12-beta2 apply
data.aws_secretsmanager_secret.example: Refreshing state...
data.aws_secretsmanager_secret_version.example: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

secret_key1_value = value1
secret_key2_value = value2

Please create new GitHub issues for any additional feature requests or bug reports with this functionality. Enjoy!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!