Avoid storing aws_ssm_parameter SecureStrings in terraform state

[acaprari]

I was trying to use the AWS Parameter Store as a way to safely store sensitive data such as passwords, api keys, etc. Creating a parameter of type Secure String looked like a good way to handle this kind of data.

For this work an aws_ssm_resource has been defined using a variable to fill the parameter value, in order to avoid storing it on the VCS:

resource "aws_ssm_parameter" "secret" {
  lifecycle {
    ignore_changes = ["value"]
  }

  type = "SecureString"
  name = "secret"
  value = "${var.secret}"
}

Unfortunately, with that definition the secret value leaks out to the terraform state too, so the sensitive information gets duplicated (as pointed out in the documentation)

Is there any way to avoid such a situation? I was wondering whether saving the value property to the terraform state is strictly required.

[tkh]

I am trying to solve this problem as well and was thinking to go a similar route.

Storing your state in an encrypted backend (ex: S3) would probably solve the issue for you to avoid having those values lying around in the open or committed to VCS if you are currently storing state there.

[jveldboom]

We're experiencing this as well and as a workaround, you can use a null_resource and local-exec within a module. Kind of hacky since you have to define count = 0 to actually remove the resource - which is a limitation of destroying provisioners at the moment.

Our usecase is to set a "defaut" value in the Paramater Store using Terraform. And then manually update the value from the console to the real secret.

module/main.tf

variable "name"   { default = "/test/terraform/inline" }
variable "type"   { default = "SecureString" }
variable "value"  { default = "default" }
variable "create" { default = true }

resource "null_resource" "ssm_test" {

  count = "${var.create}"
  triggers {
    value_change = "${var.value}"
  }

  provisioner "local-exec" {
    command = "aws ssm put-parameter --name '${var.name}' --value '${var.value}' --type ${var.type} --overwrite"
  }

  provisioner "local-exec" {
    when = "destroy"
    command = "aws ssm delete-parameter --name '${var.name}'"
  }
}

Usage:

// set "create = false" to destroy resource
module "db_password" {
  source = "module"
  name = "/test/terraform/module"
  type = "SecureString"
  value = "default password 2"
  create = true
}

[amanbisht]

We store our terraform templates in GH, passing the value of password to the module without being encrypted is a security breach. Is there anyway we could pass encrypted data and terraform could decrypt it during the apply ?

[cfbao]

I'm planing on writing a custom provider plugin to address this issue - a resource representing an SSM parameter without the value attribute at all.

If there's enough interest, I can open a PR here to hopefully add such a feature to the official AWS provider plugin. But ultimately it's up to Hashicorp to decide if they want it.

[pspot2]

What would speak against Terraform hashing the SecureString values as early as possible in memory before working with them (e.g. comparing and detecting changes)? I imagine that all values must be stored in the state in order to be able to detect changes to them. However, instead of storing plaintext values, their hashes could be written to the state instead.

Standard protection measures like encryption-at-rest of the storage (S3) and TLS-secured access to it increasingly do not suffice for security departments of various companies. Data itself must be scrambled as well.

[boarnoah]

That would certainly go further towards making this usable but doesn't cover everyone's use case unfortunately.

An ask in a lot of of the above is being able to create the parameter (and manage its lifetime) in terraform without accessing the value. For example we might want to run TF without permissions to GetSecretValue.

PS: The workaround jveldboom suggested has worked quite well for handling this without any significant downsides (as a small module to initialize multiple SSM parameters without tracking state). This should handle your use case too if you are interested.

[pspot2]

How would you then detect drift (someone/something changed the value outside of TF) without accessing the existing value? The workaround doesn't seem to cover this aspect as well.

[tometzky]

I suppose that, if "value" is in "lifecycle.ignore_changes" just saving the initial value in the state file, and just skipping GetParameter API, would be a great solution for many problems:

• parameter value can be changed by an application that uses it, while terraform might manage its creation, metadata, permissions, tagging and deletion;
• parameter real value won't be stored in state, when it can be a security risk in case of storing a database password etc.;
• parameter value can be rotated with external scheduled job, while it is not reasonable to do with terraform, which is by design interactive;
• parameter value can be protected by a KMS key, that terraform can't use (SSM parameter is read by terraform even though change lifecycle ignore_changes says to ignore it #20548);
• won't limit its usage when a user really wants to have value managed by terraform - they can just don't add "value" to "lifecycle.ignore_changes";
• avoid hacks like using null_resource with local-exec and problems with replacing and removing resource;
• can use null_resource with local-exec to just set value from some password manager or other service, but still have normal deletion and metadata and permissions managed with terraform.

[melissarh57]

@tometzky, how would you skip the GetParameter API?

[tometzky]

With AWS CLI I can use "aws ssm describe-parameters":

aws --region us-east-1 ssm describe-parameters --parameter-filters Key=Name,Option=Equals,Values=some_name

Example result is:

{
    "Parameters": [
        {
            "Name": "some_name",
            "Type": "SecureString",
            "KeyId": "18e52b72-dc17-11ec-9b27-9c7bef6a9338",
            "LastModifiedDate": "2022-04-10T13:54:05.283000+02:00",
            "LastModifiedUser": "arn:aws:iam::999999999999:user/some_user",
            "Version": 2,
            "Tier": "Standard",
            "Policies": [],
            "DataType": "text"
        }
    ]
}

When compared to

aws --region us-east-1 ssm get-parameter --name some_name

Example result:

{
    "Parameter": {
        "Name": "syncron-rnd-pm-dev-agco-rnd-us-east-1-ldap",
        "Type": "SecureString",
        "Value": "some_value",
        "Version": 2,
        "LastModifiedDate": "2022-04-10T13:54:05.283000+02:00",
        "ARN": "arn:aws:ssm:us-east-1:999999999999:parameter/some_name",
        "DataType": "text"
    }
}

Missing fields from "describe-parameters" result are "Value", which we don't need, if it is ignored, and "ARN", which can easily be calculated from current region, account_id and parameter name.

In API used by terraform-provider-aws (aws-sdk-go-v2 I think) it should be analogous.

[steven-craig]

+1 for this. the "create the SecureString in terraform and then update it manually/out of band afterwards (with a lifecycle ignore changes)" workflow is straightforward and what we want to do. it is absolutely undesirable for terraform to get the parameter value for secure strings each time.