AWS KMS/IAM integration?

[FransUrbo]

I'm trying to setup Hiera/eyaml on my infrastructure.

In my mockup, I simply created one set of private/public keys, had puppet distribute them onto all hosts and encrypted everything with those keys.

But now I'm trying to recreate this again, this time in separate AWS accounts, new secrets and different eyaml encryption for each environment.

It would be nice if I somehow could use the existing AWS infrastructure for this (such as IAM Roles and KMS keys/secrets). But I'm a little unsure how to do this in the most efficient and secure way.

Because eyaml requires both the public and secret key to be on the host that needs to decrypt a hiera-eyaml secret, it's somewhat difficult to keep the "blast radius" as small as possible.

Is there any "best practices" on how to actually use hiera-eyaml in a production environment, preferably in AWS?

[Cinderhaze]

I have seen an example at https://github.com/adenot/hiera-eyaml-kms but I have not tried it out personally.

[FransUrbo]

Looks nice, thanx!

I ended up using the new Parameter Store to put the Hiera/Eyaml keys in, then have those readable only to specific instance profiles. Those Hiera/Eyaml keys are then "extracted" from the Parameter Store by the/my bootstrap script (that runs on all instances being created) and put where needed.

So the instance is running on/with a specific profile, which only have access to decrypt the keys for that instance (this controlled with policies on both the KMS key and on the Role that's "attached to" the instance profile).

It's not pretty, but it "kinda works". I'm going to try that hiera-eyaml-kms plugin/module one of these days, because that's a much cleaner solution.

For posterity:

KMS key policy to allow the role to decrypt and describe the KMS key(s)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE>"
        ]
      },
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}

Role policy to allow getting the parameter

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:DescribeParameters"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters"
      ],
      "Resource": [
        "arn:aws:ssm:<REGION>:<ACCOUNT_ID>:parameter/<PARM_PREFIX>.*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": [
        "<KMS_KEY_ARN>"
      ]
    }
  ]
}

That policy is then attached to the role marked as <ROLE> in the key policy. And the role is then attached to the/a instance profile and that profile is attached to the instance.

Retrieving the parameter/eyaml keys

# Get the Hiera/eyaml keys from the 'Parameter Store'.
mkdir -p /etc/apache2/puppet
chown puppet:puppet /etc/apache2/puppet
chmod 0750 /etc/apache2/puppet

for type in public private; do
    aws ssm get-parameters --names "$(cat /etc/puppet_location.conf).hierakey.${type}" --with-decryption --region <REGION> | \
            jq -r '.Parameters[0].Value' \
            > /etc/apache2/puppet/$(cat /etc/puppet_environment.conf)-$(cat /etc/puppet_location.conf)_${type}.pem
done

So my Hiera/Eyaml key parameters are named like so: <LOCATION>.hierakey.[public|private], but I could technically retrieve anything that's named <LOCATION>.* (because of the policy above that states <PARM_PREFIX>.* where LOCATION = PARM_PREFIX).

Upload Hiera/Eyaml key to the Parameter Store

To upload/update the Hiera/Eyaml keys when/if I need to recycle them is somewhat more difficult. This because you first have to decrypt all Hiera/Eyaml secrets with

eyaml decrypt <file(s)>

using the old keys. Then create new keys

eyaml createkeys

upload those to the Parameter Store using

aws ssh put-parameter

and then reencrypt the file(s) with the new keys

eyaml encrypt <file(s)>

So it's not an easy/fast solution.

[Cinderhaze]

I haven't tried it, but I wonder if the hiera-eyaml-gpg module would work with the hiera-eyaml-kms plugin, facilitating key rotation provided new and old are within the same ring of trust.

The hiera-eyaml-gpg module is near. Ecause it creates an encryption key for the day, and then each authorized key gets the file's key encrypted for each of them to use, so that they can still access the secrets without sharing the same key

[rnelson0]

@FransUrbo any chance you would be willing to submit a PR with your docs, and anything else you may have learned since April? Thanks!

[FransUrbo]

@rnelson0 Because my "fix" isn't technically related to Hiera-eyaml at all but instead is an AWS setup thing, I don't see the point to document it (here).

If you think this is of value anyway, then by all means add a line or two to the README.md, linking to my comment above. I think that would be the simplest. That way, people could contribute to it here.

[rnelson0]

@FransUrbo I don't like linking to an issue that's closed if there's a possibility of comments. How about creating a gist and I can link to that?

[FransUrbo]

Ok, how does this look: https://gist.github.com/FransUrbo/88b26033cb513a8aa569bd5392a427b1

[rnelson0]

If you change the type to markdown (.md) it should look a lot better. I'll take a look at integrating that into the docs later today, thanks!

[FransUrbo]

Ah, cool! I was wondering on how to do that! Updated, thanx!