Provide encrypted secrets

[dhh]

Keeping important secrets in config/secrets.yml isn't a good idea, as all these secrets are stored in plaintext when the file is checked into version control. That's a shame. The application should be able to store actual secrets that it need to talk to internal and external services.

We can get there by providing an additional secrets file under config/secrets.yml.enc along with an interface to both edit and read this file.

To edit the file, I'm thinking something like:

$ rails secrets:edit
$ Please enter the master key: XXXXXX
$ [Rails opens config/secrets.yml.enc in unencrypted form in EDITOR]
$ [Saving the opened file will encrypt the file back into config/secrets.yml.enc]

To enable servers and services to read this file, we can set a single environment variable, like RAILS_MASTER_KEY.

We should ask for this master key if it's not available when booting the server, if config/secrets.yml.enc is present:

$ rails server
$ Please enter the master key (ENV['RAILS_MASTER_KEY'] is missing): XXXX

Note that config/secrets.yml.enc is in addition to config/secrets.yml. We can still use the latter for secrets that aren't critical, like dev/test secret_base_key. Then it'll become a user-action to add config/secrets.yml.enc (which should be created on-demand the first time rails secrets:edit is called). At the initiation time, the user will be asked to set the master key.

The two secrets files should then be merged at runtime, with enc taking priority.

[rafaelfranca]

Related with https://github.com/Shopify/ejson

[rafaelfranca]

cc @burke if you have opinions about this.

[dhh]

Yeah, ejson is good inspiration, but I'd like to retain YAML (it's way more friendly than JSON) and the interface should be integrated with Rails fully.

[kaspth]

RAILS_MASTER_KEY like straight out of Mordor!

One Key to rule them all, One Key to find them,
One Key to bring them all and in the darkness bind them

🙏

[kaspth]

Does secrets.yml.enc replace secrets.yml entirely? It might be nice with co-existence.

This way the unencrypted version can hold dev and test secrets — thereby acting as a manifest as well — and keep production secrets to the encrypted version.
Or it could just ease the usability for the dev/test keys that don't need encryption.

Although this assumes no dev or test secrets need encryption. Which I can't glean from your description if that's what you mean, @dhh.

[dhh]

Happy to keep secrets.yml as well. We'll need to anyway for backwards
compatibility.

On Sat, May 21, 2016 at 3:56 PM, Kasper Timm Hansen <
notifications@github.com> wrote:

Does secrets.yml.enc replace secrets.yml entirely? It might be nice with
co-existence.

This way the unencrypted version can hold dev and test secrets — thereby
acting as a manifest as well — and keep production secrets to the encrypted
version.
Or it could just ease the usability for the dev/test keys that don't need
encryption.

Although this assumes no dev or test secrets need encryption. Which I
can't glean from your description if that's what you mean, @dhh
https://github.com/dhh.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#25095 (comment)

[nynhex]

I think this is a great idea and would keep accidental version control commits from being a security nightmare. If you'd like some help on the implemention I'd be happy to work on this with someone else

[dhh]

Please do take a stab at it!

On Sat, May 21, 2016 at 8:08 PM, Shakycode notifications@github.com wrote:

I think this is a great idea and would keep accidental version control
commits from being a security nightmare. If you'd like some help on the
implemention I'd be happy to work on this with someone else

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#25095 (comment)

[nynhex]

@dhh Awesome, myself and @danielpclark are going to pair on this and see what we come up with. I'm heavy security minded so this would be a fun PR. Thanks for letting us work on this! 👍

[dhh]

Updated the original ticket to include the idea of keeping both an encrypted and an unencrypted secrets file at the same time.

[dnch]

If the rationale behind this is to prevent people from committing secret information to their repository, why not take the opportunity to guide people towards taking full advantage of environment variables—perhaps by incorporating something like dotenv as part of the default build—rather than implementing a potentially awkward workaround?

[jeremy]

@dnch Totally. That is the current guidance, but there's a foggy gray area around "where do these env vars come from?" We can do better. We demonstrate using config/secrets.yml with values like <%= ENV['AWS_ACCESS_KEY_ID'] %>. The envdir family of tools are a great way to inject these.

However! Some sensitive config are better off being versioned along with source code. The big bonuses are atomic deploys, versioned config, and dev/prod parity.

Atomic deploys directly from source: A change can be driven by source commits alone without careful coordination in lock step with an external provisioning mechanism (Vault, etcd, NFS-mounted shared config dir, Chef encrypted databags, packaging/slugging). That means a deploy can rolled back without special coordination and separate branches can be deployed to the same provisioned environment but use different secrets without "branching" at the secret-store layer.

Dev parity: We can use the same mechanism in development and deployment. From the application's point of view, it just has config available to it. There's nothing else to do. Decrypting config to make it available to the app is a deployment/packaging step.

(For more in this vein, check out https://github.com/elasticdog/transcrypt. It uses git clean/smudge filters to transparently encrypt/decrypt files in a git repos.)

[dhh]

My objective is rather to avoid having to deal with a rash of ENVs! So in many ways the opposite. That's the approach I consider awkward.

On May 24, 2016, at 00:24, Dan Cheail notifications@github.com wrote:

If the rationale behind this is to prevent people from committing secret information to their repository, why not take the opportunity to guide people towards taking full advantage of environment variables (perhaps by incorporating something like dotenv as part of the default build, rather than implementing a potentially awkward workaround?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[hmistry]

This is great news. I sincerely ask that when you create the new scheme, please take the time to consider the different ways a developer needs to use and handle the secrets and come up with an awesome experience that's easy and works.

I break the experiences for managing secrets into 3 areas:

1. accessing secrets on a local dev machine, a dev/staging server, and main production server
2. checking it into "source control" with versioning and distributing it to dev or servers. (Account for public source controls.)
3. servers with full access and servers with limited access like Heroku

I thought the secrets.yml was a great idea when it was released but I quickly felt it missed the mark on usability and dev happiness. I found:

• I cannot check it into source control which means I have to manually manage it and distribute it.
• Deploying it to servers with full access vs limited access dictated different management strategies.
• Servers with full access, meant I can take advantage of ignoring the file in git during development, but then have an added step of ftp'ing the file on deploy (yes this can be scripted when you get to automation level but think about all the manual times before you get to automating things).
• Servers with limited access like Heroku meant I have to check in secrets.yml into my git repo and have to be careful on every commit to not mistakenly add that file. (I'm in the camp of "not wanting to deal with ENV's", especially on my local machine.)

Lastly I know you can't cover all cases but there are a couple of major camps people fall in like Heroku and non-Heroku, or like ENV's and don't like ENV's, etc. If you can cover each major group's use case in managing secrets with a decent scheme, then just think how awesome they would all feel, the man hours saved and mistakes prevented!

[dnch]

nods

Thanks @dhh and @jeremy—I've got a better idea where you're coming from, now. The ability to do atomic deploys purely from source would be a neat win and something I hadn't considered.

The potential use / distribution of the production master key across development environments raises a red flag, though.

Do you think it would be worthwhile having Rails default to using the unencrypted secrets.yml in test / dev (thus negating the need to provide the master key on server-boot) and only reading the encrypted version in production? To reinforce the desired behaviour, it could be taken one step further by having secrets.yml added to the default .gitignore in new apps.

Would similar behaviour apply to database.yml? Is it worth incorporating database.yml into secrets.yml?

[jeremy]

The potential use / distribution of the production master key across development environments raises a red flag, though.

Yeah, no. Production secrets floating around dev laptops → 🤐. The unencrypted dev/test vs encrypted prod behavior is what's intended (and proposed). That requires config/secrets.yml to be committed to git for dev/test secrets, presumably of lower sensitivity.

Ideally, we could have separate keys for dev/test and prod/staging, suggesting that we have separate secrets files. Database credentials (but not the rest of the database configuration+settings) would be a good fit.

This is pretty susceptible to scope creep, though! Limiting this to a straightforward extension of secrets.yml behavior is just enough to ratchet forward a wee bit.

[dhh]

Agree. Let's start with the straight extension of secrets. We could think about how database.yml could potentially reference an encrypted secret.

On May 25, 2016, at 08:48, Jeremy Daer notifications@github.com wrote:

The potential use / distribution of the production master key across development environments raises a red flag, though.

Yeah, no. Production secrets floating around dev laptops → 🤐. The unencrypted dev/test vs encrypted prod behavior is what's intended (and proposed). That requires config/secrets.yml to be committed to git for dev/test secrets, presumably of lower sensitivity.

Ideally, we could have separate keys for dev/test and prod/staging, suggesting that we have separate secrets files. Database credentials (but not the rest of the database configuration+settings) would be a good fit.

This is pretty susceptible to scope creep, though! Limiting this to a straightforward extension of secrets.yml behavior is just enough to ratchet forward a wee bit.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[jgodse]

This looks like the Ansible Vault feature (http://docs.ansible.com/ansible/playbooks_vault.html), right down to the YAML file. Would it make sense to have this feature work with Ansible Vault secret files?

[dhh]

Don't think we would want to tie a Rails feature to Ansible, but let's
definitely look at the prior art for inspiration.

On Thu, May 26, 2016 at 3:20 PM, Jay Godse notifications@github.com wrote:

This looks like the Ansible Vault feature (
http://docs.ansible.com/ansible/playbooks_vault.html), right down to the
YAML file. Would it make sense to have this feature work with Ansible Vault
secret files?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#25095 (comment)

[jules2689]

Few thoughts after reading through the thread...

Env Vars

• Environment variables can lead to unintended leaks. Various tools will take a copy of the environment when submitting bugs or crashes to take a look at various other environment variables, such as GEM_HOME. Environment variables are not really a good place for secrets to be put, and IMO environment variables really should just describe the environment.
• Not to also mention that child processes would have access to env vars, and while some gem helpers will accommodate this, often times people just store env vars in their bash profile or something and now everything has access to these. Probably not best to encourage or easily accommodate something like this and instead provide a nicer encryptable way to manage this.

Different environments

• At Shopify, to manage multiple secrets files we've used the scheme name.env.yml (so secrets.development.yml secrets.production.yml etc, with the standard name.yml of course) where the most specific applies first. I would imagine this approach would work well here.

Heroku and similar

• Using encrypted files means you can place the files in github and deploy to heroku and similar.
• A Heroku build pack can be made to do the decryption up front using Heroku's env vars. Just means you need to put the credential in Heroku to use.

One key fits all?

• I wouldn't think you'd want to use the same key across many files but I you'd probably be more than welcome to do so but in Shopify's ejson, we auto-manage a _public_key entry in the file which matches a private key stored somewhere on the file system.
• I think this would make things more secure and allows the different environments to use different private keys (so that production creds aren't decrypt-able by dev machines, for example)

[mikecmpbll]

personally, in all my projects i gitignore secrets.yml and database.yml, create secrets.yml.default and database.yml.default with dev and test stuff which are checked in, and bin/setup cp's them to original filename.

new dev just has to clone and do bin/setup to get up and running, same for CI servers.

doesn't solve atomic deploy situation but might be a safer default if the primary objective is to help people avoid checking production secrets into the repo. if the primary objective is atomic deploys, however, then you'll need to have it in the repo encrypted, i guess.

[jtreitz]

we often find ourselves in situations where we have to store environment-dependent config values (e.g. "auto_logout_after_x_seconds") and end up putting them in the secrets.yml even though they are not exactly secrets simply because there doesn't seem to be a better place. ENV variables are one alternative but then again they have their own problems. Could there be something like a more general config.yml?

[burke]

FWIW I've been working on a ruby implementation of ejson (rebranded 'ecfg') with support for yaml and toml: https://github.com/shopify/ecfg-ruby.

It's pure ruby, but with a dependency on rbnacl-libsodium, which has a native extension.

Does this seem like a useful thing to pursue further?

[dhh]

burke, cool. For this I'm looking to stick with YAML and to have it integrated with our rails command. So don't think it's a direct fit.

[burke]

Cool. Note that this iteration does support yaml fwiw.

[mathieujobin]

the issue I am seeing is sharing the RAILS_MASTER_KEY with too many entities.
Since its the same file for all env, that means, all the developers, qa, test services, etc.

how do we change that master key?

a solution I am seeing is to use GPG to encrypt/decrypt.

this allow to register individual keys for employees and external services.
easily remove keys. and its easy to run a gpg-agent, so no password to type in when you run

rails secrets:edit

[mathieujobin]

I am now seeing the same red flag had already been raised and answered.

I still would like to play with the idea of using GPG, the list of allowed keys could be maintained via a list of authorized emails, with keys automatically fetched and revoked on save.

[dhh]

Happy to see a full proposal for that. Want to be sure that we hide the
plumbing enough that it doesn't become overly complicated, and nobody ends
up using it.

On Mon, Oct 24, 2016 at 3:05 PM, Mathieu Jobin notifications@github.com
wrote:

the issue I am seeing is sharing the RAILS_MASTER_KEY with too many
entities.
Since its the same file for all env, that means, all the developers, qa,
test services, etc.

how do we change that master key?

a solution I am seeing is to use GPG to encrypt/decrypt.

this allow to register individual keys for employees and external services.
easily remove keys. and its easy to run a gpg-agent, so no password to
type in when you run

rails secrets:edit

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#25095 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAKtUBdwtCMp7AnLXGG1cBva6BsGOIVks5q3Q9vgaJpZM4Ijxi1
.

[rafaelfranca]

This is what ecfg does. We can hire it behind the rails command if that is the only concern that you have about it. It does support yaml now.

[burke]

Yeah — the pure-ruby yaml parser is a little horrifying but it does exist in this version, along with toml and json: https://github.com/Shopify/ecfg-ruby/tree/master/lib/ecfg/parser

[BenMorganIO]

I have a feeling configuring RAILS_MASTER_KEY could cause a bit of confusion for those with multiple projects. If we have Project A which uses key 123 and Project B which uses ABC, then I would just do echo "export RAILS_MASTER_KEY=\"123\"" into something like my .bash_profile. If I move to Project B which I work on regularly, this might cause me to keep switching them up.

To help prevent this, we could have a config setting where we do Rails.application.config.master_key_namespace = 'PROJECT_B'. This way I can have both master keys in my bash profile and live happily ever after.

[dhh]

@BenMorganIO I don't think it's a good idea to export a global like that on a machine that runs multiple apps. Idea is that you'd start the server with that export, not keep it in .bash_profile.

[BenMorganIO]

Cool that would keep things even simpler 👍

[burke]

Was just talking with @rafaelfranca about this and ejson/ecfg. This turned out longer than I expected, sorry for the wall of text.

We can easily enough change ecfg to do whatever UX-wise; the only complex part is rewriting YAML without stripping comments or disarding human-useful formatting, which is ~95% complete in the current ecfg prototype.

There are two primary benefits we get from using ejson/ecfg at Shopify:

1. We get to commit secrets to the repo;
2. Developers can update, but not view, production secrets.

Point 2 is addressed because we use public key crypto. Each encrypted file embeds a public key, and the matching private key is retrieved from disk, in a location that is readable by the decryptor but not by the application. This permission isolation is not really feasible for Rails since we can't very well create users, setuid, etc.

I think the idea of using an environment variable is probably an okay solution, since we could just remove the variable after decrypting secrets so that the application wouldn't have trivial access to it (it would be up to the deployment to isolate the launcher script or whatever has access to the key from the application).

It's important in our use-case to have secrets separated by environment. I think the right precedence to merge things into Rails.application.secrets would be:

1. config/secrets.<environment>.ecfg.yml
2. config/secrets.<environment>.yml
3. config/secrets.ecfg.yml
4. config/secrets.yml

Files 2 and 4 would be edited directly, and files 1 and 3 would be edited via (e.g.) rails secrets:edit development and rails secrets:edit, respectively.

Workflow

If the encrypted file config/secrets.ecfg.yml contained the following:

_public_key: "c9aff3dc4fe87be13516ea97634d0f03325b5ac5a1311dc503172e78bc43e955"
s3_access_key_id: "EJ[1:LmOKDfK....<snip>...]"
s3_secret_access_key: "EJ[1:LmOKDfK...<snip>...]"

When we run rails secrets:edit, we would check the environment for the key corresponding to the listed public key. If it is found, $EDITOR will be opened with the following content:

_public_key: "c9aff3dc4fe87be13516ea97634d0f03325b5ac5a1311dc503172e78bc43e955"
s3_access_key_id: "AKIAASDFGHJKLZXCVBNM"
s3_secret_access_key: "pH8iRdXt2eyEyL3s8fPYNAyapH8iRdXt2eyEyL3s8fPYNAya"

Any changed values would be replaced and encrypted in the file on disk.

If the private key was not available locally, the file opened in $EDITOR could instead look like:

_public_key: "c9aff3dc4fe87be13516ea97634d0f03325b5ac5a1311dc503172e78bc43e955"
s3_access_key_id: "<ecfg key not available; replace to update>"
s3_secret_access_key: "<ecfg key not available; replace to update>"

When running rails secrets:edit to generate a new secrets file, we would first print the private key and instruct the user to do whatever they need to with it:

$ rails secrets:edit
Generating a new encrypted file config/secrets.ecfg.yml
The private key for this file is:
5de937da1a20a3b3f79dc1b252b6ac81af1715232456417bbdbcec2ca6ef0260

You must securely distribute this key to every machine you would like to have
access to decrypt these secrets.

Read more about key distribution at:
http://some.url/path

Continue only once you've saved this key somewhere safe.

Continue? [y/N]

Key detection

As far as key detection goes, when we start supporting multiple files with multiple keypairs, we can't just use RAILS_MASTER_KEY; we'd have to come up with some other scheme.

I think the best option would be to set RAILS_KEYS to a :-separated list like PATH, e.g.:

RAILS_KEYS=027e7185b0e2c1d60fadad06f95d5fbd5bb00b8e3e7e59f2940f44517b76a5a7:65ff8cec181e970358717cb99b8d2b40e592804bd43efd3af7eeff4963aa9ef5

Imagine the first key would be for config/secrets.ecfg.yml and the second would be for config/secrets.development.ecfg.yml.

Another option:

Given the keypair:

$ ecfg g
Public Key:
ad80e0e3c9ef5d4315d6f2dd3a2dbf92e96ec50ddaead4ccc37b17a9b36e4d0c
Private Key:
027e7185b0e2c1d60fadad06f95d5fbd5bb00b8e3e7e59f2940f44517b76a5a7

...we could set: RAILS_KEY_ad80e0=027e7185b0e2c1d60fadad06f95d5fbd5bb00b8e3e7e59f2940f44517b76a5a7

Boot/Decryption

When booting Rails in development, the process would be:

1. Load config/secrets.yml into Rails.application.secrets if present.
2. Load config/secrets.ecfg.yml into Rails.application.secrets if present,
   searching RAILS_KEYS for a valid key. Refuse to boot if no valid key is
   found.
3. As with 1, but for config/secrets.development.yml
4. As with 2, but for config/secrets.development.ecfg.yml
5. Unset RAILS_KEYS so the application has reduced ability to leak them.

[sikachu]

@burke Thank you for a good writeup. I think that made it clear on the implementation. Are you or @rafaelfranca actually going to implement the patch to Rails, or should I start working on it?

I do have some feedback from your suggestions though:

1. For some environment (read: Heroku), the only way to feed the master secret key is to use environment variable. That's not as secure, for sure, and I agree on emptying the ENV after we grab it.
2. Key detection - What do you think about just stick to RAILS_MASTER_KEY and RAILS_{ENV}_KEY instead of a colon-separated values?

On the other hand, it seems like most projects people will have at least these files:

config
├── secrets.production.ecfg.yml
├── secrets.staging.ecfg.yml
└── secrets.yml

That could get tedious over time though if you're configuring many environments and having multiple secrets. I think config/ is already messy as it is. So, I'm curious about what's your thought on using a single secrets.yml file and just have encrypted entries in them?

secrets.yml

master: &master
  _public_key: 52abdb909730f7a40262d5deb42bb1d24c4c1182122d1c76facd3a8105f6a12c
  secret_key_base: cc6cef8495...
  s3_access_key_id: AKIAASDFGHJKLZXCVBNM
  s3_secret_access_key: pH8iRdXt2eyEyL3s8fPYNAyapH8iRdXt2eyEyL3s8fPYNAya
staging:
  <<: *master
  _public_key: c9aff3dc4fe87be13516ea97634d0f03325b5ac5a1311dc503172e78bc43e955
  secret_key_base: "EJ[1:LmOKDfK...<snip>...]"
production:
  <<: *master
  _public_key: ad80e0e3c9ef5d4315d6f2dd3a2dbf92e96ec50ddaead4ccc37b17a9b36e4d0c
  secret_key_base: "EJ[1:4id8Us...<snip>...]"
  s3_access_key_id: "EJ[1:Rj4mwis...<snip>...]"
  s3_secret_access_key: "EJ[1:K4id93O...<snip>...]"

So, RAILS_STAGING_KEY would have private key that is paired with c9aff3d..., and RAILS_MASTER_KEY would have private key that is paired with 52abdb9.... However, in this case, since there's no encrypted data in master block, RAILS_MASTER_KEY is not required.

I honestly don't know if ecfg would support this specific use case, but I don't think it'd be too hard to implement. What do you think about my suggestion?

[spikex]

You might take a look at @ahoward's Sekrets Gem as (at least) a proof of concept. Provides the core functionally described above, encrypted YAML files can be loaded by calling Sekrets.settings_for(file_path).

Keys are managed it two ways, through the environment as proposed above (SEKRETS_KEY) but also in a file, RAILS_ROOT/.sekrets.key by default.

The file is an interesting approach. It simplifies per app keys, especially in development. And Sekrets ships with a Capistrano task that will upload the key file to the new deploy if it exists locally.

[BenMorganIO]

I like the concept of having a file at the root being the key. This would make things pretty straightforward, especially for beginners. Also, I love the recommendation to use ecfg. This way not only can the computer tell what is and isn't encrypted, but the human can as well.

[ahoward]

we've been using the sekrets gem @dojo4 for years with very, very good results. happy to contribute code or lessons learned. i will say that the optional file key is mandatory for newbs with many projects where env settings are the confusinz

[dhh]

Picking this up again. I like the option of a optional file key in addition to the ENV approach. That's solid. Also liking the simple API from sekrets.

Also, I'd like to start with just one secrets.enc.yml file that uses the same env mapping in there. That should be the default. Happy to have an overlay of secrets.production.enc.yml that can happen on top of that, but that shouldn't be part of the default setup.

I don't fully understand the need to use public/private key and embedding the public key, compared to just using a single key as with sekrets. Happy to be further educated, but the single key approach strikes me as simpler and good enough.

Who's interested in taking this work forward? If this is going to make it into Rails 5.1, we need to have a real PR in place within about a month.

[burke]

I don't personally have cycles to spare to drive this through anytime soon, but a couple of comments:

Public/private key is useful on larger teams. It can be useful to give everyone access to update secrets but not necessarily retrieve them, so that we don't have to rotate high-value secrets each time a developer leaves the organization.

Having separate keys per environment has an additional benefit of making it really difficult for e.g. staging to be misconfigured as production and start sending emails to customers, etc.

Shared key is definitely simpler. We (Shopify) wouldn't be able to switch to a solution this simple, but it would be a huge step forward for a lot of users.

Merging everything in a single file divided up by environment is sensible, and could even be made to work in the case of key-per-environment by just teaching the tool about public keys or shared key fingerprints or something embedded in subtrees.

[sikachu]

I think single key approach is a good start and a good step up from our current offering. Good enough for V1 MVP, I'd say. I'll take a look at sekrets gem and see how I can integrate that in.

@dhh I still can tackle this if we need it by the end of Jan.

[ahoward]

@sikachu / @dhh - feel free to ping me for any questions/help : ara@dojo4.com - i think it is simple and portable enough to drop in easily tho. the only rails/version dependent code is, i think, the cap recipe but even that might 'just work TM'

[ahoward]

@burke i also consider pub/pri keys and gave up for two reasons: 1) 80/20 rule as you point out 2) branches in git with differently encrypted keys actually solves some of the issues you point out. food for thought.

[ahoward]

last comment: if anyone would like to be a contributor to sekrets repo lemme know...