Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add credentials using a generic EncryptedConfiguration class #30067

Merged
merged 48 commits into from Sep 11, 2017
Merged

Conversation

@dhh
Copy link
Member

dhh commented Aug 3, 2017

The combination of config/secrets.yml, config/secrets.yml.enc, and SECRET_BASE_KEY is confusing. It's not clear what you should be putting in these secrets and whether the SECRET_BASE_KEY is related to the setup in general.

This PR will deprecate secrets.yml* and instead adopt config/credentials.yml.enc to signify what these secrets are specifically for: Keeping API keys, database passwords, and any other integration credentials in one place.

This new file is a flat format, not divided by environments, like secrets.yml has been. Most of the time, these credentials are only relevant in production, and if someone does need to have some keys duplicated for different environments, it can be done by hand.

This leaves us with what to do with SECRET_BASE_KEY. For test and development, we'll instead simply derive values for both based off a known constant. These secrets are not supposed to withstand any sort of attack in test or development.

It's only in production (and derivative environments, like exposed betas) where the secret actually needs to be secret. So we will birth the new credentials.yml.enc file with secret_key_base set instead on new apps.

This change will setup the encrypted credentials along with the new application skeleton, because even if you make a mistake and forget your key, it's only production that's impacted. So it won't slow down anyone just getting started with development or test.

Finally, this would leave us with a single RAILS_MASTER_KEY for services like Heroku to set. Once this has become the standard, they could even ask for this key when it's missing on the rails buildkit deploy, such that the setup is all smooth.

Note: We will just keep Rails.secrets and friends around. The Rails.application.credentials setup will be a new, concurrent approach. All new apps would use it, but we wouldn't need to screw existing apps.

Work left:

  • Implement config.require_master_key instead of config.read_encrypted_secrets.
  • Alert at the end of "rails new app" that there is a master key, what it is, and that you should put it in your password manager.
  • Update all documentation to talk about credentials rather than secrets.
  • Update all railties tests to deal with missing secrets.yml and new master.key and credentials.yml.enc
@dhh dhh changed the title WIP: Add credentials using a generic EncryptedConfiguration class Add credentials using a generic EncryptedConfiguration class Aug 4, 2017
@@ -400,6 +401,13 @@ def secrets
end
end

def credentials
@credentials ||= ActiveSupport::EncryptedConfiguration.new \
config_path: Rails.root.join("config/credentials.yml.enc"),

This comment has been minimized.

@morgoth

morgoth Aug 4, 2017 Member

@dhh This is great that ActiveSupport::EncryptedConfiguration takes this file path as an argument.
Do you think we can expand it a little and make it configurable so one can easily work with few encrypted files, ie for "production (and derivative environments, like exposed betas)"?

And have some nice tooling like:
bin/rails credentials:edit config/credentials-with-my-custom-name.yml.enc that supports out of the box finding key with relative name?

I tried to bring it to secrets in #29909 but I can work on it here if you like the idea

This comment has been minimized.

@dhh

dhh Aug 6, 2017 Author Member

I'd like to see bin/rails encrypted:edit FILE, which will use the new config/master.key as its key, and which we can make accessible through something like Rails.application.encrypted.config(path)[:key]. Let's do that as a separate PR.

@dhh
Copy link
Member Author

dhh commented Aug 4, 2017

@dhh
Copy link
Member Author

dhh commented Aug 4, 2017

Thinking about this further, I wonder if there's even greater room for simplification. Maybe we can combine the secret_key_base with the RAILS_MASTER_KEY and then simply rely on a single master key that can live in that ENV or in config/master.key. Then we wouldn't have a separate secret_key_base, and all apps would be required to have either RAILS_MASTER_KEY env or config/master.key set. We could further more easily allow other encrypted configurations to piggy back off that env/file key, kinda like we do with the message_verifier generator.

Are there any security issues with using the RAILS_MASTER_KEY as the secret_key_base as well?

@dhh dhh requested a review from kaspth Aug 7, 2017
kaspth added 5 commits Aug 7, 2017
Usually we say we change defaults, not "spec" out a release.

Can't use backticks in our sdoc generated documentation either.
@config ||= deserialize(read).deep_symbolize_keys
end

# Saves the current configuration to file, but won't persist any comments that where there already!

This comment has been minimized.

@kaspth

kaspth Aug 7, 2017 Member

"but won't persist any comments there already!" are you mentioning the updated_contents != contents line to only write contents if they're changed?

This comment has been minimized.

@dhh

dhh Aug 7, 2017 Author Member

No, I mean that #save just serializes the in-memory config hash. It doesn't involve a text editor. Maybe we should drop that entirely.

end

def config
@config ||= deserialize(read).deep_symbolize_keys

This comment has been minimized.

@kaspth

kaspth Aug 7, 2017 Member

Should we use ActiveSupport::InheritedOptions here on top of this? It would allow things like Rails.credentials.aws_access_key?

This comment has been minimized.

@dhh

dhh Aug 17, 2017 Author Member

👍

end


attr_reader :content_path, :key_path, :env_key

This comment has been minimized.

@kaspth

kaspth Aug 7, 2017 Member

What do we gain by exposing these readers?

test "change content by key file" do
@encrypted_file.write(@content)
@encrypted_file.change do |file|
file.write(file.read + " and went by the lake")

This comment has been minimized.

@kaspth

kaspth Aug 7, 2017 Member

Really dig the flow that Pathnames and the change method bring! 👏

This comment has been minimized.

@dhh

dhh Aug 7, 2017 Author Member

Pathnames are mint!

@@ -404,6 +404,32 @@ def secrets=(secrets) #:nodoc:
@secrets = secrets
end

# The secret_key_base is used as the input secret to the application's key generator, which in turn

This comment has been minimized.

@kaspth

kaspth Aug 7, 2017 Member

Had this lined up to deprecate secrets but looks like it's a bit more involved than that. Let's defer the deprecation to after this is merged.

ActiveSupport::Deprecation.warn(<<-end_of_message.strip_heredoc)
  `Rails.application.secrets` has been deprecated in favor of `Rails.application.credentials`
  and will be removed in Rails 6.0.

  See `bin/rails credentials --help` for more.
end_of_message

(Most trouble is just that our test suite relies on secrets, so will be seeing some deprecation warnings.)

@dhh

This comment has been minimized.

I'd flip this now: config.present? ? YAML.dump(config) : ""

@kaspth
Copy link
Member

kaspth commented Jul 15, 2018

@Fedcomp not sure what you mean? SECRET_KEY_BASE isn't going anywhere. We'll derive it from the app name in development and test environments. In production, credentials can have it or encrypted secrets can have it or the ENV variable can have it.

@memiux
Copy link

memiux commented Jul 30, 2018

RIP the good ol' days.

shared:
  third_party_api_key: <%= ENV["THIRD_PARTY_API_KEY"] || "sandbox-123" %>

I remember when 12factor was a thing, secrets were fine now I'm confused. 😐

@BerkhanBerkdemir
Copy link

BerkhanBerkdemir commented Aug 2, 2018

Today, I faced "the problem", env based encryption, when I trying to integrate Algolia. I think for now, the best practice is using secure_credentials gem. Is there any best practice?

@kaspth
Copy link
Member

kaspth commented Aug 2, 2018

I'd like to make some things clear: Credentials is a conceptual rethink from Secrets and Encrypted Secrets. It charters a new path and thus solves fewer use cases than the old way did. The Core team is not in anyway opposed to supporting multiple environment credentials as long as it doesn't complicate the existing use case as @dhh made clear #30067 (comment).

However, the suggestions we've seen for multi environment credentials so far haven't been up to par. For instance, the secure_credentials gem flies right in the face of the motivation stated in the PR description:

The combination of config/secrets.yml, config/secrets.yml.enc, and SECRET_BASE_KEY is confusing. It's not clear what you should be putting in these secrets and whether the SECRET_BASE_KEY is related to the setup in general.

Do volunteer some PRs yourselves! I'm pretty confident to say that multi env credentials are a lot closer than you'd think. The biggest hurdle that I see is people continuing to conceptualize and compare with the secrets way. That's the hard way. 😄

@BerkhanBerkdemir my suggestion is here #30067 (comment). True multi env credentials could perhaps smooth this over.

Thanks for all the feedback and ideas! Those who feel the burn know best how to put out the fire, so let's level Rails up together ❤️

@sidot3291
Copy link

sidot3291 commented Aug 2, 2018

Thanks for the rallying cry @kaspth

What do you think of initializing a key and credentials file for each environment within config/environments?

e.g:

> ls config/environments:
    development.rb
    development.key
    development.credentials.yml.enc
    production.rb
    production.key
    production.credentials.yml.enc
    test.rb
    test.key
    test.credentials.yml.enc

Then:

  • All keys are .gitignore'd
  • Update CLI so it's rails credentials:edit production to edit production credentials

This adds to the config directory a bit but still allows for atomic deploys of environment variables.

My hesitation with this is that my setup would require "staging" credentials, which intuitively would require a fourth "staging" environment. CLI could be built so generating this is easy, e.g:
rails environments:add staging

It's worth noting, though, Heroku explicitly pushes me away from this approach:

It may be tempting to create another custom environment such as “staging” and create a config/environments/staging.rb and deploy to a Heroku app with RAILS_ENV=staging. This is not a good practice. Instead we recommend always running in production mode and modifying any behavior by setting your config vars.
https://devcenter.heroku.com/articles/deploying-to-a-custom-rails-environment

That raises two options:

  1. Don't care. The credentials concept is intentionally pushing atomic, checked-in environment variables instead of system environment variables.
  2. Make environment settings for the credentials key and file, e.g. in production.rb:
config.credentials.key = ENV['CREDENTIALS_KEY'] || Rails.root.join('config/credentials/production.key').read
config.credentials.file = Rails.root.join(ENV['CREDENTIALS_FILE'] || 'config/credentials/production.credentials.yml.enc')
@thomasnal
Copy link

thomasnal commented Aug 2, 2018

I would give it its own folder such as config/environments or config/credentials is a great way to go. My five cents, as explained earlier, seems to support all variety of requests we have seen.

@sidot3291
Copy link

sidot3291 commented Aug 2, 2018

As is normally the case after clicking send, I find myself leaning toward the second option, since the entirety of my fourth "staging" environment would be copy of production, except for credentials.

So to round things out:

  1. Add a config/credentials folder with:
ls config/credentials
    development.key
    development.yml.enc
    production.key
    production.yml.enc
    test.key
    test.yml.enc
  1. Update the CLI
    rails credentials:edit {token} edits the corresponding file in the credentials folder
    rails credentials:add {token} generates a new key/file pair

  2. Add two environment settings to the existing environments/{env}.rb files:

config.credentials.key = ENV['CREDENTIALS_KEY'] || Rails.root.join('config/credentials/production.key').read
config.credentials.file = Rails.root.join(ENV['CREDENTIALS_FILE'] || 'config/credentials/production.credentials.yml.enc')

Would this work?

@Fedcomp
Copy link

Fedcomp commented Aug 2, 2018

Seems to be over complicated, instead it''s easier to use plain old ENV variables.

@morgoth
Copy link
Member

morgoth commented Aug 2, 2018

hey all - please take a look at #33521 if it would be enough for your usecases

@christophermlne
Copy link

christophermlne commented Aug 2, 2018

I didn't find the old system confusing but this new system is very confusing and it confused everyone in our office. The devs are now passing the master key around through Slack.

@printercu
Copy link
Contributor

printercu commented Aug 3, 2018

@kaspth

For instance, the secure_credentials gem flies right in the face of the motivation stated in the PR description

There is no secrets.yml and secrets.yml.enc in secure_credentials gem. There are multienv secrets.yml and secrets.:env.yml.enc for any other envs. I've explained why having encrypted credentials for dev/test envs is a bad practice on this wiki page: because it gives illusion of security and encourages sharing secret key across all developers.

After trying many different ways of managing credentials, I've came to not checking secrets.yml to vcs but using secrets.sample.yml. This allows to share dev credentials selectively (yes, they are secret in some cases too) and prevent occasional check-ins of this files with sensitive data. And I like the idea of storing encrypted production credentials in vcs. So I've combined both approaches in secure_credentials.

@prashantham
Copy link

prashantham commented Aug 4, 2018

I had to implement a CI/CD environment for a rails 4.2 app on google cloud. The server image needed to have all the source including secrets but in encrypted .env files. The encrypted files would then be decrypted when instance was created using kms. Now this required different files for our stage/prod envs i.e .env.stage.enc and .env.prod.env. Google project would hold metadata related to cipher file name which are available to instances. systemd task would then decrypt them as .env.

So i think we need to have 2 env variables one for master key and another for cipher file.

@kaspth
Copy link
Member

kaspth commented Aug 12, 2018

Thanks for the further feedback everyone! @morgoth even submitted a PR for us to hammer things out on! I've submitted my thoughts about a potential file structure here: #33521 (comment)

@christophermlne can you elaborate on what you and your team found confusing about this setup? Why wasn't it confusing with the old secrets.yml + secrets.yml.enc?

@printercu sorry I wasn't clear enough. It's not the files being present, it's having two similarly named files and only one being the encrypted one.


I'm less in favor of the ENV variables suggestions to do lookup. Credentials is an attempt to cut down on the number of ENV variables an app should manage. Though we could expose config.credentials.key_path or config.credentials.key and config.credentials.content_path, which you can override via ENV variables yourselves.

@kaspth
Copy link
Member

kaspth commented Sep 20, 2018

Multiple environment credentials have just landed thanks to @morgoth! Feel free to throw as many GitHub reactions on his PR as you want 😄
#33521

I'd just like to say thanks to everyone who chimed in here with all their suggestions. It resulted in us ultimately moving forward and making Rails better. We couldn't have asked for a better outcome ❤️

jbboynton added a commit to vindennl48/HipBox that referenced this pull request Oct 21, 2019
Update the project from Rails 5 to Rails 6. This update introduces two
major changes to the app: Rails credentials, and the Webpacker gem.

Rails credentials
-----------------

Rails 5.2 introduced a replacement for Secrets: [Credentials].
Credentials provides a method for saving environment variables in an
encrypted format that is safe to keep in the repository. This feature
depends on a master key, which must be shared out-of-band.

Webpacker
---------

The [Webpacker gem] allows for easy integration of [webpack] into Rails
apps. This gem is designed to work with the asset pipeline, so existing
JavaScript and SASS should continue to work.

[Credentials]: rails/rails#30067
[Webpacker gem]: https://github.com/rails/webpacker
[webpack]: https://webpack.js.org/
jbboynton added a commit to vindennl48/HipBox that referenced this pull request Nov 23, 2019
Update the project from Rails 5 to Rails 6. This update introduces two
major changes to the app: Rails credentials, and the Webpacker gem.

Rails credentials
-----------------

Rails 5.2 introduced a replacement for Secrets: [Credentials].
Credentials provides a method for saving environment variables in an
encrypted format that is safe to keep in the repository. This feature
depends on a master key, which must be shared out-of-band.

Webpacker
---------

The [Webpacker gem] allows for easy integration of [webpack] into Rails
apps. This gem is designed to work with the asset pipeline, so existing
JavaScript and SASS should continue to work.

[Credentials]: rails/rails#30067
[Webpacker gem]: https://github.com/rails/webpacker
[webpack]: https://webpack.js.org/
jbboynton added a commit to vindennl48/HipBox that referenced this pull request Nov 24, 2019
Update the project from Rails 5 to Rails 6. This update introduces two
major changes to the app: Rails credentials, and the Webpacker gem.

Rails credentials
-----------------

Rails 5.2 introduced a replacement for Secrets: [Credentials].
Credentials provides a method for saving environment variables in an
encrypted format that is safe to keep in the repository. This feature
depends on a master key, which must be shared out-of-band.

Webpacker
---------

The [Webpacker gem] allows for easy integration of [webpack] into Rails
apps. This gem is designed to work with the asset pipeline, so existing
JavaScript and SASS should continue to work.

[Credentials]: rails/rails#30067
[Webpacker gem]: https://github.com/rails/webpacker
[webpack]: https://webpack.js.org/
jbboynton added a commit to vindennl48/HipBox that referenced this pull request Dec 3, 2019
Update the project from Rails 5 to Rails 6. This update introduces two
major changes to the app: Rails credentials, and the Webpacker gem.

Rails credentials
-----------------

Rails 5.2 introduced a replacement for Secrets: [Credentials].
Credentials provides a method for saving environment variables in an
encrypted format that is safe to keep in the repository. This feature
depends on a master key, which must be shared out-of-band.

Webpacker
---------

The [Webpacker gem] allows for easy integration of [webpack] into Rails
apps. This gem is designed to work with the asset pipeline, so existing
JavaScript and SASS should continue to work.

[Credentials]: rails/rails#30067
[Webpacker gem]: https://github.com/rails/webpacker
[webpack]: https://webpack.js.org/
vindennl48 added a commit to vindennl48/HipBox that referenced this pull request Dec 3, 2019
Update the project from Rails 5 to Rails 6. This update introduces two
major changes to the app: Rails credentials, and the Webpacker gem.

Rails credentials
-----------------

Rails 5.2 introduced a replacement for Secrets: [Credentials].
Credentials provides a method for saving environment variables in an
encrypted format that is safe to keep in the repository. This feature
depends on a master key, which must be shared out-of-band.

Webpacker
---------

The [Webpacker gem] allows for easy integration of [webpack] into Rails
apps. This gem is designed to work with the asset pipeline, so existing
JavaScript and SASS should continue to work.

[Credentials]: rails/rails#30067
[Webpacker gem]: https://github.com/rails/webpacker
[webpack]: https://webpack.js.org/
pudinha added a commit to alphagov/datagovuk_find that referenced this pull request Aug 11, 2020
This file is deprecated and, for this particular app, doesn't contain
any real secret.

See rails/rails#30067
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.