Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for multi environment credentials. #33521

Merged
merged 1 commit into from Sep 19, 2018

Conversation

@morgoth
Copy link
Member

@morgoth morgoth commented Aug 2, 2018

Usage:

If one wants to use staging encrypted credentials:

rails credentials:edit --environment staging

This will create files config/credentials/staging.yml.enc and config/credentials/staging.key
When calling Rails.application.credentials in staging environment, it takes precedence over default
config/credentials.yml.enc
Default paths can be overwritten by setting config.credentials.content_path and config.credentials.key_path

It is backward compatible.

Another try for #30067 (comment)

@rails-bot
Copy link

@rails-bot rails-bot commented Aug 2, 2018

r? @schneems

(@rails-bot has picked a reviewer for you, use r? to override)

@morgoth
Copy link
Member Author

@morgoth morgoth commented Aug 2, 2018

r? @kaspth

@rails-bot rails-bot assigned kaspth and unassigned schneems Aug 2, 2018
@sidot3291
Copy link

@sidot3291 sidot3291 commented Aug 2, 2018

How do you handle editing different credential files? (What key do you use to decrypt, where do you put the different keys for the different files)

@morgoth
Copy link
Member Author

@morgoth morgoth commented Aug 2, 2018

@sidot3291 editing is already possible:
bin/rails encrypted:edit config/your-enctyped-file.yml.enc --key config/your-master.key

To decrypt on staging there is always config/master.key or RAILS_MASTER_KEY but it should be fine - on staging/production you don't need to decrypt 2 files, so you can set appropriate master key for given env

@kaspth
Copy link
Member

@kaspth kaspth commented Aug 12, 2018

Let's figure out a sensible file structure, so users don't have to muck with config.credentials_file. It'll simply use the env appropriate file or fall back on credentials.yml.enc.

For starters, let's try it out with config/environments/development/credentials.yml.enc and config/environments/development/master.key. Then credentials:edit needs to support a --environment option.

Then the global credentials file + master key is still geared towards production (and staging could use that as well) and the majority use case. But people could even do credentials:edit --production if they'd want it split.

There's more things to decide, but I think that makes the flow easy to understand and gives us an avenue to explore further.

@morgoth
Copy link
Member Author

@morgoth morgoth commented Aug 12, 2018

@kaspth thanks for feedback.
What if one wants to use development/test credentials without encryption? Is it fine to look for config/environments/development/credentials.yml.enc first and if not found, fallback to config/environments/development/credentials.yml? I know that it somehow defeats the original idea, but many mention it as a pain point.

Also this way I guess it won't be possible to share it between development/test (which usually are almost the same)

Also I think it will make a trouble for heroku-like flow, when staging/qa apps are using production environment, so it wouldn't be possible to specify separate encrypted credentials file for them - https://devcenter.heroku.com/articles/multiple-environments

@kaspth
Copy link
Member

@kaspth kaspth commented Aug 12, 2018

What if one wants to use development/test credentials without encryption?

Credentials should always be encrypted. They'll have to merge the unencrypted ones in in some other way.

Also this way I guess it won't be possible to share it between development/test (which usually are almost the same)

For now my stance is: too bad. I really don't care much for the development.credentials.yml.enc constructs or some such. That or my config.credentials.content_path suggestion in the main thread.

Also I think it will make a trouble for heroku-like flow, when staging/qa apps are using production environment, so it wouldn't be possible to specify separate encrypted credentials file for them - https://devcenter.heroku.com/articles/multiple-environments

What's the Heroku flow for the 5.2 credentials? Why do they need separate files? Isn't the point that staging uses production credentials? Sounds like a case for config.credentials.content_path.

@morgoth
Copy link
Member Author

@morgoth morgoth commented Aug 12, 2018

I think having config.credentials.content_path would solve many complains.

What about:

When calling Rails.application.credentials

  1. take a file from config.credentials.content_path if set
  2. if not set, fallback to config/credentials.yml.enc
  3. for new apps (Rails 6.0?) generate a line in config/environments/production.rb: config.credentials.content_path = "config/environments/production/credentials.yml.enc

or

  1. take a file from config.credentials.content_path always
  2. by default set it to config/credentials.yml.enc (so it's compatible with current apps)
  3. do not expose config.credentials.content_path in newly generated apps, but allow to set it (for people that need it)
  4. Eventually deprecate (in Rails 6.0) old location of config/credentials.yml.enc and by default generate file in config/environments/production/credentials.yml.enc, to even more emphasize it's a production thingy and not a general tool for all envs.

If one would really need to use development/test credentials (which cannot be unencrypted) then can set path by hand (eventually config.credentials.key_path could also be exposed). But I don't think that using encrypted development/test files make much sense overall.

When calling bin/rails credentials:edit --environment staging:

  1. just look in the folder config/environments/staging/ for credentails.yml.enc and master.key (this obviously cannot take config.credentials.content_path)

For a heroku-like flow it would be a little weird as there would be config/environments/staging folder, but no config/environments/staging.rb. However it would work at least.

There is a still gotcha what to do for development/test, but maybe exploring usage of Rails.application.config_for(:feature) could be an option? Like:

# config/s3.yml
production:
  access_key_id: <%= Rails.application.credentials.s3_access_key_id %>
staging:
  access_key_id: <%= Rails.application.credentials.s3_access_key_id %>
development:
  access_key_id: "dev-key"

and then calling Rails.application.config_for(:s3)["access_key_id"] in the app code. This still is not that nice in syntax and config_for doesn't cache read file.
Anyway that could be discussed separately - just a random thought :)

@dhh
Copy link
Member

@dhh dhh commented Aug 13, 2018

Here's what I'd like to see. By default, everything stays as it is, but we provide the following path to allow an environment-specific credentials file. It's automatically configured and loaded if the file config/credentials/$environment.yml.enc is available. The key used to decrypt will be either config/credentials/$environment.key or ENV["RAILS_#{Rails.env.upcase}_KEY"].

There's no merging. If you're in the development environment, and config/credentials/development.yml.enc is found, then that'll be loaded. We won't even try to do anything with config/credentials.yml.enc. We won't try to merge some intersection of both files either. These are separate files and separate stores of credentials.

No interest in doing anything with unencrypted credentials for this feature. The word "credentials" should mean "encrypted".

This approach will require no configuration. All we'd need is a way to specify an environment when using the credential commands.

Beyond this, I see any other use of encrypted configuration as an exercise for the developer. We have the primitives for saving and loading encrypted files. That should be enough.

@prashantham
Copy link

@prashantham prashantham commented Aug 13, 2018

Would the above work if environment is production, but the enc file is a staging one? Staging would usually run in production mode.

@dhh
Copy link
Member

@dhh dhh commented Aug 13, 2018

@morgoth
Copy link
Member Author

@morgoth morgoth commented Aug 13, 2018

OK, I can start with this, it's already some improvement.
@schneems maybe you could share your experience with this feature on Heroku. Are people using encrypted credentials?

@mladenilic
Copy link

@mladenilic mladenilic commented Aug 13, 2018

Definitely +1 for the multi environment credentials.

Here's my two cents on the subject. I like the idea of exposing config.credentials_file. This would allow credentials path to be set per environment and it can fallback to config/credentials.yml.enc, so the change is backward compatible.

I see why the no configuration approach with the config/credentials/$environment.yml.enc can be appealing, but I don't like the confusion it will introduce with multiple options for credentials: config/credentials/$environment.yml.enc and config/credentials.yml.enc.

Lastly, dropping any kind of support for unencrypted credentials sounds great.

Cheers.

@Edouard-chin
Copy link
Member

@Edouard-chin Edouard-chin commented Aug 13, 2018

It's automatically configured and loaded if the file config/credentials/$environment.yml.enc is available.

👍 on the idea but I'll suggest that we expose an environment variable CREDENTIALS_PATH (or something) that will take precedence over it.

The main reason is to be able in development to run a production-like environment if one has the production master key (which we can already set using the RAILS_MASTER_KEY env)

@rvirani1
Copy link

@rvirani1 rvirani1 commented Aug 13, 2018

I completely agree with @Edouard-chin. We should have a simple approach to set where the credentials file is. I've always used had applications that run in production mode, but are actually QA, staging, beta, or other environments. Those environments might use something like a Stripe test key that needs to be different from the production key.

@dhh
Copy link
Member

@dhh dhh commented Aug 13, 2018

@morgoth morgoth force-pushed the freeletics:multi-env-credentials branch to 4db8348 Aug 14, 2018
@morgoth
Copy link
Member Author

@morgoth morgoth commented Aug 14, 2018

I pushed some changes.
I'm not sure about having different name for env key (RAILS_PRODUCTION_KEY). I think using always RAILS_MASTER_KEY for all cases would be more convenient, as there is no need to have RAILS_PRODUCTION_KEY and RAILS_STAGING_KEY in the same time.


Fallback to config/credentials.yml.enc could go already to Rails 5.x.
What about having a new default for Rails 6.0 to always genereate config/credentials/production.yml.enc and config/credentials/production.key in new apps?
One reason to do it, would be to avoid having 2 ways of doing the same and to emphasize that encrypted credentials is for production-like environments.

This could be done by adding Rails.config.credentials.content_path and Rails.config.credentials.key_path that would be set to currently running environment by default, but allowing user to overwrite it (heroku-like flow). And also this way we would not break upgrading apps from 5.x to 6 as it could be done by load_defaults https://github.com/rails/rails/blob/master/railties/lib/rails/application/configuration.rb#L117-L132

@fredngo
Copy link

@fredngo fredngo commented Aug 15, 2018

@schneems maybe you could share your experience with this feature on Heroku. Are people using encrypted credentials?

Definitely would love to see @schneems' input here for how all of this can coexist with the Heroku ENV variable flow.

One possibility is to allow alligator tags in the encrypted yml file, like you can normally do in other yml files.

Example: In config/credentials/production.yml.enc, you would be able to write:

some_heroku_plugin_key: <%= ENV['SOME_HEROKU_PLUGIN_KEY' %>
some_other_api_key: axifdsr43jhgf9f

then the app can be deployed in different heroku container apps, as they are all running in production environment, but each still has their own ENV variables.

(I don't know if there would be any security implications in allowing alligator tags in the encrypted yml files -- I tried alligator tags on a vanilla credentials.yml.enc and it does not work, so it must have been disabled for a reason.)

@schneems
Copy link
Member

@schneems schneems commented Aug 15, 2018

Generally, I don't see people on Heroku using encrypted credential store at all. Instead, they use the heroku config env var store we provide them directly. As long as new behavior doesn't require people to use the encrypted config stores then it won't affect that story much.

@morgoth
Copy link
Member Author

@morgoth morgoth commented Sep 14, 2018

@dhh @kaspth This is what I was talking about freeletics@df12269 - it gives possibility to customize paths and not rely only on Rails env.

Let me know if this would be fine or if we should go with solution present in this PR.

@dhh
Copy link
Member

@dhh dhh commented Sep 14, 2018

@morgoth Quick cursory look says this is the right direction. Look for a environment-specific credentials first, if not found, fall back to global credentials.

@morgoth morgoth force-pushed the freeletics:multi-env-credentials branch from 4db8348 to b42c453 Sep 15, 2018
@morgoth morgoth changed the title [PoC] Add support for multi environment credentials. Add support for multi environment credentials. Sep 15, 2018
@morgoth morgoth force-pushed the freeletics:multi-env-credentials branch from b42c453 to b42efbe Sep 15, 2018
@kissu
Copy link

@kissu kissu commented Oct 11, 2018

Hi , sorry but I do not understand how we do actually use it.
I've pulled 5.2.1 and tried bin/rails credentials:edit --environment staging but no success.

I did not find anything in the docs neither. 😓

@morgoth
Copy link
Member Author

@morgoth morgoth commented Oct 11, 2018

@kissu it will be only available in Rails 6.0

@kissu
Copy link

@kissu kissu commented Oct 11, 2018

@morgoth I guess I understood this part poorly

This is an upgrade for Rails 6.0. But if you wanted to extract the changes into a gem and have that as a stop-gap, feel free! Or you can upgrade to this version of master and verify that it works.

So, until then, what can we use to get multi environment credentials ?

I guess sinsoku/rails-env-credentials should make it. :D

tdeck added a commit to tdeck/ally that referenced this pull request Dec 14, 2018
NOTE: This version of the code+config deploys fine with Capistrano

Since clients can't have more than one redirect URI, we've got a
separate one for local development and prod. This just hacks that in
using regular encrypted credentials.

We will want to revisit once this is released:
rails/rails#33521
tdeck added a commit to tdeck/ally that referenced this pull request Dec 14, 2018
Since clients can't have more than one redirect URI, we've got a
separate one for local development and prod. This just hacks that in
using regular encrypted credentials.

We will want to revisit once this is released:
rails/rails#33521
tdeck added a commit to tdeck/ally that referenced this pull request Dec 15, 2018
Since clients can't have more than one redirect URI, we've got a
separate one for local development and prod. This just hacks that in
using regular encrypted credentials.

We will want to revisit once this is released:
rails/rails#33521
@serggl
Copy link

@serggl serggl commented Dec 21, 2018

while we all wait for Rails 6 release, there is an easy way to add a multi environment credentials functionality to your Rails 5.2 apps without 3rd party gems. Just add a little piece of code to your application.rb:

module YourAppModule
  class Application < Rails::Application
    def credentials
      if Rails.env.production?
        super
      else
        encrypted(
          "config/credentials.#{Rails.env.downcase}.yml.enc",
          key_path: "config/#{Rails.env.downcase}.key"
        )
      end
    end
  end
end

Running RAILS_ENV=staging bin/rails credentials:edit will add some staging credentials for you (create a staging.key key file first ofcourse)

@christos
Copy link
Contributor

@christos christos commented Dec 24, 2018

While evaluating this feature, I came across a problem trying to create a new environment for my app.

Once you have created your first environment (e.g. development) you might start adding configuration in initialisers that tries to access encrypted credentials like so:

# config/initializers/sidekiq.rb
Sidekiq::Web.set :session_secret, Rails.application.credentials.secret_key_base!

That works fine when running rails credentials:edit --environment development

If you now decide to create a staging environment, doing rails credentials:edit --environment staging will fail with something similar to ...ordered_options.rb:49:in 'method_missing': :secret_key_base is blank (KeyError)

A great many gems that access 3rd party APIs use an initialiser for configuring credentials so the above situation is quite likely to occur.

I worked around the issue by creating a new empty Rails app, running rails credentials:edit --environment staging in it, and copying across the generated credential files.

Still, I now have to be careful to add any new keys I need to all existing environments before writing the code that uses them.

An alternative solution would be for rails credentials:edit to not load the application just to edit a file.

Is there any reason the entire application is loaded before editing the credentials?

@dhh
Copy link
Member

@dhh dhh commented Dec 24, 2018

@christos
Copy link
Contributor

@christos christos commented Dec 25, 2018

@dhh I had a stab at it here: #34789

Not entirely sure about my testing strategy though. I couldn't quite get my head around the entire railties testing harness.

Also, seemingly unrelated failure on Travis https://travis-ci.org/rails/rails/jobs/472199122 (mysql2/mariadb)

rokumatsumoto added a commit to rokumatsumoto/boyutluseyler that referenced this pull request Aug 4, 2019
… 6.0

rails/rails#33521
issue #18
rokumatsumoto added a commit to rokumatsumoto/boyutluseyler that referenced this pull request Aug 24, 2019
… 6.0

rails/rails#33521
issue #18
def credentials
@credentials ||= encrypted("config/credentials.yml.enc")
@credentials ||= encrypted(config.credentials.content_path, key_path: config.credentials.key_path)

This comment has been minimized.

@l33z3r

l33z3r Apr 20, 2020

Is there an issue here where this line is ignoring env_key ? If I set the key I used to encrypt my credentials file in RAILS_STAGING_KEY for example, I think it is being ignored here

This comment has been minimized.

@jonathanhefner

jonathanhefner Apr 20, 2020
Member

Although RAILS_#{options[:environment].upcase}_KEY was originally supported, it was removed in #33928. Use RAILS_MASTER_KEY instead.

This comment has been minimized.

@l33z3r

l33z3r Apr 21, 2020

Perfect, I see how to use it now thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.