rails assets:precompile in production failed due to missing master key

[alphabt]

Steps to reproduce

$ rails new master_key_test
$ # Set config.require_master_key = true in production.rb
$ cat config/environments/production.rb | grep config.require_master_key
  config.require_master_key = true
$ rm config/master.key
$ RAILS_ENV=production rails assets:precompile
Missing encryption key to decrypt file with. Ask your team for your master key and write it to config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

I dockerize my Rails app and run rails assets:precompile with RAILS_ENV=production and let GitLab CI build the image, but it failed because master.key is not in the repo for security and I can't inject RAILS_MASTER_KEY during CI's build phase.

Do we really need to check for the existence of master key during assets:precompile?

Expected behavior

rails assets:precompile doesn't depend on the existence of master key even when config.require_master_key = true

Actual behavior

rails assets:precompile failed with

Missing encryption key to decrypt file with. Ask your team for your master key and write it to config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

System configuration

Rails 5.2.0
Ruby 2.5.1

[y-yagi]

This is intentional behavior.
If require_master_key is set to true, a check is made as to whether the master key exists, regardless of the executed task.

[bjacobso]

Even when I set require_master_key to false I am getting this error message. A master key seems unnecessary when building assets.

[top4ek]

Same for me. Can't precompile on build stage.

[kaspth]

@bjacobso is this the error you're seeing?

rails aborted!
ArgumentError: Missing `secret_key_base` for 'production' environment, set this string with `rails credentials:edit`

I don't think we're looking to maintain a list of commands to skip the certain aspects of production.rb with though.

[top4ek]

Using workaround SECRET_KEY_BASE=`bin/rake secret` bin/rake assets:precompile for now.

[kaspth]

Clever! 😋

[yarmiganosca]

@kaspth I understand the resistance to having some commands not load the whole environment and some load the whole environment, but assets:precompile in particular is a real pain for containers. Right now our build tooling has to provide a database connection during image build, which we do with some nonsense, and now it looks like we're going have to do a similar workaround with SECRET_KEY_BASE. I don't think this task making building containers much more awkward is that niche a concern, and it would be really great for Rails to address it.

[kaspth]

@yarmiganosca I'm not against addressing it (and I never said it was a niche concern), I'd like Rails to do so! I'm just saying that (my own interpretation here)

maintain a list of commands to skip the certain aspects of production.rb with

is a no go for us.

Likewise,

having some commands not load the whole environment and some load the whole environment

looks wrong too.

Perhaps we could defer secret_key_base assignment/validation until "something" that assets:precompile and other commands like that don't touch? Please do investigate!

---

Right now our build tooling has to provide a database connection during image build

We've been doing work to not connect to a database during boot. It could be a problem in your app. 😊

[alphabt]

@kaspth rails assets:precompile is basically a build step. Should we really need the master key during precompile? I had to perform wacky workaround by injecting dummy key into my CI build (because I don't want to expose my real key during build) to get around this issue 😢

Workaround in Dockerfile

# Precompile assets
# We use dummy master.key and credentials.yml.enc to workaround the fact that
# assets:precompile needs them but we don't want the real master.key to be built
# into the container. We will inject RAILS_MASTER_KEY env var when starting the
# container.

RUN if [[ "$RAILS_ENV" == "production" ]]; then \
      mv config/credentials.yml.enc config/credentials.yml.enc.backup; \
      mv config/credentials.yml.enc.sample config/credentials.yml.enc; \
      mv config/master.key.sample config/master.key; \
      bundle exec rails assets:precompile; \
      mv config/credentials.yml.enc.backup config/credentials.yml.enc; \
      rm config/master.key; \
    fi

[StephenEsser]

Have there been any decisions made for this issue? We're running into this issue while upgrading to Rails 5.2 on our CI builds. Without the secret key base builds cannot pass. We're cautious about adding a junk key into the CI stage when it's otherwise unnecessary.

Running the following command will fail ( locally and on CI ) unless we provide a SECRET_KEY_BASE:

RAILS_ENV=production bundle exec rake assets:precompile

The error message

ArgumentError: Missing `secret_key_base` for 'production' environment, set this string with `rails credentials:edit`

cc: @kaspth

[James-D-Wood]

I've run into this problem as well. A solution I've seen in other forums is to pass a dummy value for secret_key_base SECRET_KEY_BASE=1 RAILS_ENV=production bin/rails assets:precompile. This satisfies the presence check and the precompile process can execute successfully. Would this cause any side effects?

[whithajess]

^ Ran into this problem also in build environment - really needs to be looked at especially because currently giving an actual secret to a DockerFile is quite a pain see: https://docs.docker.com/develop/develop-images/build_enhancements/

[chemic]

Is there a proper solution for this problem, except injecting dummy SECRET_KEY_BASE to dockerfile? As We are using Gitlab Auto Devops (without dockerfile etc) and there's no way to do such a hack there.

🤔

[whithajess]

^ @y-yagi Could you possibly look at this again considering the additional information around problems?

[jmadkins]

Currently, I'm running into an issue when trying to run SECRET_KEY_BASE=1 RAILS_ENV=production bundle exec rake assets:precompile.

Currently, I'm setting up ActionMailer in my application.rb and loading a username/password from Rails.application.credentials. In the past where I haven't needed to access the credentials I've been able to pass in SECRET_KEY_BASE=1 to get everything working. Unfortunately, that doesn't work in this situation.

[jmadkins]

This really needs an official solution and shouldn't be a closed issue. It boggles my mind why it works this way and why I need to load my credentials to compile assets.

My issue is that a NoMethodError: undefined method '[]' for nil:NilClass is being thrown whenever the credentials are accessed. For my first workaround, I followed this suggestion and accessed credentials via the dig method.

This worked but several gems then complained about empty strings for the values that should have been set from the credentials. As a workaround, I'm now building assets in my Dockerfile by running RUN ASSETS_PRECOMPILE=1 SECRET_KEY_BASE=1 RAILS_ENV=production bundle exec rake assets:precompile. Then, in every initializer that uses a credential I added return if ENV['ASSETS_PRECOMPILE'].present?

I wasn't able to deploy my app for a week until I got this issue taken care of using a hack. This needs more attention.

[castlese]

I still don't know how to deal with this in Rails 6, Docker build using credentials

[RemainZany]

@alphabt - Have you found a real solution here? I implemented what you suggested above and it worked - thank you! Would love to have something that's not a workaround though.

[Oldharlem]

For me using docker build args solved this issue

My Dockerfile:

...
ARG RAILS_MASTER_KEY
RUN RAILS_MASTER_KEY=${RAILS_MASTER_KEY} RAILS_ENV=production bundle exec rails assets:precompile

I can build it with docker build --build-arg RAILS_MASTER_KEY={{master_key}}
I integrated this into github actions using github's repository secrets and passing the env to the docker build command.

[jcolbyfisher]

Using workaround SECRET_KEY_BASE=`bin/rake secret` bin/rake assets:precompile for now.

After going at this for a few hours, this was the workaround that finally worked for me. Thank you @top4ek 🙌

System Configuration:

• Ruby: 2.7.0
• Rails: 6.0.2

Environment:

• GitLab CI/CD shared runner

Excerpt from my .gitlab-ci.yml file:

build:assets_precompile:
  extends: .base_db
  stage: build
  script:
    - echo 'Precompiling assets...'
    - SECRET_KEY_BASE=`bin/rake secret` bin/rake assets:precompile --trace

[caljess599]

As @Oldharlem notes, it is doable to inject the needed variables into a containerized precompile. I've been doing this all along, and it's not a terrible inconvenience when accessing ENV variables is straightforward.

But there are plenty of situations where such access is less than straightforward. For instance, I'm designing a AMI for backup deployment in the case that the local deployment server we use goes down. So anything I need in my ENV I have to get there--securely.

In this context, it is rather disappointing to have to manage the definition of variables that aren't even necessary to the task I'm trying to accomplish. It's a clear violation of basic best practice to need a workaround in this case.

Put another way, even if declaring dummy variables works, I'm not going to be happy about it. We can do better.

[andreobrown]

For anyone using GitLab Auto DevOps and facing this problem, here is how I got it working using an Auto DevOps customisation that leverages Docker BuildKit secrets:

1. Enable docker experimental features by adding this to the top of your Dockerfile:
   # syntax = docker/dockerfile:experimental
2. For the asset compilation stage, use this command:
   RUN --mount=type=secret,id=auto-devops-build-secrets . /run/secrets/auto-devops-build-secrets && RAILS_ENV=production bundle exec rails assets:precompile

Here is my Dockerfile in full:

# syntax = docker/dockerfile:experimental
FROM ruby:2.5.1

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN apt-get update && apt-get install -y nodejs mysql-client postgresql-client sqlite3 vim --no-install-recommends && rm -rf /var/lib/apt/lists/*

ENV RAILS_ENV production
ENV RAILS_SERVE_STATIC_FILES true
ENV RAILS_LOG_TO_STDOUT true

COPY Gemfile /usr/src/app/
COPY Gemfile.lock /usr/src/app/
RUN bundle config --global frozen 1
RUN bundle install --without development test

COPY . /usr/src/app

RUN --mount=type=secret,id=auto-devops-build-secrets . /run/secrets/auto-devops-build-secrets && RAILS_ENV=production bundle exec rails assets:precompile

COPY docker-entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]

EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]

3. In GitLab add a CI/CD variable for your rails master key (mask the variable)
   RAILS_MASTER_KEY = <your real rails master key>
4. In GitLab add a CI/CD variable to forward the RAILS_MASTER_KEY variable to the Docker build process
   AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES = RAILS_MASTER_KEY

My previous attempts for others who land here:

I'm using GitLab Auto DevOps. I attempted two of the workarounds above with the following outcomes:

1. @alphabt's workaround: worked for the build process, but attempting to run the container with the real master key failed. I'm using devise and got a devise error on application boot. Devise doesn't seem to play well with using one key in the precompile process and a different key at runtime.

[+] Running 2/2
 ⠿ Container app_db_1    Running                                                                     0.0s
 ⠿ Container app_prod_1  Recreated                                                                   0.1s
Attaching to prod_1
prod_1  | rails aborted!
prod_1  | ActiveSupport::MessageEncryptor::InvalidMessage: ActiveSupport::MessageEncryptor::InvalidMessage
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/message_encryptor.rb:206:in `rescue in _decrypt'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/message_encryptor.rb:183:in `_decrypt'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/message_encryptor.rb:157:in `decrypt_and_verify'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/messages/rotator.rb:21:in `decrypt_and_verify'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_file.rb:79:in `decrypt'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_file.rb:42:in `read'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_configuration.rb:21:in `read'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_configuration.rb:33:in `config'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_configuration.rb:38:in `options'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/core_ext/module/delegation.rb:271:in `method_missing'
prod_1  | /usr/src/app/config/initializers/devise.rb:284:in `block (2 levels) in <main>'
prod_1  | /usr/local/bundle/gems/devise-jwt-0.6.0/lib/devise/jwt.rb:20:in `jwt'
prod_1  | /usr/src/app/config/initializers/devise.rb:283:in `block in <main>'
prod_1  | /usr/local/bundle/gems/devise-4.4.3/lib/devise.rb:307:in `setup'
prod_1  | /usr/src/app/config/initializers/devise.rb:5:in `<main>'

2. @Oldharlem's workaround of passing the rails master key as a build arg to Docker worked, but I was uncomfortable about it being exposed in the build process.

[itsaphel]

Suggested workaround earlier in the issue of SECRET_KEY_BASE=`bundle exec rake secret` RAILS_ENV=production bundle exec rake assets:precompile does not work for me.

Error: Missing encryption key to decrypt file with. Ask your team for your master key and write it to /config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

Setting RAILS_MASTER_KEY manually in env gives errors of improper format. I'll probably see how Rails is generating it behind the scenes and duplicate that. Must say, it's really weird this issue is a wontfix.

[TeamRainless]

Wow... It's been so long that I can't even remember how I solved this! hahahaha! :) But if you just scroll through the issue and find my name you'll see what my solution was. It's still what my server uses to this day... So I guess it worked!

…

On Fri, Aug 6, 2021 at 5:39 PM aphel ***@***.***> wrote: Suggested workaround earlier in the issue of SECRET_KEY_BASE=bundle exec rake secret RAILS_ENV=production bundle exec rake assets:precompile does not work for me. Error: Missing encryption key to decrypt file with. Ask your team for your master key and write it to /config/master.key or put it in the ENV['RAILS_MASTER_KEY']. Setting RAILS_MASTER_KEY manually in env gives errors of improper format. I'll probably see how Rails is generating it behind the scenes and duplicate that. Must say, it's really weird this issue is a wontfix. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#32947 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2KI54X6JLIDXMGS32HAUTT3P64RANCNFSM4FA4GXPQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[BrianSigafoos]

This appears to be the best solution for now from @aghalarp 👏

... Use the new Docker BuildKit features to pass in secrets to your Dockerfile.

Your Dockerfile would look something like this:

RUN --mount=type=secret,id=master_key,dst=/path/to/rails_app/config/master.key \
  RAILS_ENV=production bin/rails assets:precompile

And then build the image like so:

DOCKER_BUILDKIT=1 docker build --secret id=master_key,src=config/master.key .

Hope this helps any other Docker users that may have ran into this issue!

This works well with a Github Action to publish an image to a container registry.
You just need to set RAILS_MASTER_KEY as a repo secret in the Github repo settings.
Then use a Github action like docker/build-push-action@v2 that passes the secrets correctly in for you.

# publish_image.yml
# ...
      # https://github.com/docker/build-push-action
      - name: Build and push Docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          #  ...
          secrets: |
            "master_key=${{ secrets.RAILS_MASTER_KEY }}"

[pyrareae]

This is really irritating that the new asset system assumes the full environment along with the master key is available just to invoke webpack...

My solution is just to add a env var to skip the config in production.rb during asset compilation.

production.rb

Rails.application.configure do
#...
end unless ENV['RAILS_BUILD']

Dockerfile

#...
RUN SECRET_KEY_BASE=1 RAILS_BUILD=1 bundle exec rake assets:precompile

It would be nice to instead just create a standalone script to invoke Webpack(er) without needing to mess with this.

[curt-32a]

For anyone who encountered this issue using Capistrano when deploying to an environment, check the position of the environment variable in your .bashrc file. We put the environment variable at the end of the .bashrc and didn't realize there is a block of code at the top that kicks non-interactive shells out of .bashrc without parsing it. By moving the environment variable (RAILS_MASTER_KEY in our case) to the top, it was made available to the environment and then our assets precompiled without error.

[dhh]

Working on solving this properly here: #46760

[dhh]

Ran into exactly the same issue. We can just use the same dummy key approach as with development and test. Apologies for the (very, very) long delay on such a simple, but annoying issue.

[dhh]

If you'd like to use the SECRET_KEY_BASE_DUMMY approach, but you're not going to run against edge Rails, you can reopen the Rails::Application in your config/environment.rb file and add:

class Rails::Application
  def secret_key_base
    if Rails.env.development? || Rails.env.test? || ENV["SECRET_KEY_BASE_DUMMY"]
      secrets.secret_key_base ||= generate_development_secret
    else
      validate_secret_key_base(
        ENV["SECRET_KEY_BASE"] || credentials.secret_key_base || secrets.secret_key_base
      )
    end
  end
end

Above the Rails.application.initialize! line. That'll give you the same effect as running a version of Rails with #46760 included.

[arianf]

DHH's solution doesn't solve the problem for me.

I have two requirements:

1. I would like to have a friendly message whenever someone tries to run rails c or rails s without a key.
2. However, I would also like to have the ability to run rails assets:precompile without a key (for us within docker).

My solution is:

config.require_master_key = true if Rake.application.top_level_tasks.exclude?('assets:precompile')

# or even if you need to run `assets:clean` without a key

config.require_master_key = true if Rake.application.top_level_tasks.none? { |task| task.start_with?('assets:') }

[bradgessler]

However, I would also like to have the ability to run rails assets:precompile without a key (for us within docker).

At Fly.io, there's a lot of support issues and confusion related to this requirement. Eliminating the need to pass a "dummy" key into the precompile command would make more deploys successful. I'd be interested in putting a patch together if I could get some guidance on knowing whether or not this was attempted the past.

I recall a time when there was an asset group (I think that's what it was called) in the Gemfile that probably tried solving this problem, but that was removed. Curious what other attempts have been made at eliminating the dummy key requirement.

[martinstreicher]

Here is an approach that could simplify things... and provide options in all sorts of build/deployment/execution scenarios.

RAILS_MASTER_KEY is already an ENV var, configurable in there usual ways in the shell and within Docker files.

Could another ENV var be added -- RAILS_CREDENTIALS_FILE -- to point to what credentials file to use?

One could then set the RAILS_MASTER_KEY to a non-sensitive key and RAILS_CREDENTIALS_FILE to a corresponding encrypted credentials file. If the latter ENV var is not set, the RAILS_ENV is used to find the proper file as is done now.

If viable, I'd be happy to post a PR.

[zzak]

@bradgessler @martinstreicher I'm not advocating either of you invest time and effort into this, but generally I'd expect you to get better feedback on feature requests to discuss, or in the form of a PR. 🙏