New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errno::EACCES: Permission denied #77

Closed
filiptepper opened this Issue Jul 31, 2017 · 26 comments

Comments

Projects
None yet
@filiptepper

filiptepper commented Jul 31, 2017

When we added bootsnap to our application we started getting the following errors whenever we run anything:

"Errno::EACCES: Permission denied",
"/app/vendor/bundle/ruby/2.4.0/gems/bootsnap-1.1.2/lib/bootsnap/compile_cache/iseq.rb:37:in `fetch'",
"/app/vendor/bundle/ruby/2.4.0/gems/bootsnap-1.1.2/lib/bootsnap/compile_cache/iseq.rb:37:in `load_iseq'",
"/app/vendor/bundle/ruby/2.4.0/gems/bootsnap-1.1.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'",
"/app/vendor/bundle/ruby/2.4.0/gems/bootsnap-1.1.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'",
"/app/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:292:in `block in require'",
"/app/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:258:in `load_dependency'",
"/app/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.2/lib/active_support/dependencies.rb:292:in `require'",
"/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.2/lib/active_record/railtie.rb:34:in `block (3 levels) in <class:Railtie>'",
"/app/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'",

Cache path is /app/tmp/cache/bootsnap-compile-cache, which writable for the processes running tasks.

Also, some files are created:

$ /app/tmp/cache/bootsnap-compile-cache$ ls
00  05  0a  0f  14  19  1e  23  28  2d  32  37  3c  41  46  4b  50  55  5a  5f  64  69  6e  73  78  7d  82  87  8c  91  96  9b  a0  a5  aa  af  b4  b9  be  c3  c8  cd  d2  d7  dc  e1  e6  eb  f0  f5  fa  ff
01  06  0b  10  15  1a  1f  24  29  2e  33  38  3d  42  47  4c  51  56  5b  60  65  6a  6f  74  79  7e  83  88  8d  92  97  9c  a1  a6  ab  b0  b5  ba  bf  c4  c9  ce  d3  d8  dd  e2  e7  ec  f1  f6  fb
02  07  0c  11  16  1b  20  25  2a  2f  34  39  3e  43  48  4d  52  57  5c  61  66  6b  70  75  7a  7f  84  89  8e  93  98  9d  a2  a7  ac  b1  b6  bb  c0  c5  ca  cf  d4  d9  de  e3  e8  ed  f2  f7  fc
03  08  0d  12  17  1c  21  26  2b  30  35  3a  3f  44  49  4e  53  58  5d  62  67  6c  71  76  7b  80  85  8a  8f  94  99  9e  a3  a8  ad  b2  b7  bc  c1  c6  cb  d0  d5  da  df  e4  e9  ee  f3  f8  fd
04  09  0e  13  18  1d  22  27  2c  31  36  3b  40  45  4a  4f  54  59  5e  63  68  6d  72  77  7c  81  86  8b  90  95  9a  9f  a4  a9  ae  b3  b8  bd  c2  c7  cc  d1  d6  db  e0  e5  ea  ef  f4  f9  fe

Ruby 2.4.1, bootsnap 1.1.2.

@daniel-rikowski

This comment has been minimized.

daniel-rikowski commented Jul 31, 2017

I get these on a regular basis. I noticted it's triggered by updating certain gems, especially gems with C extensions. The permission error also seems to be related to the Ruby VM memory and not the file system.

Anyway, after clearing the bootsnap-compile-cache directory the error always goes away (until the next bundle update 😉)

@lacco

This comment has been minimized.

lacco commented Oct 1, 2017

I was eager to try bootsnap out since the 1.0 release, but I am getting this exact error message after deploying it to a server (powered by Nginx/ Passenger). I have no problems so far locally (Puma web server) in development mode. I would love to get bootsnap running on our servers!

@askl56

This comment has been minimized.

@burke

This comment has been minimized.

Member

burke commented Dec 11, 2017

If anyone is still getting this, I'd be curious to see the trace with 1.1.6.beta3.

@lacco

This comment has been minimized.

lacco commented Dec 12, 2017

@burke gem 'bootsnap', '= 1.1.6', require: false doesn't fix the problem for me:

[ E 2017-12-12 17:52:45.1600 16129/Tl age/Cor/App/Implementation.cpp:291 ]: Could not spawn process for application [...]/current: An error occurred while starting up the preloader.
  Error ID: e93e6fb6
  Error details saved to: /tmp/passenger-error-LVyCt9.html
  Message from application: Permission denied - bs_fetch:atomic_write_cache_file:open (Errno::EACCES)
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/compile_cache/iseq.rb:37:in `fetch'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/compile_cache/iseq.rb:37:in `load_iseq'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/deprecation.rb:1:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/duration.rb:5:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/core_ext/time/calculations.rb:1:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/file_update_checker.rb:1:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/i18n_railtie.rb:2:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/railtie.rb:2:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails.rb:15:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/railties-5.1.4/lib/rails/all.rb:1:in `<main>'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
  [...]/shared/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
  [...]/releases/20171212164822/config/application.rb:3:in `<top (required)>'
  [...]/releases/20171212164822/config/environment.rb:2:in `require_relative'
  [...]/releases/20171212164822/config/environment.rb:2:in `<top (required)>'
  config.ru:3:in `require'
  config.ru:3:in `block in <main>'
  [...]/shared/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'
  [...]/shared/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger-enterprise/helper-scripts/rack-preloader.rb:94:in `eval'
  /usr/share/passenger-enterprise/helper-scripts/rack-preloader.rb:94:in `preload_app'
  /usr/share/passenger-enterprise/helper-scripts/rack-preloader.rb:140:in `<module:App>'
  /usr/share/passenger-enterprise/helper-scripts/rack-preloader.rb:14:in `<module:PhusionPassenger>'
  /usr/share/passenger-enterprise/helper-scripts/rack-preloader.rb:13:in `<main>'
@burke

This comment has been minimized.

Member

burke commented Dec 12, 2017

I did some digging but didn't find any smoking gun.

So it's failing in this code:

  tmp_path = mktemp(template);
  fd = open(tmp_path, O_WRONLY | O_CREAT, 0644);
  if (fd < 0) { /* HITS THIS PATH */
    if (mkpath(path, 0755) < 0) { /* RETURNS SUCCESS */
      *errno_provenance = (char *)"bs_fetch:atomic_write_cache_file:mkpath";
      return -1;
    }
    fd = open(tmp_path, O_WRONLY | O_CREAT, 0644); /* RETURNS EACCES */
    if (fd < 0) {
      *errno_provenance = (char *)"bs_fetch:atomic_write_cache_file:open";
      return -1;
    }
  }

EACCES is defined on open(2) as:

The requested access to the file is not allowed, or search permission is denied for one of the directories in the path prefix of pathname, or the file did not exist yet and write access to the parent directory is not allowed. (See also path_resolution(7).)

tmp_path is set by:

  dest = strncpy(template, path, MAX_CACHEPATH_SIZE);
  strcat(dest, ".tmp.XXXXXX");

  tmp_path = mktemp(template);

...which means it should be identical to path, but suffixed with .tmp.123456 or some other integer.

mkpath creates the parent directory to the provided path, unless the implementation is incorrect. So that seems to work, and it uses the mode 0755, which is sufficient for the open to complete successfully, since it'll clearly be done by the same user.

Then maybe mkpath is wrong? I can't really find any issue with it. Hrm.

@burke

This comment has been minimized.

Member

burke commented Dec 12, 2017

One thing:

If your cache was already populated; that is, all of those 00, 01, etc. directories already existed -- and then you ran the application as a different user without write permission to those directories -- I can see this happening.

@lacco

This comment has been minimized.

lacco commented Dec 13, 2017

@burke Thank you for the digging, I think your comments are pointing in the right direction: The application's code is used by two different UNIX users within the same group which is why we usually use 0775 / 0664 if we need to write files. I know that this might be a quite unusual setup, but can this be solved in bootsnap or do I need to "fix" our application setup?

@burke

This comment has been minimized.

Member

burke commented Dec 13, 2017

Hm, I guess I don't really see why we couldn't make it group-usable.

@burke

This comment has been minimized.

Member

burke commented Dec 13, 2017

I just released 1.1.7, which writes caches with 0775 and 0644. We could probably catch that error in ruby-land and raise a more descriptive error but I'm going to wait and see if anyone else complains now that group-writability is toggled on.

@burke burke closed this Dec 13, 2017

@lacco

This comment has been minimized.

lacco commented Dec 14, 2017

@burke Thank you very much, with this fix I was able to deploy bootsnap to our production system! After deleting the cache from a previous deployment, everything seems to work now! Awesome work!

@lacco

This comment has been minimized.

lacco commented Dec 15, 2017

Unfortunately I needed to rollback the change, I received the permission errors again. It seems like that bootsnap is behaving correctly but that my server setup is messy. If somebody else has similar problems, I am happy to hear about possible solutions!

@burke

This comment has been minimized.

Member

burke commented Dec 15, 2017

You can't just chown -R $appuser tmp/cache before starting?

@lacco

This comment has been minimized.

lacco commented Jan 10, 2018

@burke I did some deeper digging and it feels like that my problem might be related to umasks. I can reproduce it like this:

  1. Delete bootsnap-compile-cache directory
  2. Restart nginx application process so that the cache is generated
  3. Start the Rails console from another user within the same group

This way the files are generated with wrong permissions, which is not the case when I exchange steps 2 + 3. The nginx application process and normal user have different umasks which explains the problem.

I am not sure how to fix this, it feels like that our whole server setup is just messed up. On the other hand, it might be possible to bypass umask configurations as described in https://stackoverflow.com/a/39737630/745266 , but I am not sure if this is actually a good practice or not.

@lacco

This comment has been minimized.

lacco commented Jan 10, 2018

Or maybe these permissions should be configurable, similar to https://github.com/carrierwaveuploader/carrierwave#configuring-carrierwave ?

@ozgun

This comment has been minimized.

ozgun commented Feb 6, 2018

I had a similar problem:

I've upgraded to Rails 5.2.0.rc1. The owner of the cahce/bootsnap-compile-cache directory is nginx. While I was trying to run bundle exec rails c -e p as anohter user, I was getting the error below:

.../shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.5/lib/bootsnap/compile_cache/iseq.rb:37:in `fetch': Permission denied (Errno::EACCES)
.../shared/bundle/ruby/2.4.0/gems/bootsnap-1.1.5/lib/bootsnap/compile_cache/iseq.rb:37:in `load_iseq'

Changing the permissions of cahce/bootsnap-compile-cache directory to 777 fixed the problem. Howerver, I wonder if this might cause a security problem?

Thanks.

@brunnopleffken

This comment has been minimized.

brunnopleffken commented Apr 12, 2018

I had this issue (Permission denied - bs_fetch:atomic_write_cache_file:open) when upgrading my project to Rails 5.2 on Ubuntu and Nginx. Changing permissions of cache/bootsnap-compile-cache to 777 solved the problem - although it doesn't sound right.

@maurosbu

This comment has been minimized.

maurosbu commented May 29, 2018

I have the same issue since I have two different users, one for the deploy through Capistrano, one to run the apps.
In order to be able to deploy & use apps I have to delete tmp/cache/bootsnap-compile-cache and tmp/cache/bootsnap-load-path-cache before & after the deploy.

@maurosbu

This comment has been minimized.

maurosbu commented May 29, 2018

Setting 777 seems to be the easiest solution... I don't know whether this may have security drawbacks.

@cpg

This comment has been minimized.

cpg commented Jun 1, 2018

When running my app with phusion passenger, tmp/cache/bootsnap-compile-cache/ and all of its content is created owned by root:root ... yet the app runs as apache:apache.

This probably implies that bootsnap is run for the first time at a lower level in the "passenger stack" before the app gets to switch to apache:apache. Some parts of passenger do run as root before the app server is started.

We tried to work around this, however, the app cannot really do much to work-around the fact that some files/folders are root-owned, hence permission issues are raised.

One useful feature would be to enable a bootsnap configuration for the permissions (or a umask) that the files/folders created or perhaps another flag to not put things there until whatever time later the app changes users. I realize this may not be straightforward. Perhaps a simple solution with some other way to initialize things?

We have verified that once the app is running, if we remove the tmp/cache/bootsnap* folders, they will get regenerated (as apache:apache) later, even i the app is restarted in passenger, so this means it's a first-time initialization issue.

@einSelbst

This comment has been minimized.

einSelbst commented Jun 25, 2018

I get this error with a Rails app deployed via AWS ElasticBeanstalk whenever I want to run a rails/rake command on the EC2 instance. I couldn't find a work-around so far.

@zawhtutwin

This comment has been minimized.

zawhtutwin commented Jul 27, 2018

For me, I removed bootsnap gem from the Gemfile as well as a line of code loading bootsnap from boot.rb. The problem solved.

@thehunmonkgroup

This comment has been minimized.

thehunmonkgroup commented Sep 2, 2018

This is an issue for me on Debian 8.x serving a Rails 5.2 app via Puma (managed by systemd).

Shouldn't this be re-opened given the ongoing reports?

@niyando

This comment has been minimized.

niyando commented Sep 3, 2018

@einSelbst how did you fix it?

@einSelbst

This comment has been minimized.

einSelbst commented Sep 3, 2018

@niyando we added the ec2 user to the same group as the user who creates the app. Credit goes to @a14m.

This is the ebextensions file:

commands:
  25_add_ec2_user_to_webapp_group:
    command: "usermod -a -G webapp ec2-user"
  30_create_post_deploy_dir:
    command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
 files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/99_grant_group_ownership_over_new_app_version":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      chmod -R g=u /var/app/current
@Zyphrax

This comment has been minimized.

Zyphrax commented Dec 3, 2018

On our production environment the Rails app runs as a Systemd service under a user with limited system access. If I run the console as our devops user the file permissions of the cache change and errors start popping up.

My workaround is that I skip Bootsnap in production console using:

boot.rb

ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)

require 'bundler/setup'

# Gather environment information
in_console = (ARGV & ['c', 'console']).any?
in_development = ENV['RAILS_ENV'] != 'production'

# Do not use bootsnap in production console to prevent cache file permission issues
if in_development || !in_console
  require 'bootsnap/setup'
end

It is difficult to detect console mode so early on in the Rails initialization process, but this allows us to use Bootsnap until I find a better way to work around the file permission issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment