Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operation not permitted - bs_fetch:atomic_write_cache_file:chmod (Errno::EPERM) #353

Closed
dmolesUC opened this issue Apr 9, 2021 · 20 comments
Closed

Comments

@dmolesUC
Copy link

@dmolesUC dmolesUC commented Apr 9, 2021

I'm running a Docker stack with two instances of the same Rails app: one, updater that just runs rails setup and exits, and another, app that's the application proper, and I'm often but not always seeing variations on the above error, but only when my stack mounts the current working directory (i.e. my development checkout) to the Rails root directory.

volumes:
  - ./:/opt/app

Probably related to #177 and possibly #262, but still present with Bootsnap 1.7.3 (Rails 6.0.3.6). It doesn't always seem to happen at the same point in the startup sequence, and it doesn't always happen with both instances.

Example 1 (updater only):

updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch': Operation not permitted - bs_fetch:atomic_write_cache_file:chmod (Errno::EPERM)
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:55:in `load_iseq'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/atomic/thread_local_var.rb:3:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/atomic/reentrant_read_write_lock.rb:5:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent/atomics.rb:8:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/concurrent-ruby-1.1.8/lib/concurrent-ruby/concurrent.rb:6:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/logger_thread_safe_level.rb:5:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/logger_silence.rb:5:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/logger.rb:3:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support.rb:29:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command.rb:3:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands.rb:3:in `<main>'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | 	from /opt/app/bin/rails:4:in `<main>'
altmedia_updater_1 exited with code 1

Example 2 (app and updater both):

app_1              | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch': Operation not permitted - bs_fetch:atomic_write_cache_file:chmod (Errno::EPERM)
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:55:in `load_iseq'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/activerecord-6.0.3.6/lib/arel/attributes.rb:3:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/activerecord-6.0.3.6/lib/arel.rb:15:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/activerecord-6.0.3.6/lib/active_record.rb:29:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/activerecord-6.0.3.6/lib/active_record/railtie.rb:3:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/all.rb:21:in `block in <main>'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/all.rb:19:in `each'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/all.rb:19:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /opt/app/config/application.rb:6:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:141:in `block in perform'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:138:in `tap'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:138:in `perform'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command/base.rb:69:in `perform'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command.rb:46:in `invoke'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /opt/app/bin/rails:4:in `<main>'
app_1              | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch': Operation not permitted - bs_fetch:atomic_write_cache_file:chmod (Errno::EPERM)
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:55:in `load_iseq'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/actionmailbox-6.0.3.6/lib/action_mailbox/mail_ext.rb:6:in `block in <main>'
app_1              | 	from /usr/local/bundle/gems/actionmailbox-6.0.3.6/lib/action_mailbox/mail_ext.rb:6:in `each'
app_1              | 	from /usr/local/bundle/gems/actionmailbox-6.0.3.6/lib/action_mailbox/mail_ext.rb:6:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/actionmailbox-6.0.3.6/lib/action_mailbox.rb:3:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/actionmailbox-6.0.3.6/lib/action_mailbox/engine.rb:9:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/all.rb:21:in `block in <main>'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/all.rb:19:in `each'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/all.rb:19:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /opt/app/config/application.rb:6:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:141:in `block in perform'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:138:in `tap'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:138:in `perform'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command/base.rb:69:in `perform'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command.rb:46:in `invoke'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /opt/app/bin/rails:4:in `<main>'
altmedia_app_1 exited with code 1
app_1              | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch': Operation not permitted - bs_fetch:atomic_write_cache_file:chmod (Errno::EPERM)
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:55:in `load_iseq'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/netaddr-1.5.1/lib/netaddr.rb:14:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:74:in `block (2 levels) in require'
app_1              | 	from /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:69:in `each'
app_1              | 	from /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:69:in `block in require'
app_1              | 	from /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:58:in `each'
app_1              | 	from /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:58:in `require'
app_1              | 	from /usr/local/lib/ruby/2.7.0/bundler.rb:174:in `require'
app_1              | 	from /opt/app/config/application.rb:7:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
app_1              | 	from /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:141:in `block in perform'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:138:in `tap'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/server/server_command.rb:138:in `perform'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
app_1              | 	from /usr/local/bundle/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command/base.rb:69:in `perform'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command.rb:46:in `invoke'
app_1              | 	from /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<main>'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
app_1              | 	from /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
app_1              | 	from /opt/app/bin/rails:4:in `<main>'
altmedia_app_1 exited with code 1
updater_1          | rails aborted!
updater_1          | Errno::EPERM: Operation not permitted - bs_fetch:atomic_write_cache_file:chmod
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/compile_cache/iseq.rb:55:in `load_iseq'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
updater_1          | /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
updater_1          | /usr/local/bundle/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:53:in `require_relative'
updater_1          | /usr/local/bundle/gems/rubocop-0.74.0/lib/rubocop.rb:197:in `<main>'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | /opt/app/config/application.rb:7:in `<main>'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | /opt/app/rakefile:4:in `<main>'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
updater_1          | /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/rake/rake_command.rb:22:in `block in perform'
updater_1          | /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands/rake/rake_command.rb:20:in `perform'
updater_1          | /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/command.rb:48:in `invoke'
updater_1          | /usr/local/bundle/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<main>'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
updater_1          | /usr/local/bundle/gems/bootsnap-1.7.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
updater_1          | /opt/app/bin/rails:4:in `<main>'
updater_1          | (See full trace by running task with --trace)
altmedia_updater_1 exited with code 1
@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 9, 2021

Seems like the common theme in all these reports is multiple processes using the same cache, which hints at a race condition.

If I'm right, as a workaround you can set a different BOOTSNAP_CACHE_DIR on one of the two process.

If that works, I'd be interested to know.

@arbesulo
Copy link

@arbesulo arbesulo commented Apr 14, 2021

I have a similar issue: the containers starting resque for background job processing fail to start.

@casperisfine Setting BOOTSNAP_CACHE_DIR to something else than the other services seems to solve the problem.

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 14, 2021

Thanks a lot for letting me know @arbesulo, that seem to confirm my suspicion, I might be able to develop a blind fix.

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 14, 2021

Ok, so I suspect it's because the two docker containers don't have the same UID, so the 0644 permission causes issues.

In the vast majority of the cases, the two process will read each others cache fine, but in rare race conditions situations, they might both try to generate a missing cache, and overwrite each others, causing one process to try to chmod a 0644 file owned by another UID.

I see a few solutions out of this:

  • Go with more liberal permissions, e.g 0666 for files and 777 for directories, but that's not great for security.
  • Totally namespaces the caches based on the UID. That will use more space, but were talking about a fringe use case.

@arbesulo and @dmolesUC I'd appreciate if one of you could confirm my theory. Could you confirm wether Process.uid matches in each of your ruby processes?

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 14, 2021

Out of curiosity, I just studied how python's does it with it's .pyc. I first executed a script as root, and the cache is 644 as well.

After modifying the script and running it as a regular users, the cache was unchanged, so I'm pretty sure Python simply gracefully recover from this class of problem, which I admit kind makes sense. The performance would be impacted, and the user probably wouldn't know why, but the failure mode is somewhat more controlled.

@arbesulo
Copy link

@arbesulo arbesulo commented Apr 14, 2021

Could you confirm wether Process.uid matches in each of your ruby processes?

I just put puts Process.uid in one of the initializers (if that is not the right place let me know where a better place might be). The ouput is 0 in both cases

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 14, 2021

Urk, here goes my theory.

So both process are root, which makes little sense.

Is your error chmod (Errno::EPERM)? Or some variation of that? Because I don't quite see how root could ever get a EPERM.

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 14, 2021

I pushed a branch with a change that I believe to fix this class of issues. I'd appreciate if people who experience this issue could use that branch for a while and tell me in a few days wether they've seen the issue:

gem 'bootsnap', github: 'Shopify/bootsnap', branch: 'handle-race-conditions'
@arbesulo
Copy link

@arbesulo arbesulo commented Apr 14, 2021

I'll give it a try. Thanks a lot for the quick fix!

@joshuapinter
Copy link

@joshuapinter joshuapinter commented Apr 14, 2021

👍 I'll do the same on my end. I had a fairly reproducible case with our 24-thread Mac Pro running parallel_tests on all threads concurrently. I'll first try and reproduce the issue on master and then see if it's resolved in your handle-race-conditions branch.

@arbesulo
Copy link

@arbesulo arbesulo commented Apr 15, 2021

No problems so far with the fix in that branch

@navidemad
Copy link

@navidemad navidemad commented Apr 15, 2021

The branch fixed the first issue, but i have now that one:

worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch': Operation not permitted - bs_fetch:open_cache_file:open (Errno::EPERM)
worker_1     | Operation not permitted - bs_fetch:open_cache_file:open
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/compile_cache/iseq.rb:30:in `fetch'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/compile_cache/iseq.rb:55:in `load_iseq'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
worker_1     | /usr/local/bundle/gems/sib-api-v3-sdk-7.4.0/lib/sib-api-v3-sdk.rb:189:in `<main>'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
worker_1     | /usr/local/bundle/bundler/gems/bootsnap-d878622782f8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
worker_1     | /usr/local/bundle/gems/bundler-2.2.16/lib/bundler/runtime.rb:66:in `block (2 levels) in require'
worker_1     | /usr/local/bundle/gems/bundler-2.2.16/lib/bundler/runtime.rb:61:in `each'
worker_1     | /usr/local/bundle/gems/bundler-2.2.16/lib/bundler/runtime.rb:61:in `block in require'
worker_1     | /usr/local/bundle/gems/bundler-2.2.16/lib/bundler/runtime.rb:50:in `each'
worker_1     | /usr/local/bundle/gems/bundler-2.2.16/lib/bundler/runtime.rb:50:in `require'
worker_1     | /usr/local/bundle/gems/bundler-2.2.16/lib/bundler.rb:173:in `require'
worker_1     | /app/config/application.rb:7:in `<top (required)>'
worker_1     | /app/config/environment.rb:2:in `require_relative'
worker_1     | /app/config/environment.rb:2:in `<top (required)>'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
worker_1     | /usr/local/bundle/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
worker_1     | /usr/local/bundle/gems/sidekiq-5.2.5/lib/sidekiq/cli.rb:288:in `boot_system'
worker_1     | /usr/local/bundle/gems/sidekiq-5.2.5/lib/sidekiq/cli.rb:46:in `run'
worker_1     | /usr/local/bundle/gems/sidekiq-5.2.5/bin/sidekiq:12:in `<top (required)>'
worker_1     | bin/sidekiq:29:in `load'
worker_1     | bin/sidekiq:29:in `<main>'
worker_1 exited with code 1
@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 16, 2021

Hum, interesting. I suppose we need to be resilient to failed reads as well. I'll try another change.

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 16, 2021

@navidemad I pushed 503e9d5 in that same branch. bundle update bootsnap should get it for you.

Thanks a lot for testing these patches.

@navidemad
Copy link

@navidemad navidemad commented Apr 16, 2021

Working so far with your last patch 💯

@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 16, 2021

Great to hear! Let's keep this issue open for now, if no-one report any more errors with this branch I'll cut a release around mid next week.

@davidharting
Copy link

@davidharting davidharting commented Apr 21, 2021

I just ran into this issue on Rails 6.1.3.1 and Ruby 3.0.1 with Bootsnap 1.7.2.
Upgrading to 1.7.3 resolved the issue.
Thank you! 🎉

davidharting added a commit to davidharting/scorecards that referenced this issue Apr 21, 2021
davidharting added a commit to davidharting/scorecards that referenced this issue Apr 21, 2021
* Docker container for dev environment was upgraded to Ruby 3.0.1. So updated Gemfile ruby version.
* After that, I ran into Shopify/bootsnap#353, so updated Bootsnap.
* Updated Profile to run migrations and to use bundler / binstubs
@casperisfine
Copy link
Contributor

@casperisfine casperisfine commented Apr 21, 2021

I just released a 1.7.4. Many thanks to all the people who helped validate this fix.

@joshuapinter
Copy link

@joshuapinter joshuapinter commented Apr 21, 2021

🎉 Awesome job @casperisfine! We've been running your branch hard for the last week and it's been rock solid. Thanks again for resolving this and releasing a new version so quickly. 👍

@dmolesUC
Copy link
Author

@dmolesUC dmolesUC commented Apr 21, 2021

Sorry to be late to my own party, but thanks, @casperisfine!

akhil-gautam added a commit to bigbinary/wheel that referenced this issue May 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants