New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bootsnap to default Gemfile #29313

Merged
merged 1 commit into from Jul 17, 2017

Conversation

Projects
None yet
@burke
Contributor

burke commented Jun 1, 2017

This adds bootsnap to newly-generated projects, which generally reduces application boot times by over 50%.


I heard through the grapevine that people would be open to this.

Early releases of bootsnap only supported macOS, but recent versions now additionally support linux and (at least in theory) most *BSDs. Windows support is still not implemented.

This change is small, but not trivial, and it's been a long time since I've been in this codebase. I'm looking for guidance on how best to get this change documented, tested, and merged.

cc @rafaelfranca

@rails-bot

This comment has been minimized.

Show comment
Hide comment
@rails-bot

rails-bot Jun 1, 2017

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @eileencodes (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

rails-bot commented Jun 1, 2017

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @eileencodes (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

Show outdated Hide outdated railties/lib/rails/generators/rails/app/templates/config/boot.rb
load_path_cache: true,
autoload_paths_cache: true,
compile_cache_iseq: true,
compile_cache_yaml: true

This comment has been minimized.

@matthewd

matthewd Jun 1, 2017

Member

This is way too much / too specific configuration.

@matthewd

matthewd Jun 1, 2017

Member

This is way too much / too specific configuration.

This comment has been minimized.

@kaspth

kaspth Jun 1, 2017

Member

Maybe we should follow bundlers lead:

require "bundler/setup"    # Set up gems listed in the Gemfile
require "bootsnap/setup" # Caches and compiles gems for faster boot.

Which then just includes the above conventionalized config.

@kaspth

kaspth Jun 1, 2017

Member

Maybe we should follow bundlers lead:

require "bundler/setup"    # Set up gems listed in the Gemfile
require "bootsnap/setup" # Caches and compiles gems for faster boot.

Which then just includes the above conventionalized config.

This comment has been minimized.

@kaspth

kaspth Jun 1, 2017

Member

Could also be require "bootsnap/railtie" or Rails could just set up bootsnap automatically after executing boot.rb if the gem is present, e.g.:

# Somewhere in the Rails bowels:
begin
  require "bootsnap"

  Bootsnap.setup(…)
rescue LoadError
end
@kaspth

kaspth Jun 1, 2017

Member

Could also be require "bootsnap/railtie" or Rails could just set up bootsnap automatically after executing boot.rb if the gem is present, e.g.:

# Somewhere in the Rails bowels:
begin
  require "bootsnap"

  Bootsnap.setup(…)
rescue LoadError
end
Show outdated Hide outdated railties/lib/rails/generators/rails/app/templates/config/boot.rb
@@ -1,3 +1,17 @@
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)
require 'bundler/setup' # Set up gems listed in the Gemfile.
if RUBY_ENGINE == 'ruby' && RUBY_PLATFORM =~ /darwin|linux|bsd/

This comment has been minimized.

@matthewd

matthewd Jun 1, 2017

Member

We should always require bootsnap and let it decide whether it supports the current environment.

@matthewd

matthewd Jun 1, 2017

Member

We should always require bootsnap and let it decide whether it supports the current environment.

Show outdated Hide outdated railties/lib/rails/generators/rails/app/templates/Gemfile
@@ -25,6 +25,10 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
<%- end -%>
<% if RUBY_ENGINE == 'ruby' -%>
# Reduces boot times through caching; configured in config/boot.rb
gem 'bootsnap', '~> 1.0.0', platforms: [:mri]

This comment has been minimized.

@matthewd

matthewd Jun 1, 2017

Member

Can we loosen this version constraint a bit?

Can bootsnap be installed on all platforms, even if it won't Do Cool Stuff?

@matthewd

matthewd Jun 1, 2017

Member

Can we loosen this version constraint a bit?

Can bootsnap be installed on all platforms, even if it won't Do Cool Stuff?

This comment has been minimized.

@y-yagi

y-yagi Jun 4, 2017

Member

bootsnap only supports Ruby 2.3.0 and higher, but Rails still supports Ruby 2.2.
https://github.com/Shopify/bootsnap/blob/master/bootsnap.gemspec#L26

Should we check the Ruby version?

@y-yagi

y-yagi Jun 4, 2017

Member

bootsnap only supports Ruby 2.3.0 and higher, but Rails still supports Ruby 2.2.
https://github.com/Shopify/bootsnap/blob/master/bootsnap.gemspec#L26

Should we check the Ruby version?

This comment has been minimized.

@matthewd

matthewd Jun 4, 2017

Member

@burke could bootsnap allow installation on 2.2, and just operate in a degraded mode? AIUI path caching would still work -- it's the iseq cache that is strictly 2.3+, right?

@matthewd

matthewd Jun 4, 2017

Member

@burke could bootsnap allow installation on 2.2, and just operate in a degraded mode? AIUI path caching would still work -- it's the iseq cache that is strictly 2.3+, right?

@yivo

This comment has been minimized.

Show comment
Hide comment
@yivo

yivo Jun 4, 2017

This is not necessary thing for most Rails apps. Things must be kept as simple as possible. In future this change may bring some bugs. If people need this it is better to create separate official gem bootsnap-rails which is officially tested and ready for Rails. Merging everything is a hell.

yivo commented Jun 4, 2017

This is not necessary thing for most Rails apps. Things must be kept as simple as possible. In future this change may bring some bugs. If people need this it is better to create separate official gem bootsnap-rails which is officially tested and ready for Rails. Merging everything is a hell.

@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 5, 2017

Contributor

Ok -- I don't have any time this week to revise this, but here's what I'm going to do:

  1. Get bootsnap to install correctly even on platforms where it can't use the native extension;
  2. Have it understand which features can be enabled based on the current platform;
  3. Add a bootsnap/setup requirable that sets up the maximal set of available features based on the environment, and assuming ActiveSupport will be in use.

The method used here will still be available, but bootsnap/setup will wrap it.

Contributor

burke commented Jun 5, 2017

Ok -- I don't have any time this week to revise this, but here's what I'm going to do:

  1. Get bootsnap to install correctly even on platforms where it can't use the native extension;
  2. Have it understand which features can be enabled based on the current platform;
  3. Add a bootsnap/setup requirable that sets up the maximal set of available features based on the environment, and assuming ActiveSupport will be in use.

The method used here will still be available, but bootsnap/setup will wrap it.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Jun 5, 2017

Member

@yivo we are not merging anything. It is not even being added as a dependency of a framework. It is being added to the generated Gemfile and if people don't want they can just remove. Rails give people sane defaults so they don't need to spend time thinking if they want to use something or not.

@burke seems like a good plan 👍

Member

rafaelfranca commented Jun 5, 2017

@yivo we are not merging anything. It is not even being added as a dependency of a framework. It is being added to the generated Gemfile and if people don't want they can just remove. Rails give people sane defaults so they don't need to spend time thinking if they want to use something or not.

@burke seems like a good plan 👍

@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 12, 2017

Contributor

I've updated this PR to use bootsnap-1.1.0.pre. I'll update again to 1.1.0 after some more testing/feedback.

Changes:

  • Added a jruby version that doesn't include the native extension and doesn't support compile caching;
  • config/boot.rb now just requires bootsnap/setup;
  • require is no longer conditional on platform;
  • bootsnap/setup.rb enables whichever features are available given the current environment;
  • bootsnap/setup.rb attempts to derive the cache directory from caller.

See also bootsnap/setup.rb.

Thoughts?

Contributor

burke commented Jun 12, 2017

I've updated this PR to use bootsnap-1.1.0.pre. I'll update again to 1.1.0 after some more testing/feedback.

Changes:

  • Added a jruby version that doesn't include the native extension and doesn't support compile caching;
  • config/boot.rb now just requires bootsnap/setup;
  • require is no longer conditional on platform;
  • bootsnap/setup.rb enables whichever features are available given the current environment;
  • bootsnap/setup.rb attempts to derive the cache directory from caller.

See also bootsnap/setup.rb.

Thoughts?

Show outdated Hide outdated railties/lib/rails/generators/rails/app/templates/Gemfile
@@ -19,6 +19,9 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development
# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', '~> 1.1.0.pre', require: false

This comment has been minimized.

@jeremy

jeremy Jun 12, 2017

Member

Do we need a conservative ~> dep? How about >= 1.1.0.pre

@jeremy

jeremy Jun 12, 2017

Member

Do we need a conservative ~> dep? How about >= 1.1.0.pre

This comment has been minimized.

@burke

burke Jun 12, 2017

Contributor

Sure, updated.

@burke

burke Jun 12, 2017

Contributor

Sure, updated.

@grosser

This comment has been minimized.

Show comment
Hide comment
@grosser

grosser Jun 19, 2017

Contributor

compile_cache_iseq broke code-coverage single_cov ... so had to disable it in test zendesk/samson#2038 ... so would be nice to see an easy escape hatch for this like an ENV var or defined?(Coverage) ... not urgent, would still love to see this merged, but FYI

Contributor

grosser commented Jun 19, 2017

compile_cache_iseq broke code-coverage single_cov ... so had to disable it in test zendesk/samson#2038 ... so would be nice to see an easy escape hatch for this like an ENV var or defined?(Coverage) ... not urgent, would still love to see this merged, but FYI

@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 19, 2017

Contributor

Yeah -- it's unfortunately hard to detect if coverage is currently enabled. Ruby, perplexingly, doesn't provide an API for this. defined?(Coverage) is about as close as we can get. I'll change that.

Contributor

burke commented Jun 19, 2017

Yeah -- it's unfortunately hard to detect if coverage is currently enabled. Ruby, perplexingly, doesn't provide an API for this. defined?(Coverage) is about as close as we can get. I'll change that.

@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 19, 2017

Contributor

Bootsnap bumped to 1.1.0. This fixes @grosser's issue, and a few other things.

IMO this is ready to go.

Contributor

burke commented Jun 19, 2017

Bootsnap bumped to 1.1.0. This fixes @grosser's issue, and a few other things.

IMO this is ready to go.

@guilleiguaran

This comment has been minimized.

Show comment
Hide comment
@guilleiguaran

guilleiguaran Jun 23, 2017

Member

Not sure what can I be doing wrong but adding bootsnap to my app doubles the loading time:

~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  10.34s user 3.74s system 56% cpu 24.864 total
~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  7.22s user 1.69s system 17% cpu 51.722 total

I've tested this a dozen times getting similar numbers every time.

Member

guilleiguaran commented Jun 23, 2017

Not sure what can I be doing wrong but adding bootsnap to my app doubles the loading time:

~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  10.34s user 3.74s system 56% cpu 24.864 total
~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  7.22s user 1.69s system 17% cpu 51.722 total

I've tested this a dozen times getting similar numbers every time.

@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 23, 2017

Contributor

Are you running it in a VM? Is the application source in a shared folder? If so, set bootsnap's cache directory outside of your app, or symlink tmp/cache to some other location on disk.

Contributor

burke commented Jun 23, 2017

Are you running it in a VM? Is the application source in a shared folder? If so, set bootsnap's cache directory outside of your app, or symlink tmp/cache to some other location on disk.

@guilleiguaran

This comment has been minimized.

Show comment
Hide comment
@guilleiguaran

guilleiguaran Jun 23, 2017

Member

Are you running it in a VM? Is the application source in a shared folder?

No and no. I tried removing the tmp/cache folder anyway and creating a symlink under /tmp but I don't have luck yet.

During the first run (when cache isn't created yet) the time is very similar to the normal boot time of the app:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  8.67s user 2.57s system 44% cpu 25.136 total

During subsequent boots the time goes up to 2x the normal time:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  6.42s user 1.42s system 15% cpu 49.072 total
Member

guilleiguaran commented Jun 23, 2017

Are you running it in a VM? Is the application source in a shared folder?

No and no. I tried removing the tmp/cache folder anyway and creating a symlink under /tmp but I don't have luck yet.

During the first run (when cache isn't created yet) the time is very similar to the normal boot time of the app:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  8.67s user 2.57s system 44% cpu 25.136 total

During subsequent boots the time goes up to 2x the normal time:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  6.42s user 1.42s system 15% cpu 49.072 total
@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 23, 2017

Contributor

Very strange, so it sounds like RubyVM::InstructionSequence.load_from_binary is running slower than loading the file directly from source.

What do you get back from:

ruby -v
uname -msr
Contributor

burke commented Jun 23, 2017

Very strange, so it sounds like RubyVM::InstructionSequence.load_from_binary is running slower than loading the file directly from source.

What do you get back from:

ruby -v
uname -msr
@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jun 23, 2017

Contributor

It might make sense to move this to an issue on bootsnap.

Contributor

burke commented Jun 23, 2017

It might make sense to move this to an issue on bootsnap.

@guilleiguaran

This comment has been minimized.

Show comment
Hide comment
Member

guilleiguaran commented Jun 23, 2017

@kaspth

Just one more thing… plus squash your commits and add a changelog entry. Then I think we're good to go! 🤘

@@ -56,6 +56,9 @@ gem "libxml-ruby", platforms: :ruby
# Action View. For testing Erubis handler deprecation.
gem "erubis", "~> 2.7.0", require: false
# for railties app_generator_test
gem "bootsnap", ">= 1.1.0", require: false

This comment has been minimized.

@kaspth

kaspth Jul 15, 2017

Member

I don't like us needing to bundle Bootsnap in the main gemfile for our Railties tests to work. Can't we find out what Gemfile template / gems the Railties generated Rails app uses? @rafaelfranca are you familiar with gems and railties tests?

Or maybe we should just use require "bootsnap/setup" rescue LoadError in config/boot.rb.

@kaspth

kaspth Jul 15, 2017

Member

I don't like us needing to bundle Bootsnap in the main gemfile for our Railties tests to work. Can't we find out what Gemfile template / gems the Railties generated Rails app uses? @rafaelfranca are you familiar with gems and railties tests?

Or maybe we should just use require "bootsnap/setup" rescue LoadError in config/boot.rb.

This comment has been minimized.

@rafaelfranca

rafaelfranca Jul 17, 2017

Member

The application inside the railties test suite uses the Rails main Gemfile. I don't see a problem on adding bootsnap in the main gemfile, we already do to gems like jbuilder, erubis, etc.

@rafaelfranca

rafaelfranca Jul 17, 2017

Member

The application inside the railties test suite uses the Rails main Gemfile. I don't see a problem on adding bootsnap in the main gemfile, we already do to gems like jbuilder, erubis, etc.

Add bootsnap to default Gemfile:
Bootsnap precomputes load path resolution and caches ruby ISeq
and YAML parsing/compilation, reducing application boot time by
approximately 50% on supported configurations.
@burke

This comment has been minimized.

Show comment
Hide comment
@burke

burke Jul 17, 2017

Contributor

Rebased, squashed, and added changelog entry.

I'm not against searching for an alternative solution if the addition to the root Gemfile is unacceptable on balance, but I'm strongly against rescuing LoadError in every downstream project to support not adding a gem to the Rails development Gemfile.

Contributor

burke commented Jul 17, 2017

Rebased, squashed, and added changelog entry.

I'm not against searching for an alternative solution if the addition to the root Gemfile is unacceptable on balance, but I'm strongly against rescuing LoadError in every downstream project to support not adding a gem to the Rails development Gemfile.

@rafaelfranca rafaelfranca merged commit 8acca89 into rails:master Jul 17, 2017

2 checks passed

codeclimate All good!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@rafaelfranca rafaelfranca deleted the Shopify:bootsnap branch Jul 17, 2017

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jul 17, 2017

Member

I was just going to merge this, but @rafaelfranca's too fast for me! 😄

Member

kaspth commented Jul 17, 2017

I was just going to merge this, but @rafaelfranca's too fast for me! 😄

@tarebyte tarebyte referenced this pull request Jul 18, 2017

Merged

Add bootsnap gem #1028

@kirs

This comment has been minimized.

Show comment
Hide comment
@kirs

kirs Jul 23, 2017

Member

🎉 🎉 🎉

Member

kirs commented Jul 23, 2017

🎉 🎉 🎉

@Watson1978 Watson1978 referenced this pull request Dec 16, 2017

Open

Add bootsnap to optimize boot time #11230

4 of 4 tasks complete

prathamesh-sonpatki added a commit to codetriage/codetriage that referenced this pull request Dec 27, 2017

Bump bootsnap to avoid the error on require 'bootsnap/setup'
$ bin/rails test
/Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `require': cannot load such file -- bootsnap/setup (LoadError)
	from /Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `<top (required)>'
	from bin/rails:3:in `require_relative'
	from bin/rails:3:in `<main>'

- Rails uses bootstanp greater than 1.1.0 ~ rails/rails#29313/

prathamesh-sonpatki added a commit to codetriage/codetriage that referenced this pull request Apr 17, 2018

Bump bootsnap to avoid the error on require 'bootsnap/setup'
$ bin/rails test
/Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `require': cannot load such file -- bootsnap/setup (LoadError)
	from /Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `<top (required)>'
	from bin/rails:3:in `require_relative'
	from bin/rails:3:in `<main>'

- Rails uses bootstanp greater than 1.1.0 ~ rails/rails#29313/

@thomasdziedzic-pd thomasdziedzic-pd referenced this pull request Jun 14, 2018

Closed

enable bootsnap #512

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment