Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bootsnap to default Gemfile #29313

Merged
merged 1 commit into from Jul 17, 2017
Merged

Add bootsnap to default Gemfile #29313

merged 1 commit into from Jul 17, 2017

Conversation

@burke
Copy link
Contributor

burke commented Jun 1, 2017

This adds bootsnap to newly-generated projects, which generally reduces application boot times by over 50%.


I heard through the grapevine that people would be open to this.

Early releases of bootsnap only supported macOS, but recent versions now additionally support linux and (at least in theory) most *BSDs. Windows support is still not implemented.

This change is small, but not trivial, and it's been a long time since I've been in this codebase. I'm looking for guidance on how best to get this change documented, tested, and merged.

cc @rafaelfranca

@rails-bot
Copy link

rails-bot commented Jun 1, 2017

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @eileencodes (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

railties/lib/rails/generators/rails/app/templates/config/boot.rb Outdated
load_path_cache: true,
autoload_paths_cache: true,
compile_cache_iseq: true,
compile_cache_yaml: true

This comment has been minimized.

Copy link
@matthewd

matthewd Jun 1, 2017

Member

This is way too much / too specific configuration.

This comment has been minimized.

Copy link
@kaspth

kaspth Jun 1, 2017

Member

Maybe we should follow bundlers lead:

require "bundler/setup"    # Set up gems listed in the Gemfile
require "bootsnap/setup" # Caches and compiles gems for faster boot.

Which then just includes the above conventionalized config.

This comment has been minimized.

Copy link
@kaspth

kaspth Jun 1, 2017

Member

Could also be require "bootsnap/railtie" or Rails could just set up bootsnap automatically after executing boot.rb if the gem is present, e.g.:

# Somewhere in the Rails bowels:
begin
  require "bootsnap"

  Bootsnap.setup()
rescue LoadError
end
railties/lib/rails/generators/rails/app/templates/config/boot.rb Outdated
@@ -1,3 +1,17 @@
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)

require 'bundler/setup' # Set up gems listed in the Gemfile.

if RUBY_ENGINE == 'ruby' && RUBY_PLATFORM =~ /darwin|linux|bsd/

This comment has been minimized.

Copy link
@matthewd

matthewd Jun 1, 2017

Member

We should always require bootsnap and let it decide whether it supports the current environment.

railties/lib/rails/generators/rails/app/templates/Gemfile Outdated
@@ -25,6 +25,10 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }

<%- end -%>
<% if RUBY_ENGINE == 'ruby' -%>

# Reduces boot times through caching; configured in config/boot.rb
gem 'bootsnap', '~> 1.0.0', platforms: [:mri]

This comment has been minimized.

Copy link
@matthewd

matthewd Jun 1, 2017

Member

Can we loosen this version constraint a bit?

Can bootsnap be installed on all platforms, even if it won't Do Cool Stuff?

This comment has been minimized.

Copy link
@y-yagi

y-yagi Jun 4, 2017

Member

bootsnap only supports Ruby 2.3.0 and higher, but Rails still supports Ruby 2.2.
https://github.com/Shopify/bootsnap/blob/master/bootsnap.gemspec#L26

Should we check the Ruby version?

This comment has been minimized.

Copy link
@matthewd

matthewd Jun 4, 2017

Member

@burke could bootsnap allow installation on 2.2, and just operate in a degraded mode? AIUI path caching would still work -- it's the iseq cache that is strictly 2.3+, right?

@yivo
Copy link

yivo commented Jun 4, 2017

This is not necessary thing for most Rails apps. Things must be kept as simple as possible. In future this change may bring some bugs. If people need this it is better to create separate official gem bootsnap-rails which is officially tested and ready for Rails. Merging everything is a hell.

@burke
Copy link
Contributor Author

burke commented Jun 5, 2017

Ok -- I don't have any time this week to revise this, but here's what I'm going to do:

  1. Get bootsnap to install correctly even on platforms where it can't use the native extension;
  2. Have it understand which features can be enabled based on the current platform;
  3. Add a bootsnap/setup requirable that sets up the maximal set of available features based on the environment, and assuming ActiveSupport will be in use.

The method used here will still be available, but bootsnap/setup will wrap it.

@rafaelfranca
Copy link
Member

rafaelfranca commented Jun 5, 2017

@yivo we are not merging anything. It is not even being added as a dependency of a framework. It is being added to the generated Gemfile and if people don't want they can just remove. Rails give people sane defaults so they don't need to spend time thinking if they want to use something or not.

@burke seems like a good plan 👍

@burke burke force-pushed the Shopify:bootsnap branch 3 times, most recently Jun 12, 2017
@burke
Copy link
Contributor Author

burke commented Jun 12, 2017

I've updated this PR to use bootsnap-1.1.0.pre. I'll update again to 1.1.0 after some more testing/feedback.

Changes:

  • Added a jruby version that doesn't include the native extension and doesn't support compile caching;
  • config/boot.rb now just requires bootsnap/setup;
  • require is no longer conditional on platform;
  • bootsnap/setup.rb enables whichever features are available given the current environment;
  • bootsnap/setup.rb attempts to derive the cache directory from caller.

See also bootsnap/setup.rb.

Thoughts?

@burke burke force-pushed the Shopify:bootsnap branch Jun 12, 2017
railties/lib/rails/generators/rails/app/templates/Gemfile Outdated
@@ -19,6 +19,9 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', '~> 1.1.0.pre', require: false

This comment has been minimized.

Copy link
@jeremy

jeremy Jun 12, 2017

Member

Do we need a conservative ~> dep? How about >= 1.1.0.pre

This comment has been minimized.

Copy link
@burke

burke Jun 12, 2017

Author Contributor

Sure, updated.

@burke burke force-pushed the Shopify:bootsnap branch 4 times, most recently Jun 13, 2017
@grosser
Copy link
Contributor

grosser commented Jun 19, 2017

compile_cache_iseq broke code-coverage single_cov ... so had to disable it in test zendesk/samson#2038 ... so would be nice to see an easy escape hatch for this like an ENV var or defined?(Coverage) ... not urgent, would still love to see this merged, but FYI

@burke
Copy link
Contributor Author

burke commented Jun 19, 2017

Yeah -- it's unfortunately hard to detect if coverage is currently enabled. Ruby, perplexingly, doesn't provide an API for this. defined?(Coverage) is about as close as we can get. I'll change that.

@burke
Copy link
Contributor Author

burke commented Jun 19, 2017

Bootsnap bumped to 1.1.0. This fixes @grosser's issue, and a few other things.

IMO this is ready to go.

@burke burke force-pushed the Shopify:bootsnap branch Jun 19, 2017
@guilleiguaran
Copy link
Member

guilleiguaran commented Jun 23, 2017

Not sure what can I be doing wrong but adding bootsnap to my app doubles the loading time:

~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  10.34s user 3.74s system 56% cpu 24.864 total
~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  7.22s user 1.69s system 17% cpu 51.722 total

I've tested this a dozen times getting similar numbers every time.

@burke
Copy link
Contributor Author

burke commented Jun 23, 2017

Are you running it in a VM? Is the application source in a shared folder? If so, set bootsnap's cache directory outside of your app, or symlink tmp/cache to some other location on disk.

@guilleiguaran
Copy link
Member

guilleiguaran commented Jun 23, 2017

Are you running it in a VM? Is the application source in a shared folder?

No and no. I tried removing the tmp/cache folder anyway and creating a symlink under /tmp but I don't have luck yet.

During the first run (when cache isn't created yet) the time is very similar to the normal boot time of the app:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  8.67s user 2.57s system 44% cpu 25.136 total

During subsequent boots the time goes up to 2x the normal time:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  6.42s user 1.42s system 15% cpu 49.072 total
@burke
Copy link
Contributor Author

burke commented Jun 23, 2017

Very strange, so it sounds like RubyVM::InstructionSequence.load_from_binary is running slower than loading the file directly from source.

What do you get back from:

ruby -v
uname -msr
@burke
Copy link
Contributor Author

burke commented Jun 23, 2017

It might make sense to move this to an issue on bootsnap.

@guilleiguaran
Copy link
Member

guilleiguaran commented Jun 23, 2017

Copy link
Member

kaspth left a comment

Just one more thing… plus squash your commits and add a changelog entry. Then I think we're good to go! 🤘

@@ -56,6 +56,9 @@ gem "libxml-ruby", platforms: :ruby
# Action View. For testing Erubis handler deprecation.
gem "erubis", "~> 2.7.0", require: false

# for railties app_generator_test
gem "bootsnap", ">= 1.1.0", require: false

This comment has been minimized.

Copy link
@kaspth

kaspth Jul 15, 2017

Member

I don't like us needing to bundle Bootsnap in the main gemfile for our Railties tests to work. Can't we find out what Gemfile template / gems the Railties generated Rails app uses? @rafaelfranca are you familiar with gems and railties tests?

Or maybe we should just use require "bootsnap/setup" rescue LoadError in config/boot.rb.

This comment has been minimized.

Copy link
@rafaelfranca

rafaelfranca Jul 17, 2017

Member

The application inside the railties test suite uses the Rails main Gemfile. I don't see a problem on adding bootsnap in the main gemfile, we already do to gems like jbuilder, erubis, etc.

@burke burke force-pushed the Shopify:bootsnap branch Jul 17, 2017
Bootsnap precomputes load path resolution and caches ruby ISeq
and YAML parsing/compilation, reducing application boot time by
approximately 50% on supported configurations.
@burke burke force-pushed the Shopify:bootsnap branch to 0312a5c Jul 17, 2017
@burke
Copy link
Contributor Author

burke commented Jul 17, 2017

Rebased, squashed, and added changelog entry.

I'm not against searching for an alternative solution if the addition to the root Gemfile is unacceptable on balance, but I'm strongly against rescuing LoadError in every downstream project to support not adding a gem to the Rails development Gemfile.

@rafaelfranca rafaelfranca merged commit 8acca89 into rails:master Jul 17, 2017
2 checks passed
2 checks passed
codeclimate All good!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@rafaelfranca rafaelfranca deleted the Shopify:bootsnap branch Jul 17, 2017
@kaspth
Copy link
Member

kaspth commented Jul 17, 2017

I was just going to merge this, but @rafaelfranca's too fast for me! 😄

@kirs
Copy link
Member

kirs commented Jul 23, 2017

🎉 🎉 🎉

@Watson1978 Watson1978 mentioned this pull request Dec 16, 2017
4 of 4 tasks complete
prathamesh-sonpatki added a commit to codetriage/CodeTriage that referenced this pull request Dec 27, 2017
$ bin/rails test
/Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `require': cannot load such file -- bootsnap/setup (LoadError)
	from /Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `<top (required)>'
	from bin/rails:3:in `require_relative'
	from bin/rails:3:in `<main>'

- Rails uses bootstanp greater than 1.1.0 ~ rails/rails#29313
prathamesh-sonpatki added a commit to codetriage/CodeTriage that referenced this pull request Apr 17, 2018
$ bin/rails test
/Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `require': cannot load such file -- bootsnap/setup (LoadError)
	from /Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `<top (required)>'
	from bin/rails:3:in `require_relative'
	from bin/rails:3:in `<main>'

- Rails uses bootstanp greater than 1.1.0 ~ rails/rails#29313
va-bot added a commit to department-of-veterans-affairs/caseflow that referenced this pull request Sep 20, 2018
### Description
Rails 5.2 ships with a new default gem called `bootsnap`, which [speeds up Rails boot time by 50%](rails/rails#29313). I've added it here to see if it'll work for Caseflow. Faster boot times mean a shorter development loop, less CI contention, and faster boot/restart during deploys.

[How it works](https://github.com/Shopify/bootsnap#how-does-this-work)
tl;dr avoiding slow scanning of $LOAD_PATH and caching of yaml files (both are slow parts of Rails boot)

Results for caseflow, locally testing a couple results of `time bundle exec rails s` until the 'Ctrl-C' prompt appears:

before:
real  0m8.811s
real  0m8.325s

after:
real  0m3.939s
real  0m4.098s

~53% decrease in bootup time, seems legit.

### Potential risks
* This gem does affect how files are loaded during app boot, so there is some risk that app boot in a non-dev environment goes awry. A test deploy should alleviate those concerns.
* It's possible that bootsnap negatively affects development by caching filepaths too aggressively (see 'stable entries' under https://github.com/Shopify/bootsnap#path-pre-scanning). I don't know of a use case where those paths need to be uncached.

I'm adding this gem to every group, since there's nothing development-specific about it and it seems better to both take advantage of speed gains in other envs and run as similar of a setup in dev and prod as possible. Shopify uses this gem in every environment and it's the new Rails default.

Tangentially, I'm also curious if we've tried upgrading to Rails 5.2 outright.

### Acceptance Criteria
- [x] Tests pass

### Testing Plan
In addition to working with this gem enabled locally for development, I'd like to do a test deploy to an AWS environment and restart to confirm that both of these succeed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.