Add bootsnap to default Gemfile #29313

burke · 2017-06-01T18:43:17Z

This adds bootsnap to newly-generated projects, which generally reduces application boot times by over 50%.

I heard through the grapevine that people would be open to this.

Early releases of bootsnap only supported macOS, but recent versions now additionally support linux and (at least in theory) most *BSDs. Windows support is still not implemented.

This change is small, but not trivial, and it's been a long time since I've been in this codebase. I'm looking for guidance on how best to get this change documented, tested, and merged.

cc @rafaelfranca

rails-bot · 2017-06-01T18:43:20Z

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @eileencodes (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

matthewd · 2017-06-01T18:59:30Z

railties/lib/rails/generators/rails/app/templates/config/boot.rb

+    load_path_cache:      true,
+    autoload_paths_cache: true,
+    compile_cache_iseq:   true,
+    compile_cache_yaml:   true


This is way too much / too specific configuration.

Maybe we should follow bundlers lead:

require "bundler/setup" # Set up gems listed in the Gemfile require "bootsnap/setup" # Caches and compiles gems for faster boot.

Which then just includes the above conventionalized config.

Could also be require "bootsnap/railtie" or Rails could just set up bootsnap automatically after executing boot.rb if the gem is present, e.g.:

# Somewhere in the Rails bowels: begin require "bootsnap" Bootsnap.setup(…) rescue LoadError end

matthewd · 2017-06-01T19:00:30Z

railties/lib/rails/generators/rails/app/templates/config/boot.rb

@@ -1,3 +1,17 @@
 ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)

 require 'bundler/setup' # Set up gems listed in the Gemfile.
+
+if RUBY_ENGINE == 'ruby' && RUBY_PLATFORM =~ /darwin|linux|bsd/


We should always require bootsnap and let it decide whether it supports the current environment.

matthewd · 2017-06-01T19:04:07Z

railties/lib/rails/generators/rails/app/templates/Gemfile

@@ -25,6 +25,10 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }

 <%- end -%>
 <% if RUBY_ENGINE == 'ruby' -%>
+
+# Reduces boot times through caching; configured in config/boot.rb
+gem 'bootsnap', '~> 1.0.0', platforms: [:mri]


Can we loosen this version constraint a bit?

Can bootsnap be installed on all platforms, even if it won't Do Cool Stuff?

bootsnap only supports Ruby 2.3.0 and higher, but Rails still supports Ruby 2.2.
https://github.com/Shopify/bootsnap/blob/master/bootsnap.gemspec#L26

Should we check the Ruby version?

@burke could bootsnap allow installation on 2.2, and just operate in a degraded mode? AIUI path caching would still work -- it's the iseq cache that is strictly 2.3+, right?

yivo · 2017-06-04T08:19:27Z

This is not necessary thing for most Rails apps. Things must be kept as simple as possible. In future this change may bring some bugs. If people need this it is better to create separate official gem bootsnap-rails which is officially tested and ready for Rails. Merging everything is a hell.

burke · 2017-06-05T15:00:47Z

Ok -- I don't have any time this week to revise this, but here's what I'm going to do:

Get bootsnap to install correctly even on platforms where it can't use the native extension;
Have it understand which features can be enabled based on the current platform;
Add a bootsnap/setup requirable that sets up the maximal set of available features based on the environment, and assuming ActiveSupport will be in use.

The method used here will still be available, but bootsnap/setup will wrap it.

rafaelfranca · 2017-06-05T21:00:15Z

@yivo we are not merging anything. It is not even being added as a dependency of a framework. It is being added to the generated Gemfile and if people don't want they can just remove. Rails give people sane defaults so they don't need to spend time thinking if they want to use something or not.

@burke seems like a good plan 👍

burke · 2017-06-12T19:47:23Z

I've updated this PR to use bootsnap-1.1.0.pre. I'll update again to 1.1.0 after some more testing/feedback.

Changes:

Added a jruby version that doesn't include the native extension and doesn't support compile caching;
config/boot.rb now just requires bootsnap/setup;
require is no longer conditional on platform;
bootsnap/setup.rb enables whichever features are available given the current environment;
bootsnap/setup.rb attempts to derive the cache directory from caller.

See also bootsnap/setup.rb.

Thoughts?

jeremy · 2017-06-12T20:02:50Z

railties/lib/rails/generators/rails/app/templates/Gemfile

@@ -19,6 +19,9 @@ git_source(:github) { |repo| "https://github.com/#{repo}.git" }
 # Use Capistrano for deployment
 # gem 'capistrano-rails', group: :development

+# Reduces boot times through caching; required in config/boot.rb
+gem 'bootsnap', '~> 1.1.0.pre', require: false


Do we need a conservative ~> dep? How about >= 1.1.0.pre

Sure, updated.

grosser · 2017-06-19T04:21:49Z

compile_cache_iseq broke code-coverage single_cov ... so had to disable it in test zendesk/samson#2038 ... so would be nice to see an easy escape hatch for this like an ENV var or defined?(Coverage) ... not urgent, would still love to see this merged, but FYI

burke · 2017-06-19T16:44:59Z

Yeah -- it's unfortunately hard to detect if coverage is currently enabled. Ruby, perplexingly, doesn't provide an API for this. defined?(Coverage) is about as close as we can get. I'll change that.

burke · 2017-06-19T20:13:45Z

Bootsnap bumped to 1.1.0. This fixes @grosser's issue, and a few other things.

IMO this is ready to go.

guilleiguaran · 2017-06-23T00:11:50Z

Not sure what can I be doing wrong but adding bootsnap to my app doubles the loading time:

~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  10.34s user 3.74s system 56% cpu 24.864 total

~/src/app[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  7.22s user 1.69s system 17% cpu 51.722 total

I've tested this a dozen times getting similar numbers every time.

burke · 2017-06-23T17:12:23Z

Are you running it in a VM? Is the application source in a shared folder? If so, set bootsnap's cache directory outside of your app, or symlink tmp/cache to some other location on disk.

guilleiguaran · 2017-06-23T19:08:06Z

Are you running it in a VM? Is the application source in a shared folder?

No and no. I tried removing the tmp/cache folder anyway and creating a symlink under /tmp but I don't have luck yet.

During the first run (when cache isn't created yet) the time is very similar to the normal boot time of the app:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  8.67s user 2.57s system 44% cpu 25.136 total

During subsequent boots the time goes up to 2x the normal time:

~/src/cookpad[master] % time bin/rails runner "puts Rails.env"
development
bin/rails runner "puts Rails.env"  6.42s user 1.42s system 15% cpu 49.072 total

burke · 2017-06-23T19:13:32Z

Very strange, so it sounds like RubyVM::InstructionSequence.load_from_binary is running slower than loading the file directly from source.

What do you get back from:

ruby -v
uname -msr

burke · 2017-06-23T19:14:59Z

It might make sense to move this to an issue on bootsnap.

guilleiguaran · 2017-06-23T19:20:02Z

@burke sure!! Shopify/bootsnap#68

kaspth

Just one more thing… plus squash your commits and add a changelog entry. Then I think we're good to go! 🤘

kaspth · 2017-07-15T19:06:45Z

Gemfile

@@ -56,6 +56,9 @@ gem "libxml-ruby", platforms: :ruby
 # Action View. For testing Erubis handler deprecation.
 gem "erubis", "~> 2.7.0", require: false

+# for railties app_generator_test
+gem "bootsnap", ">= 1.1.0", require: false


I don't like us needing to bundle Bootsnap in the main gemfile for our Railties tests to work. Can't we find out what Gemfile template / gems the Railties generated Rails app uses? @rafaelfranca are you familiar with gems and railties tests?

Or maybe we should just use require "bootsnap/setup" rescue LoadError in config/boot.rb.

The application inside the railties test suite uses the Rails main Gemfile. I don't see a problem on adding bootsnap in the main gemfile, we already do to gems like jbuilder, erubis, etc.

Bootsnap precomputes load path resolution and caches ruby ISeq and YAML parsing/compilation, reducing application boot time by approximately 50% on supported configurations.

burke · 2017-07-17T17:19:48Z

Rebased, squashed, and added changelog entry.

I'm not against searching for an alternative solution if the addition to the root Gemfile is unacceptable on balance, but I'm strongly against rescuing LoadError in every downstream project to support not adding a gem to the Rails development Gemfile.

kaspth · 2017-07-17T18:00:49Z

I was just going to merge this, but @rafaelfranca's too fast for me! 😄

kirs · 2017-07-23T15:26:08Z

🎉 🎉 🎉

$ bin/rails test /Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `require': cannot load such file -- bootsnap/setup (LoadError) from /Users/prathamesh/Projects/sources/codetriage/config/boot.rb:4:in `<top (required)>' from bin/rails:3:in `require_relative' from bin/rails:3:in `<main>' - Rails uses bootstanp greater than 1.1.0 ~ rails/rails#29313

### Description Rails 5.2 ships with a new default gem called `bootsnap`, which [speeds up Rails boot time by 50%](rails/rails#29313). I've added it here to see if it'll work for Caseflow. Faster boot times mean a shorter development loop, less CI contention, and faster boot/restart during deploys. [How it works](https://github.com/Shopify/bootsnap#how-does-this-work) tl;dr avoiding slow scanning of $LOAD_PATH and caching of yaml files (both are slow parts of Rails boot) Results for caseflow, locally testing a couple results of `time bundle exec rails s` until the 'Ctrl-C' prompt appears: before: real 0m8.811s real 0m8.325s after: real 0m3.939s real 0m4.098s ~53% decrease in bootup time, seems legit. ### Potential risks * This gem does affect how files are loaded during app boot, so there is some risk that app boot in a non-dev environment goes awry. A test deploy should alleviate those concerns. * It's possible that bootsnap negatively affects development by caching filepaths too aggressively (see 'stable entries' under https://github.com/Shopify/bootsnap#path-pre-scanning). I don't know of a use case where those paths need to be uncached. I'm adding this gem to every group, since there's nothing development-specific about it and it seems better to both take advantage of speed gains in other envs and run as similar of a setup in dev and prod as possible. Shopify uses this gem in every environment and it's the new Rails default. Tangentially, I'm also curious if we've tried upgrading to Rails 5.2 outright. ### Acceptance Criteria - [x] Tests pass ### Testing Plan In addition to working with this gem enabled locally for development, I'd like to do a test deploy to an AWS environment and restart to confirm that both of these succeed.

rails-bot assigned eileencodes Jun 1, 2017

burke mentioned this pull request Jun 1, 2017

Contributing speed improvements to Ruby? Shopify/bootsnap#44

Closed

matthewd reviewed Jun 1, 2017

View reviewed changes

eileencodes assigned rafaelfranca and unassigned eileencodes Jun 2, 2017

dankohn mentioned this pull request Jun 10, 2017

Gem diet / reducing memory use coreinfrastructure/best-practices-badge#793

Closed

burke force-pushed the bootsnap branch 3 times, most recently from cf1b495 to 71719bb Compare June 12, 2017 19:41

burke force-pushed the bootsnap branch from 71719bb to d141404 Compare June 12, 2017 19:50

jeremy reviewed Jun 12, 2017

View reviewed changes

burke force-pushed the bootsnap branch 4 times, most recently from 7fa15d0 to 78a4d76 Compare June 14, 2017 15:20

burke force-pushed the bootsnap branch from fdd4d35 to 5446708 Compare June 19, 2017 20:14

guilleiguaran mentioned this pull request Jun 23, 2017

Getting 2x boot times using bootsnap Shopify/bootsnap#68

Closed

kaspth reviewed Jul 15, 2017

View reviewed changes

burke force-pushed the bootsnap branch from 5446708 to e322458 Compare July 17, 2017 17:15

Add bootsnap to default Gemfile:

0312a5c

Bootsnap precomputes load path resolution and caches ruby ISeq and YAML parsing/compilation, reducing application boot time by approximately 50% on supported configurations.

burke force-pushed the bootsnap branch from e322458 to 0312a5c Compare July 17, 2017 17:17

rafaelfranca merged commit 8acca89 into rails:master Jul 17, 2017

rafaelfranca deleted the bootsnap branch July 17, 2017 18:00

tarebyte mentioned this pull request Jul 18, 2017

Add bootsnap gem github-education-resources/classroom#1028

Merged

Aqualon mentioned this pull request Jul 27, 2017

Cannot define multiple 'included' blocks for a Concern Shopify/bootsnap#75

Closed

Watson1978 mentioned this pull request Dec 16, 2017

Add bootsnap to optimize boot time fastlane/fastlane#11230

Closed

4 tasks

thomasdziedzic-pd mentioned this pull request Jun 14, 2018

enable bootsnap lobsters/lobsters#512

Closed

kencheeto mentioned this pull request Sep 18, 2018

add bootsnap gem for faster rails boot department-of-veterans-affairs/caseflow#7042

Merged

1 task

snickell mentioned this pull request May 2, 2023

Use bootsnap gem to decrease dashboard boot time code-dot-org/code-dot-org#51636

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bootsnap to default Gemfile #29313

Add bootsnap to default Gemfile #29313

burke commented Jun 1, 2017

rails-bot commented Jun 1, 2017

matthewd Jun 1, 2017

kaspth Jun 1, 2017

kaspth Jun 1, 2017

matthewd Jun 1, 2017

matthewd Jun 1, 2017

y-yagi Jun 4, 2017

matthewd Jun 4, 2017

yivo commented Jun 4, 2017

burke commented Jun 5, 2017

rafaelfranca commented Jun 5, 2017

burke commented Jun 12, 2017 •

edited

Loading

jeremy Jun 12, 2017

burke Jun 12, 2017

grosser commented Jun 19, 2017 •

edited

Loading

burke commented Jun 19, 2017

burke commented Jun 19, 2017 •

edited

Loading

guilleiguaran commented Jun 23, 2017

burke commented Jun 23, 2017

guilleiguaran commented Jun 23, 2017

burke commented Jun 23, 2017

burke commented Jun 23, 2017

guilleiguaran commented Jun 23, 2017

kaspth left a comment

kaspth Jul 15, 2017

rafaelfranca Jul 17, 2017

burke commented Jul 17, 2017

kaspth commented Jul 17, 2017

kirs commented Jul 23, 2017

Add bootsnap to default Gemfile #29313

Add bootsnap to default Gemfile #29313

Conversation

burke commented Jun 1, 2017

rails-bot commented Jun 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yivo commented Jun 4, 2017

burke commented Jun 5, 2017

rafaelfranca commented Jun 5, 2017

burke commented Jun 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grosser commented Jun 19, 2017 • edited Loading

burke commented Jun 19, 2017

burke commented Jun 19, 2017 • edited Loading

guilleiguaran commented Jun 23, 2017

burke commented Jun 23, 2017

guilleiguaran commented Jun 23, 2017

burke commented Jun 23, 2017

burke commented Jun 23, 2017

guilleiguaran commented Jun 23, 2017

kaspth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

burke commented Jul 17, 2017

kaspth commented Jul 17, 2017

kirs commented Jul 23, 2017

burke commented Jun 12, 2017 •

edited

Loading

grosser commented Jun 19, 2017 •

edited

Loading

burke commented Jun 19, 2017 •

edited

Loading