Lazy load dependencies to reduce memory on startup

[dgynn]

I noticed that hair_trigger was requiring a lot of memory on startup. Using derailed_benchmarks
I tracked that down to the loading of the ruby_parser and ruby2ruby gems.
Since those libraries are only needed by the HairTrigger::MigrationReader it
is possible to use autoload to lazy-load those libraries only when needed.

In a sample case of a Gemfile that only includes activerecord and hairtrigger
the total memory allocated just to load the Gemfile dropped from 53MB to 23MB. At least 6MB of that memory is not able to be garbage collected. The observed process memory dropped from ~25MB to ~16MB.

Here is a process for recreating those performance results...

Create a Gemfile.derailed file in the project root directory

source 'https://rubygems.org'

gem 'derailed_benchmarks', github: 'schneems/derailed_benchmarks'
# Explicitly require active_record before hairtrigger. The gem itself does not auto-require.
gem 'activerecord', require: 'active_record'
# Load the local gem
gem 'hairtrigger', path: '.'

Bundle install the gemfile and execute the bundle:objects and bundle:mem tasks to see the measurements.
This can be run on master and on the branch for comparison.

BUNDLE_GEMFILE=Gemfile.derailed bundle install
BUNDLE_GEMFILE=Gemfile.derailed derailed bundle:objects
BUNDLE_GEMFILE=Gemfile.derailed derailed bundle:mem

Before (master): bundle:objects

Measuring objects created by gems in groups [:default, "production"]
Total allocated: 53073723 bytes (515158 objects)
Total retained:  16405204 bytes (23770 objects)

allocated memory by gem
-----------------------------------
  28503001  ruby_parser-3.7.1
  10112373  activesupport-4.2.4
   7719942  activerecord-4.2.4
   2568679  arel-6.0.3
   1559339  ruby-2.1.6/lib
   1501968  hair_trigger/lib
    271812  activemodel-4.2.4
    217211  ruby2ruby-2.2.0
    209736  i18n-0.7.0
    164964  thread_safe-0.3.5
     90035  json-1.8.3
     82898  bundler-1.9.7
     55860  tzinfo-1.2.2
     15865  sexp_processor-4.6.0
        40  derailed_benchmarks-eaadc3b055c3

retained memory by gem
-----------------------------------
   6457767  ruby_parser-3.7.1
   3641440  activerecord-4.2.4
   3376225  activesupport-4.2.4
    905738  arel-6.0.3
    851115  hair_trigger/lib
    634583  ruby-2.1.6/lib
    125036  ruby2ruby-2.2.0
    116774  activemodel-4.2.4
     81883  i18n-0.7.0
     64343  thread_safe-0.3.5
     52988  tzinfo-1.2.2
     52709  bundler-1.9.7
     29538  json-1.8.3
     15065  sexp_processor-4.6.0

After (pull request): bundle:objects

Measuring objects created by gems in groups [:default, "production"]
Total allocated: 23382817 bytes (128503 objects)
Total retained:  9209065 bytes (16538 objects)

allocated memory by gem
-----------------------------------
  10112893  activesupport-4.2.4
   7720222  activerecord-4.2.4
   2568679  arel-6.0.3
   1551135  ruby-2.1.6/lib
    554863  hair_trigger/lib
    271812  activemodel-4.2.4
    209736  i18n-0.7.0
    164964  thread_safe-0.3.5
     89715  json-1.8.3
     82898  bundler-1.9.7
     55860  tzinfo-1.2.2
        40  derailed_benchmarks-eaadc3b055c3

retained memory by gem
-----------------------------------
   3642532  activerecord-4.2.4
   3376509  activesupport-4.2.4
    905738  arel-6.0.3
    629049  ruby-2.1.6/lib
    256726  hair_trigger/lib
    116774  activemodel-4.2.4
     81883  i18n-0.7.0
     64343  thread_safe-0.3.5
     52988  tzinfo-1.2.2
     52985  bundler-1.9.7
     29538  json-1.8.3

Before (master): 3 runs of bundle:mem - the results vary

TOP: 24.0352 MiB
  hairtrigger: 20.5234 MiB
    [DETAILS TRIMMED OUT FOR EACH OF THESE]
  active_record: 3.4688 MiB
TOP: 25.1172 MiB
  hairtrigger: 21.3281 MiB
  active_record: 3.75 MiB
TOP: 25.3047 MiB
  hairtrigger: 20.875 MiB
  active_record: 4.3516 MiB

After (pull request): 3 runs of bundle:mem

TOP: 16.7266 MiB
  hairtrigger: 12.3633 MiB
  active_record: 4.293 MiB
TOP: 17.9961 MiB
  hairtrigger: 12.6094 MiB
  active_record: 5.2773 MiB
TOP: 14.5156 MiB
  hairtrigger: 10.8945 MiB
  active_record: 3.5781 MiB

[jenseng]

nice detective work 👍