Skip to content

gemspec loaded too many times => slowdown! #1481

Closed
sunaku opened this Issue Oct 14, 2011 · 9 comments

4 participants

@sunaku
sunaku commented Oct 14, 2011

The bundle commands are loading my project's gemspec multiple times, as demonstrated below. Fixing this might speed up bundler's responsiveness and make people (or me, at the very least) happier! Thanks for your consideration.

1. generate a new gem

/tmp> ruby --version
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-linux]

/tmp> gem --version
1.8.11

/tmp> bundle --version
Bundler version 1.0.21

/tmp> bundle gem bugreport
      create  bugreport/Gemfile
      create  bugreport/Rakefile
      create  bugreport/.gitignore
      create  bugreport/bugreport.gemspec
      create  bugreport/lib/bugreport.rb
      create  bugreport/lib/bugreport/version.rb
Initializating git repo in /tmp/bugreport

/tmp> cd bugreport 

/tmp/bugreport> sed -i '$s/.*/  puts nil, caller, nil\n&/' bugreport.gemspec 

2. run bundle for the first time

/tmp/bugreport> bundle    

/usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1346:in `initialize'
/tmp/bugreport/bugreport.gemspec:5:in `new'
/tmp/bugreport/bugreport.gemspec:5:in `<main>'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `rescue in block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:236:in `block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `chdir'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:32:in `gemspec'
/tmp/bugreport/Gemfile:4:in `evaluate'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:7:in `instance_eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:7:in `evaluate'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:17:in `build'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:138:in `definition'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/cli.rb:219:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/task.rb:22:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor.rb:263:in `dispatch'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/base.rb:386:in `start'
/home/sunny/app/rubygems/gems/bundler-1.0.21/bin/bundle:13:in `<top (required)>'
/home/sunny/app/rubygems/bin/bundle:19:in `load'
/home/sunny/app/rubygems/bin/bundle:19:in `<main>'


/usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1346:in `initialize'
/tmp/bugreport/bugreport.gemspec:5:in `new'
/tmp/bugreport/bugreport.gemspec:5:in `<main>'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `rescue in block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:236:in `block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `chdir'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:353:in `block in load_spec_files'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:352:in `each'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:352:in `load_spec_files'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:385:in `local_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:356:in `block in converge_locked_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:345:in `each'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:345:in `converge_locked_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:143:in `resolve'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:90:in `specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:85:in `resolve_remotely!'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:43:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:8:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/cli.rb:219:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/task.rb:22:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor.rb:263:in `dispatch'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/base.rb:386:in `start'
/home/sunny/app/rubygems/gems/bundler-1.0.21/bin/bundle:13:in `<top (required)>'
/home/sunny/app/rubygems/bin/bundle:19:in `load'
/home/sunny/app/rubygems/bin/bundle:19:in `<main>'

Fetching source index for http://rubygems.org/
Using bugreport (0.0.1) from source at /tmp/bugreport 
Using bundler (1.0.21) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

3. run bundle for the second time

/tmp/bugreport> bundle

/usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1346:in `initialize'
/tmp/bugreport/bugreport.gemspec:5:in `new'
/tmp/bugreport/bugreport.gemspec:5:in `<main>'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `rescue in block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:236:in `block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `chdir'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:32:in `gemspec'
/tmp/bugreport/Gemfile:4:in `evaluate'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:7:in `instance_eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:7:in `evaluate'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:17:in `build'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:138:in `definition'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/cli.rb:219:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/task.rb:22:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor.rb:263:in `dispatch'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/base.rb:386:in `start'
/home/sunny/app/rubygems/gems/bundler-1.0.21/bin/bundle:13:in `<top (required)>'
/home/sunny/app/rubygems/bin/bundle:19:in `load'
/home/sunny/app/rubygems/bin/bundle:19:in `<main>'


/usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1346:in `initialize'
/tmp/bugreport/bugreport.gemspec:5:in `new'
/tmp/bugreport/bugreport.gemspec:5:in `<main>'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `rescue in block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:236:in `block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `chdir'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:32:in `gemspec'
/tmp/bugreport/Gemfile:4:in `evaluate'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:7:in `instance_eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/dsl.rb:7:in `evaluate'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:17:in `build'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:32:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:8:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/cli.rb:219:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/task.rb:22:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor.rb:263:in `dispatch'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/base.rb:386:in `start'
/home/sunny/app/rubygems/gems/bundler-1.0.21/bin/bundle:13:in `<top (required)>'
/home/sunny/app/rubygems/bin/bundle:19:in `load'
/home/sunny/app/rubygems/bin/bundle:19:in `<main>'


/usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1346:in `initialize'
/tmp/bugreport/bugreport.gemspec:5:in `new'
/tmp/bugreport/bugreport.gemspec:5:in `<main>'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `rescue in block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:236:in `block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `chdir'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:353:in `block in load_spec_files'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:352:in `each'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:352:in `load_spec_files'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:385:in `local_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:372:in `block in converge_locked_specs'
/usr/lib/ruby/1.9.1/forwardable.rb:182:in `each'
/usr/lib/ruby/1.9.1/forwardable.rb:182:in `each'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:361:in `converge_locked_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:143:in `resolve'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:116:in `missing_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:33:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:8:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/cli.rb:219:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/task.rb:22:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor.rb:263:in `dispatch'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/base.rb:386:in `start'
/home/sunny/app/rubygems/gems/bundler-1.0.21/bin/bundle:13:in `<top (required)>'
/home/sunny/app/rubygems/bin/bundle:19:in `load'
/home/sunny/app/rubygems/bin/bundle:19:in `<main>'


/usr/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:1346:in `initialize'
/tmp/bugreport/bugreport.gemspec:5:in `new'
/tmp/bugreport/bugreport.gemspec:5:in `<main>'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `eval'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:241:in `rescue in block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:236:in `block in load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `chdir'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler.rb:234:in `load_gemspec'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:353:in `block in load_spec_files'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:352:in `each'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:352:in `load_spec_files'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/source.rb:385:in `local_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:372:in `block in converge_locked_specs'
/usr/lib/ruby/1.9.1/forwardable.rb:182:in `each'
/usr/lib/ruby/1.9.1/forwardable.rb:182:in `each'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:361:in `converge_locked_specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:143:in `resolve'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/definition.rb:90:in `specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/environment.rb:27:in `specs'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:49:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/installer.rb:8:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/cli.rb:219:in `install'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/task.rb:22:in `run'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor.rb:263:in `dispatch'
/home/sunny/app/rubygems/gems/bundler-1.0.21/lib/bundler/vendor/thor/base.rb:386:in `start'
/home/sunny/app/rubygems/gems/bundler-1.0.21/bin/bundle:13:in `<top (required)>'
/home/sunny/app/rubygems/bin/bundle:19:in `load'
/home/sunny/app/rubygems/bin/bundle:19:in `<main>'

Using bugreport (0.0.1) from source at . 
Using bundler (1.0.21) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
@koraktor

Good find.

Seems to be true for 1.1rc, too.

@indirect
Bundler member

Unfortunately, the way Bundler works requires that it load all the gem specs and check their dependency requirements more than once. It is most likely possible to cache the loaded gemspecs somewhere, and that seems like a good plan eventually. The vast majority of time isn't spent on that right now, though, and we're targeting slower things first.

@tenderlove

Unfortunately (for me) it loads every gemspec on my system twice, not just gems included in my bundle.

Here is a video.

Here is a gist with reproduction steps.

@indirect
Bundler member

Yes -- gemspecs on your system are included in the pool of "gems that Bundler can try to use to satisfy the bundle", and so they are all read. The double load is because of the attempt to satisfying the bundle purely out of locally available specs, followed by an attempt to satisfy the bundle using local specs plus any available on the remote sources.

@tenderlove

This is actually loading the gemspec files from disk twice. There isn't any reason why each gemspec should actually be read from disk twice (that I can think of). The information hasn't changed since the last time it was read.

@indirect
Bundler member
@tenderlove

This pull request should help mitigate the issue.

Just a little background, the gem method will actually cause rubygems to pull in gemspecs. The binstubs generated by rubygems have the gem method inside. So the bin stub will activate all gemspecs, bundler clears the cache, then rubygems rereads all of them again later. Hope that helps!

@sunaku
sunaku commented Jan 11, 2012

@tenderlove I see that your pull request #1567 was merged. Does this mean we can close this issue?

@tenderlove

Yes, this ticket should be closed. /cc @indirect

@indirect indirect closed this Jan 12, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.