Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

'gem install' on Rubygems 2.2 / Ruby 2.1 dramatically slower than 2.0/2.0 #766

Closed
mipearson opened this Issue · 10 comments

3 participants

@mipearson

I suspect that this is another case of a latency-specific problem: ie, one that doesn't manifest until you try to use 'gem' from somewhere that's not the US. Notable is the many redirects to mirrors for very small gemspec files. Also noting that 2.2 downloads the gemspecs for java/ming32 versions as well.

Platform is Ubuntu 12.04. Internet connection is a symmetric 10mbit fibre link in Melbourne, Australia.

TL;DR: Ruby 2.0: 1m24s, Ruby 2.1: 5m44s

Ruby 2.0 Install log: https://gist.github.com/mipearson/a662d3dd9d4b8acc9d50
Ruby 2.1 Install log: https://gist.github.com/mipearson/ed78541193b7a24ea7c9

Benchmark log:

$ sudo bash
$ rm -rf /usr/lib/ruby ~/.gem
$ apt-get install ruby2.0
$ gem env

RubyGems Environment:
  - RUBYGEMS VERSION: 2.0.3
  - RUBY VERSION: 2.0.0 (2013-06-27 patchlevel 247) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/lib/ruby/gems/2.0.0
  - RUBY EXECUTABLE: /usr/bin/ruby
  - EXECUTABLE DIRECTORY: /usr/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/lib/ruby/gems/2.0.0
     - /home/mp/.gem/ruby/2.0.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/

$ time gem install --verbose bundler chef --no-ri --no-rdoc > ~/ruby2.0

real  1m24.161s
user  0m9.093s
sys 0m0.692s

$ apt-get remove ruby2.0
$ rm -rf /usr/lib/ruby ~/.gem
$ gem env

  - RUBYGEMS VERSION: 2.2.0
  - RUBY VERSION: 2.1.0 (2013-12-25 patchlevel 0) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/lib/ruby/gems/2.1.0
  - RUBY EXECUTABLE: /usr/bin/ruby
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /home/mp/.gem/specs
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/lib/ruby/gems/2.1.0
     - /home/mp/.gem/ruby/2.1.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
     - /usr/local/rvm/bin

$ time gem install --verbose bundler chef --no-ri --no-rdoc > ~/ruby2.1

real 5m44.150s
user  0m7.564s
sys 0m0.792s    
@mipearson

If you'd like to use the exact same rubies as we used, add http://apt.teg.io (key "http://apt.teg.io/packages@bikeexchange.com.au.gpg.key") to your apt sources.

@mipearson

From a m1.large in us-east:

Ruby 2.1 / Rubygems 2.2: 3m10.086s
Ruby 2.0 / Rubygems 2.0: 1m30s

@mipearson

From a m1.large in us-west-2 (same region as api.rubygems.org):

Ruby 2.1 / Rubygems 2.2: 1m59.330s
Ruby 2.0 / Rubygems 2.0: 1m31s

So, still slower on Ruby 2.1 / Rubygems 2.2 even after removing latency as a factor.

@mipearson mipearson referenced this issue from a commit
@drbrain drbrain Reduce HTTP requests by using BestSet
RubyGems 2.2.0 introduced use of the bundler API, but accidentally this
was not complete.  `gem install` of a gem was left using the marshal API
like 2.1.x and earlier which makes lots of requests.

Now RubyGems uses BestSet inside InstallerSet for remote requests.  This
should speed up install during the dependency fetching phase.

This should also address #762
204d1b9
@drbrain
Owner

I think this was addressed by @204d1b9. With master RubyGems installs at around twice the speed as the 2.0 branch.

I will release RubyGems 2.2.1 today so you can check it, then it can be closed.

@drbrain drbrain referenced this issue from a commit
@drbrain drbrain Add #766 to History d75c51f
@johnl

This patch fixed the problem for me. I can see from the verbose output that it changed from downloading each .rz file for every version of every dependency to using api.rubygems.org.

For what it's worth, it's still hitting api.rubygems.org for each request and then getting redirected to bundler.rubygems.org - with the SSL restablishment each time, this is still slower than it could be (but still way faster than it was :)

@drbrain
Owner

We've got #772 and #773 to further reduce the number of requests made, but they are still under discussion. We may be able to improve the performance of the resolver itself, too.

@drbrain
Owner

Ran gem install mechanize with cache under -rprofile. Here are the results for > 0.1% of time:

https://gist.github.com/drbrain/8446901

@mipearson

Yep, I'm seeing better results, too. Sorry - I thought I'd already closed this. Thanks all :)

@mipearson mipearson closed this
@drbrain
Owner

Heh! I found a way to make it faster still!

@drbrain
Owner

Here's a profile from master. You'll note that the output looks much more sane because I removed the duplicate work the DependencyInstaller performs.

Unfortunately this doesn't seem to be any faster (no difference proven at 95.0% confidence for three runs), probably because network time swamps the extra work RubyGems was doing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.