Skip to content

Updating Gems Is Extremely Slow #70

Closed
eonflare opened this Issue Jul 6, 2012 · 6 comments

6 participants

@eonflare
eonflare commented Jul 6, 2012

Hi Tom,

We've been using Geminabox at my company for awhile now, and first and for most thanks for putting this out there. It has been incredibly helpful.

Our issue however is that gems seem to not appear to be properly indexed upon loading them with the gem inabox command. They show up on the page, and they even show up when one does gem search $NAME --source=http://private.server --both.

However, gem install $NAME always fails unless we hit the reindex URL on the private server. However over time that indexing has become slower and slower, to the point where we can't increase the timeout anymore on the request to allow it to succeed (it's currently at 2 minutes).

I tried setting incremental updates to true to see if that would help. Our config.ru looks like this:

require "rubygems"
require "geminabox"

Geminabox.data = "/var/www/geminabox/data" # …or wherever
Geminabox.configure do |config|
  config.set(:incremental_updates, true)
end

run Geminabox

But it seems to not help at all. Any ideas as to what could be wrong? We are currently hosting 789 gems on our gem server.

@tomlea
Gem in a Box member
tomlea commented Aug 21, 2012

I could never get incremental indexing to work reliably. It's still in the back of my mind, but writing up patches for rubygems is bottom of my wish list ;)

Reindexing gems seems to take an impossibly long time using the Gem::Indexer, which is what we are doing now.

As a work around, you could set up a cron job that does a gem generate_index at a regular interval? It is however looking likely that I'll have to make the reindex task asynchronous in the near future. This makes me sad, because of the complexity it adds, but it's all I can think of now.

With the number of gems you're handling, you may wish to consider using http://www.gemfury.com/ and making it someone else's problem!

@tomlea tomlea closed this Aug 21, 2012
@jfoy
jfoy commented Aug 24, 2012

I have just submitted pull request rubygems/rubygems#364 to Rubygems addressing incremental indexing. Our site has over 1600 gems; with this change in place and incremental indexing enabled, an individual gem upload under geminabox takes 0.6 seconds.

@tomlea
Gem in a Box member
tomlea commented Aug 26, 2012

Nice work! I'll see if I can monkey patch for now / for old versions of rubygems in the future.

@tomlea tomlea reopened this Aug 26, 2012
@mcg
mcg commented May 15, 2013

Wondering if this was ever done. We are bumping up against really slow indexing now.

@cromulus
cromulus commented Dec 4, 2013

👍 for speedier indexing. Upstream rubygems merged @jfoy's pull request.

@mereghost

👍

@tomlea tomlea added the Stale label Feb 15, 2016
@tomlea tomlea closed this Feb 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.