Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Updating Gems Is Extremely Slow #70

Open
eonflare opened this Issue · 6 comments

6 participants

@eonflare

Hi Tom,

We've been using Geminabox at my company for awhile now, and first and for most thanks for putting this out there. It has been incredibly helpful.

Our issue however is that gems seem to not appear to be properly indexed upon loading them with the gem inabox command. They show up on the page, and they even show up when one does gem search $NAME --source=http://private.server --both.

However, gem install $NAME always fails unless we hit the reindex URL on the private server. However over time that indexing has become slower and slower, to the point where we can't increase the timeout anymore on the request to allow it to succeed (it's currently at 2 minutes).

I tried setting incremental updates to true to see if that would help. Our config.ru looks like this:

require "rubygems"
require "geminabox"

Geminabox.data = "/var/www/geminabox/data" # …or wherever
Geminabox.configure do |config|
  config.set(:incremental_updates, true)
end

run Geminabox

But it seems to not help at all. Any ideas as to what could be wrong? We are currently hosting 789 gems on our gem server.

@cwninja
Owner

I could never get incremental indexing to work reliably. It's still in the back of my mind, but writing up patches for rubygems is bottom of my wish list ;)

Reindexing gems seems to take an impossibly long time using the Gem::Indexer, which is what we are doing now.

As a work around, you could set up a cron job that does a gem generate_index at a regular interval? It is however looking likely that I'll have to make the reindex task asynchronous in the near future. This makes me sad, because of the complexity it adds, but it's all I can think of now.

With the number of gems you're handling, you may wish to consider using http://www.gemfury.com/ and making it someone else's problem!

@cwninja cwninja closed this
@jfoy

I have just submitted pull request rubygems/rubygems#364 to Rubygems addressing incremental indexing. Our site has over 1600 gems; with this change in place and incremental indexing enabled, an individual gem upload under geminabox takes 0.6 seconds.

@cwninja
Owner

Nice work! I'll see if I can monkey patch for now / for old versions of rubygems in the future.

@cwninja cwninja reopened this
@mcg

Wondering if this was ever done. We are bumping up against really slow indexing now.

@cromulus

:+1: for speedier indexing. Upstream rubygems merged @jfoy's pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.