New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate_index gets a SystemStackError when the gem set is large #286

Closed
evanphx opened this Issue Feb 23, 2012 · 13 comments

Comments

Projects
None yet
@hackhowtofaq

This comment has been minimized.

Show comment
Hide comment
@hackhowtofaq

hackhowtofaq Mar 8, 2012

gem --version => 1.6.2 doesn't have this issue, just to know.

hackhowtofaq commented Mar 8, 2012

gem --version => 1.6.2 doesn't have this issue, just to know.

@ghost ghost assigned evanphx Mar 9, 2012

@Telmo

This comment has been minimized.

Show comment
Hide comment
@Telmo

Telmo May 17, 2012

running Ruby 1.9.3 and gem 1.8.11 and running into the same issue after running gem mirror

ERROR:  Unable to process /app/copa/atlas/yum/gems/gems/riak-client-0.8.2.gem
(<unknown>): couldn't parse YAML at line 193 column 2 (Psych::SyntaxError)
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:154:in `parse'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:154:in `parse_stream'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:125:in `parse'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:112:in `load'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/specification.rb:488:in `from_yaml'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:195:in `load_gemspec'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:60:in `block in initialize'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_reader.rb:64:in `block in each'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_reader.rb:55:in `loop'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_reader.rb:55:in `each'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:35:in `initialize'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:20:in `new'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:20:in `open'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package.rb:44:in `open'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:62:in `from_io'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:46:in `block in from_file_by_path'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:45:in `open'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:45:in `from_file_by_path'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:347:in `block in map_gems_to_specs'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:340:in `map'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:340:in `map_gems_to_specs'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:129:in `build_indicies'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:456:in `generate_index'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/commands/generate_index_command.rb:118:in `execute'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/command.rb:278:in `invoke'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/command_manager.rb:147:in `process_args'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/command_manager.rb:117:in `run'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/gem_runner.rb:65:in `run'
    /app/interpreters/ruby/1.9.3/bin/gem:21:in `<main>'
/app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:129: stack level too deep (SystemStackError)

Telmo commented May 17, 2012

running Ruby 1.9.3 and gem 1.8.11 and running into the same issue after running gem mirror

ERROR:  Unable to process /app/copa/atlas/yum/gems/gems/riak-client-0.8.2.gem
(<unknown>): couldn't parse YAML at line 193 column 2 (Psych::SyntaxError)
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:154:in `parse'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:154:in `parse_stream'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:125:in `parse'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/psych.rb:112:in `load'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/specification.rb:488:in `from_yaml'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:195:in `load_gemspec'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:60:in `block in initialize'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_reader.rb:64:in `block in each'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_reader.rb:55:in `loop'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_reader.rb:55:in `each'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:35:in `initialize'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:20:in `new'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package/tar_input.rb:20:in `open'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/package.rb:44:in `open'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:62:in `from_io'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:46:in `block in from_file_by_path'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:45:in `open'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/format.rb:45:in `from_file_by_path'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:347:in `block in map_gems_to_specs'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:340:in `map'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:340:in `map_gems_to_specs'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:129:in `build_indicies'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:456:in `generate_index'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/commands/generate_index_command.rb:118:in `execute'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/command.rb:278:in `invoke'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/command_manager.rb:147:in `process_args'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/command_manager.rb:117:in `run'
    /app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/gem_runner.rb:65:in `run'
    /app/interpreters/ruby/1.9.3/bin/gem:21:in `<main>'
/app/interpreters/ruby/1.9.3/lib/ruby/1.9.1/rubygems/indexer.rb:129: stack level too deep (SystemStackError)
@hackhowtofaq

This comment has been minimized.

Show comment
Hide comment
@hackhowtofaq

hackhowtofaq May 21, 2012

gem --version => 1.8.23 same problem

hackhowtofaq commented May 21, 2012

gem --version => 1.8.23 same problem

@haidangwa

This comment has been minimized.

Show comment
Hide comment
@haidangwa

haidangwa Sep 25, 2012

We've been running an internal mirror of rubygems.org behind our firewalls. This problem is blocking us now, because our indexes haven't been updated in a couple of months due to this error. Any updates?

We're running them on RHEL5.5 (Ruby 1.8.7/rubygems 1.8.24) and RHEL6 (ruby 1.9.2/rubygems 1.8.24) servers.

haidangwa commented Sep 25, 2012

We've been running an internal mirror of rubygems.org behind our firewalls. This problem is blocking us now, because our indexes haven't been updated in a couple of months due to this error. Any updates?

We're running them on RHEL5.5 (Ruby 1.8.7/rubygems 1.8.24) and RHEL6 (ruby 1.9.2/rubygems 1.8.24) servers.

@hackhowtofaq

This comment has been minimized.

Show comment
Hide comment
@hackhowtofaq

hackhowtofaq Oct 31, 2012

The repo provided messes up the indexes (haidangwa thank you ;) ) so still there is no way to mirror gems locally!
The only way I am thinking is to downgrade

gem update --system 1.6.2 

and then try the generate_index

@haidangwa

git clone https://github.com/huacnlee/rubygems-mirror

http://www.hackhowtofaq.com/blog/mirror-ruby-gems-locally/

hackhowtofaq commented Oct 31, 2012

The repo provided messes up the indexes (haidangwa thank you ;) ) so still there is no way to mirror gems locally!
The only way I am thinking is to downgrade

gem update --system 1.6.2 

and then try the generate_index

@haidangwa

git clone https://github.com/huacnlee/rubygems-mirror

http://www.hackhowtofaq.com/blog/mirror-ruby-gems-locally/

@drbrain

This comment has been minimized.

Show comment
Hide comment
@drbrain

drbrain Nov 28, 2012

Member

What is the minimum number of gems needed to reproduce this?

Member

drbrain commented Nov 28, 2012

What is the minimum number of gems needed to reproduce this?

@xartron

This comment has been minimized.

Show comment
Hide comment
@xartron

xartron Dec 7, 2012

I was able to build an index with 16,757 gems. I got ambitious and bumped the count up to 80707 and ran # gem generate_index --update -d /srv/rubygems/. After letting it run most of the day and through the night it appeared to stopped processing. I killed the process, removes all of the artifacts and ran a clean # gem generate_index -d /srv/rubygems/ and I got the this error: /usr/local/lib/ruby/1.9.1/gems/indexer.rb:129: stack level too deep (SystemStackError)

Hope this helps.

I pruned the gems directory down to only include gems with a 2010 or newer date. Total number of gem files I have in /srv/rubygems/gems/ is 152393 from 200K+

I'm using ruby 1.9.3p286 with libyaml-0.1.4 on fedora 16 in a vm.

xartron commented Dec 7, 2012

I was able to build an index with 16,757 gems. I got ambitious and bumped the count up to 80707 and ran # gem generate_index --update -d /srv/rubygems/. After letting it run most of the day and through the night it appeared to stopped processing. I killed the process, removes all of the artifacts and ran a clean # gem generate_index -d /srv/rubygems/ and I got the this error: /usr/local/lib/ruby/1.9.1/gems/indexer.rb:129: stack level too deep (SystemStackError)

Hope this helps.

I pruned the gems directory down to only include gems with a 2010 or newer date. Total number of gem files I have in /srv/rubygems/gems/ is 152393 from 200K+

I'm using ruby 1.9.3p286 with libyaml-0.1.4 on fedora 16 in a vm.

@mlartz

This comment has been minimized.

Show comment
Hide comment
@mlartz

mlartz Jan 23, 2013

Looked into this a bit, and figured out that it was because all the gem specs were being splated to Gem::Specification.add_specs. When you splat things, they are all being placed on Ruby's stack, which has a max size.

Here is a quick example with 1.9.3:

def boom *args; end

boom *((1..200000).to_a)
# => SystemStackError: stack level too deep

Here is my first attempt at a fix. Note that the test takes forever (hours) to run, since it creates a ton of gems and tries to index them. I'll see if I can speed it up a bit

mlartz@f7c1def

mlartz commented Jan 23, 2013

Looked into this a bit, and figured out that it was because all the gem specs were being splated to Gem::Specification.add_specs. When you splat things, they are all being placed on Ruby's stack, which has a max size.

Here is a quick example with 1.9.3:

def boom *args; end

boom *((1..200000).to_a)
# => SystemStackError: stack level too deep

Here is my first attempt at a fix. Note that the test takes forever (hours) to run, since it creates a ton of gems and tries to index them. I'll see if I can speed it up a bit

mlartz@f7c1def

@Karunamon

This comment has been minimized.

Show comment
Hide comment
@Karunamon

Karunamon Sep 9, 2013

Still happens as of 1.8.25. Unfortunately the fix in mlartz@f7c1def does not appear to solve the issue.

Karunamon commented Sep 9, 2013

Still happens as of 1.8.25. Unfortunately the fix in mlartz@f7c1def does not appear to solve the issue.

@carlivar

This comment has been minimized.

Show comment
Hide comment
@carlivar

carlivar Dec 4, 2013

I am trying to setup a complete internal mirror of rubygems but this seems to be a blocker.

Is it thus currently impossible to run your own complete mirror?

FYI current complete mirror has about 355k files:

c=0; for i in gems/*.gem ; do ((c++)); done; echo $c
355471

carlivar commented Dec 4, 2013

I am trying to setup a complete internal mirror of rubygems but this seems to be a blocker.

Is it thus currently impossible to run your own complete mirror?

FYI current complete mirror has about 355k files:

c=0; for i in gems/*.gem ; do ((c++)); done; echo $c
355471
@carlivar

This comment has been minimized.

Show comment
Hide comment
@carlivar

carlivar Dec 9, 2013

To follow-up on my previous comment and for anyone that stumbles upon this issue in their attempts to make their own rubygems mirror: I figured out how to do it.

This generate_index command is not what you want for a full mirror. You want a mirror command that synchronizes all of the gems and index data in the first place.

That mirror gem is NOT the mainstream rubygems-mirror under the 'rubygems' github account. This is what threw me off. Instead, you need to use a fork of it:

https://github.com/huacnlee/rubygems-mirror

This fork will retrieve all the data you need, including the quick/ subdirectory that I was missing. It's too bad this behavior isn't merged back into the "regular" rubygems-mirror gem.

carlivar commented Dec 9, 2013

To follow-up on my previous comment and for anyone that stumbles upon this issue in their attempts to make their own rubygems mirror: I figured out how to do it.

This generate_index command is not what you want for a full mirror. You want a mirror command that synchronizes all of the gems and index data in the first place.

That mirror gem is NOT the mainstream rubygems-mirror under the 'rubygems' github account. This is what threw me off. Instead, you need to use a fork of it:

https://github.com/huacnlee/rubygems-mirror

This fork will retrieve all the data you need, including the quick/ subdirectory that I was missing. It's too bad this behavior isn't merged back into the "regular" rubygems-mirror gem.

@duckinator

This comment has been minimized.

Show comment
Hide comment
@duckinator

duckinator Jan 6, 2016

Member

Has this been resolved?

Member

duckinator commented Jan 6, 2016

Has this been resolved?

@copiousfreetime

This comment has been minimized.

Show comment
Hide comment
@copiousfreetime

copiousfreetime Jan 19, 2016

Contributor

I believe this has been resolved. I just tested with snapshot of 21k gems and generate_index, although not the fasted thing in the world, did generate everything just fine.

Contributor

copiousfreetime commented Jan 19, 2016

I believe this has been resolved. I just tested with snapshot of 21k gems and generate_index, although not the fasted thing in the world, did generate everything just fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment