New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"invalid gemspec" with old version gem on bundler 1.1 #1715

Closed
tomykaira opened this Issue Mar 10, 2012 · 22 comments

Comments

Projects
None yet
5 participants
@tomykaira
Contributor

tomykaira commented Mar 10, 2012

Gemfile:

source :rubygems

gem 'sinatra'
gem 'milkode', '0.4.0'

In bundler 1.1, bundle fails with the following error.

Unfortunately, the gem milkode (0.2.3) has an invalid gemspec. As a result, Bundler cannot install this Gemfile. Please ask the gem author to yank the bad version to fix this issue. For more information, see http://bit.ly/syck-defaultkey.

But no error with bundler 1.0.

I'm not sure what is the cause of this, but it seems to be a bug for me.

@rohit

This comment has been minimized.

Show comment
Hide comment
@rohit

rohit Mar 12, 2012

Contributor

This is not a bug in Bundler but a problem with Rubygems.org ― please read the link provided in the error message => http://blog.rubygems.org/2011/08/31/shaving-the-yaml-yak.html

Contributor

rohit commented Mar 12, 2012

This is not a bug in Bundler but a problem with Rubygems.org ― please read the link provided in the error message => http://blog.rubygems.org/2011/08/31/shaving-the-yaml-yak.html

@tomykaira

This comment has been minimized.

Show comment
Hide comment
@tomykaira

tomykaira Mar 12, 2012

Contributor

@rohit, Thank you for response.

Sure. I agree that the main cause is in Rubygems.org.

I am sorry that my question was obscure.

I mean, "Why bundler 1.1 references an old gemspec?"
The latest version of milkode was 0.4.0 (when I tested, it's now 0.5.0), but bundler reports the error with milkode 0.2.3.

I know little about bundler, but I heard that the mechanism of dependency resolution was changed in 1.1.
I suspect this change made such buggy behavior.

Contributor

tomykaira commented Mar 12, 2012

@rohit, Thank you for response.

Sure. I agree that the main cause is in Rubygems.org.

I am sorry that my question was obscure.

I mean, "Why bundler 1.1 references an old gemspec?"
The latest version of milkode was 0.4.0 (when I tested, it's now 0.5.0), but bundler reports the error with milkode 0.2.3.

I know little about bundler, but I heard that the mechanism of dependency resolution was changed in 1.1.
I suspect this change made such buggy behavior.

@rohit

This comment has been minimized.

Show comment
Hide comment
@rohit

rohit Mar 12, 2012

Contributor

Can you test again with both Bundler 1.0.x and Bundler 1.1 ― by my guesstimate this has to do with your version of Rubygems. I think if you update rubygems then the error should go away.

Contributor

rohit commented Mar 12, 2012

Can you test again with both Bundler 1.0.x and Bundler 1.1 ― by my guesstimate this has to do with your version of Rubygems. I think if you update rubygems then the error should go away.

@tomykaira

This comment has been minimized.

Show comment
Hide comment
@tomykaira

tomykaira Mar 13, 2012

Contributor

There are full logs of bundling with 1.0.22 and 1.1.0.

I am willing to try this with other set of versions, if it is needed.

Gemfile:

source :rubygems

gem 'sinatra'
gem 'milkode', '0.5.0'

With ruby 1.9.2-p180, rubygems 1.8.17, bundler 1.0.22

~/programs/ruby/gitomb (bundler-test)> ruby --version
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
~/programs/ruby/gitomb (bundler-test)> gem --version
1.8.17
~/programs/ruby/gitomb (bundler-test)> rvm list

rvm rubies

   ruby-1.8.7-p334 [ i386 ]
   ruby-1.9.3-preview1 [ i386 ]
   ruby-head [ i386 ]
=> ruby-1.9.2-p180 [ i386 ]
   ruby-1.9.2-p290 [ i386 ]
   ruby-1.8.7-p352 [ i386 ]

~/programs/ruby/gitomb (bundler-test)> bundle --version
Bundler version 1.0.22
~/programs/ruby/gitomb (bundler-test)> rm .bundle/ vendor/ Gemfile.lock
~/programs/ruby/gitomb (bundler-test)> bundle install --path vendor/gems
Fetching source index for http://rubygems.org/
Installing addressable (2.2.7) 
Installing io-like (0.3.0) 
Installing archive-zip (0.5.0) 
Installing coderay (0.9.8) 
Installing daemons (1.1.8) 
Installing eventmachine (0.12.10) with native extensions 
Installing haml (3.1.4) 
Installing highline (1.6.11) 
Installing hpricot (0.8.6) with native extensions 
Installing json (1.6.5) with native extensions 
Installing launchy (2.0.5) 
Installing nokogiri (1.5.2) with native extensions 
Installing rack (1.4.1) 
Installing pkg-config (1.1.3) 
Installing rroonga (1.3.1) with native extensions 
Installing sass (3.1.15) 
Installing rack-protection (1.2.0) 
Installing tilt (1.3.3) 
Installing sinatra (1.3.2) 
Installing termcolor (1.2.1) 
Installing thin (1.3.1) with native extensions 
Installing milkode (0.5.0) 
Using bundler (1.0.22) 
Your bundle is complete! It was installed into ./vendor/gems

With ruby 1.9.2-p180, rubygems 1.8.17, bundler 1.1.0

~/programs/ruby/gitomb (bundler-test)> gem install bundler 
Fetching: bundler-1.1.0.gem (100%)
Successfully installed bundler-1.1.0
1 gem installed
~/programs/ruby/gitomb (bundler-test)> ruby --version
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
~/programs/ruby/gitomb (bundler-test)> gem --version
1.8.17
~/programs/ruby/gitomb (bundler-test)> bundle --version
Bundler version 1.1.0
~/programs/ruby/gitomb (bundler-test)> rm .bundle/ vendor/ Gemfile.lock
~/programs/ruby/gitomb (bundler-test)> bundle install --path vendor/gems
Fetching gem metadata from http://rubygems.org/.
Unfortunately, the gem milkode (0.2.3) has an invalid gemspec. As a result, Bundler cannot install this Gemfile. Please ask the gem author to yank the bad version to fix this issue. For more information, see http://bit.ly/syck-defaultkey.
Contributor

tomykaira commented Mar 13, 2012

There are full logs of bundling with 1.0.22 and 1.1.0.

I am willing to try this with other set of versions, if it is needed.

Gemfile:

source :rubygems

gem 'sinatra'
gem 'milkode', '0.5.0'

With ruby 1.9.2-p180, rubygems 1.8.17, bundler 1.0.22

~/programs/ruby/gitomb (bundler-test)> ruby --version
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
~/programs/ruby/gitomb (bundler-test)> gem --version
1.8.17
~/programs/ruby/gitomb (bundler-test)> rvm list

rvm rubies

   ruby-1.8.7-p334 [ i386 ]
   ruby-1.9.3-preview1 [ i386 ]
   ruby-head [ i386 ]
=> ruby-1.9.2-p180 [ i386 ]
   ruby-1.9.2-p290 [ i386 ]
   ruby-1.8.7-p352 [ i386 ]

~/programs/ruby/gitomb (bundler-test)> bundle --version
Bundler version 1.0.22
~/programs/ruby/gitomb (bundler-test)> rm .bundle/ vendor/ Gemfile.lock
~/programs/ruby/gitomb (bundler-test)> bundle install --path vendor/gems
Fetching source index for http://rubygems.org/
Installing addressable (2.2.7) 
Installing io-like (0.3.0) 
Installing archive-zip (0.5.0) 
Installing coderay (0.9.8) 
Installing daemons (1.1.8) 
Installing eventmachine (0.12.10) with native extensions 
Installing haml (3.1.4) 
Installing highline (1.6.11) 
Installing hpricot (0.8.6) with native extensions 
Installing json (1.6.5) with native extensions 
Installing launchy (2.0.5) 
Installing nokogiri (1.5.2) with native extensions 
Installing rack (1.4.1) 
Installing pkg-config (1.1.3) 
Installing rroonga (1.3.1) with native extensions 
Installing sass (3.1.15) 
Installing rack-protection (1.2.0) 
Installing tilt (1.3.3) 
Installing sinatra (1.3.2) 
Installing termcolor (1.2.1) 
Installing thin (1.3.1) with native extensions 
Installing milkode (0.5.0) 
Using bundler (1.0.22) 
Your bundle is complete! It was installed into ./vendor/gems

With ruby 1.9.2-p180, rubygems 1.8.17, bundler 1.1.0

~/programs/ruby/gitomb (bundler-test)> gem install bundler 
Fetching: bundler-1.1.0.gem (100%)
Successfully installed bundler-1.1.0
1 gem installed
~/programs/ruby/gitomb (bundler-test)> ruby --version
ruby 1.9.2p180 (2011-02-18 revision 30909) [i686-linux]
~/programs/ruby/gitomb (bundler-test)> gem --version
1.8.17
~/programs/ruby/gitomb (bundler-test)> bundle --version
Bundler version 1.1.0
~/programs/ruby/gitomb (bundler-test)> rm .bundle/ vendor/ Gemfile.lock
~/programs/ruby/gitomb (bundler-test)> bundle install --path vendor/gems
Fetching gem metadata from http://rubygems.org/.
Unfortunately, the gem milkode (0.2.3) has an invalid gemspec. As a result, Bundler cannot install this Gemfile. Please ask the gem author to yank the bad version to fix this issue. For more information, see http://bit.ly/syck-defaultkey.
@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Mar 13, 2012

Member

Hi guys. This bug is slightly complex, but there is unfortunately no way to solve it inside Bundler.

Here's why things go wrong:

  1. The milkode 0.2.3 gemspec on rubygems.org is invalid.
  2. When you have a gemfile that includes milkode, Bundler needs to know about every version of the gem in order to find a version that it can install. That means downloading the milkode 0.2.3 gemspec.
  3. The milkode 0.2.3 gemspec is invalid, and so Bundler explodes.

In Bundler 1.0, the gemspec is never downloaded. Instead, Bundler downloads the entire list of every single gem version that has ever existed, and uses that list. Since that list is marshalled Ruby instead of YAML, it is not corrupt, and does not cause an exception.

You can force Bundler 1.1 to use the old (and much slower) approach, you can force it by running bundle install --full-index. I suggest that you try to get the bad gem yanked instead, though. :)

Member

indirect commented Mar 13, 2012

Hi guys. This bug is slightly complex, but there is unfortunately no way to solve it inside Bundler.

Here's why things go wrong:

  1. The milkode 0.2.3 gemspec on rubygems.org is invalid.
  2. When you have a gemfile that includes milkode, Bundler needs to know about every version of the gem in order to find a version that it can install. That means downloading the milkode 0.2.3 gemspec.
  3. The milkode 0.2.3 gemspec is invalid, and so Bundler explodes.

In Bundler 1.0, the gemspec is never downloaded. Instead, Bundler downloads the entire list of every single gem version that has ever existed, and uses that list. Since that list is marshalled Ruby instead of YAML, it is not corrupt, and does not cause an exception.

You can force Bundler 1.1 to use the old (and much slower) approach, you can force it by running bundle install --full-index. I suggest that you try to get the bad gem yanked instead, though. :)

@indirect indirect closed this Mar 13, 2012

@tomykaira

This comment has been minimized.

Show comment
Hide comment
@tomykaira

tomykaira Mar 13, 2012

Contributor

Thanks @indirect!!

I did not know the --full-index option.
At least, I no longer have to downgrade bundler every time :)

Contributor

tomykaira commented Mar 13, 2012

Thanks @indirect!!

I did not know the --full-index option.
At least, I no longer have to downgrade bundler every time :)

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Mar 15, 2012

Maybe I'm missing something obvious, but couldn't Bundler simply choose not to "explode" when it hits an invalid gemspec? The current code catches the error and raises another, bringing everything to a screeching halt, but it could just dump the error message to STDERR and carry on.

Also, I found http://blog.rubygems.org/2011/08/31/shaving-the-yaml-yak.html quite unhelpful, it explains why it's blowing up but doesn't actually give users any way to solve the problem. Linking to this discussion, or simply suggesting the use of --full-index, would be much more useful.

Maybe I'm missing something obvious, but couldn't Bundler simply choose not to "explode" when it hits an invalid gemspec? The current code catches the error and raises another, bringing everything to a screeching halt, but it could just dump the error message to STDERR and carry on.

Also, I found http://blog.rubygems.org/2011/08/31/shaving-the-yaml-yak.html quite unhelpful, it explains why it's blowing up but doesn't actually give users any way to solve the problem. Linking to this discussion, or simply suggesting the use of --full-index, would be much more useful.

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Mar 15, 2012

One more thing: while --full-index works nicely for bundle install, it's not available for bundle update, which also spits the dummy on encountering an old invalid gemspec. =/

One more thing: while --full-index works nicely for bundle install, it's not available for bundle update, which also spits the dummy on encountering an old invalid gemspec. =/

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Mar 15, 2012

Member

What exactly do you mean by "carry on"? Pretend that the gem doesn't exist? That would produce completely different resolves between 1.0 and 1.1, and isn't acceptable.

The solution for gem end-users is to get the gem owner to yank bad gems. Or possibly get the rubygems.org maintainers to yank the bad gems, if the gem owners aren't available.

If you're interested in patching update to support --full-index, that seems reasonable to me.

Member

indirect commented Mar 15, 2012

What exactly do you mean by "carry on"? Pretend that the gem doesn't exist? That would produce completely different resolves between 1.0 and 1.1, and isn't acceptable.

The solution for gem end-users is to get the gem owner to yank bad gems. Or possibly get the rubygems.org maintainers to yank the bad gems, if the gem owners aren't available.

If you're interested in patching update to support --full-index, that seems reasonable to me.

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Mar 15, 2012

No, pretend that version of the gem doesn't exist, just return a nil instead of an array and call compact. Here's a quick shot at it that seems to work: #1763

No, pretend that version of the gem doesn't exist, just return a nil instead of an array and call compact. Here's a quick shot at it that seems to work: #1763

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Mar 15, 2012

Member

That's what I mean. It's a complete non-starter to implement anything that results in Bundler 1.0 creating a lock file that 1.1 can't install. Since there's no way for 1.0 to tell which versions have bad gemspecs, 1.1 has to abort when it runs into any.

Member

indirect commented Mar 15, 2012

That's what I mean. It's a complete non-starter to implement anything that results in Bundler 1.0 creating a lock file that 1.1 can't install. Since there's no way for 1.0 to tell which versions have bad gemspecs, 1.1 has to abort when it runs into any.

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Mar 15, 2012

Not quite following you here. 1.1 is already breaking on lots of Gemfiles that used to work just fine under 1.0, so you have already broken compatibility. My suggested fix would stop it from breaking in the vast majority cases where the broken gemspec is for a version the user does not actually want to install.

Not quite following you here. 1.1 is already breaking on lots of Gemfiles that used to work just fine under 1.0, so you have already broken compatibility. My suggested fix would stop it from breaking in the vast majority cases where the broken gemspec is for a version the user does not actually want to install.

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Mar 15, 2012

Member

We have different definitions of "broken". You just want to install your gem. I have to make sure that Bundler cannot ever, under any circumstances, install a different set of gems between 1.0 and 1.1. That means making --full-index available if you need it, and aborting instead of silently causing huge potential problems later.

Member

indirect commented Mar 15, 2012

We have different definitions of "broken". You just want to install your gem. I have to make sure that Bundler cannot ever, under any circumstances, install a different set of gems between 1.0 and 1.1. That means making --full-index available if you need it, and aborting instead of silently causing huge potential problems later.

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Mar 16, 2012

OK, I think I finally see where you're coming from: so if the Gemfile is written loosely, and the latest matching version of the gem has an invalid gemspec, then you don't want Bundler 1.0 to resolve to v.invalid and Bundler 1.1 to skip v.invalid and resolve to v.invalid-minus-1. This is the case you're concerned about, right? So would it be possible to make it abort only if the matched version has an invalid gemspec, and just raise a warning otherwise? Because if the gemfile is requiring 0.4.0, and it's 0.2.3 that's broken, then neither version will ever resolve to 0.2.3 and there is no potential for later problems.

OK, I think I finally see where you're coming from: so if the Gemfile is written loosely, and the latest matching version of the gem has an invalid gemspec, then you don't want Bundler 1.0 to resolve to v.invalid and Bundler 1.1 to skip v.invalid and resolve to v.invalid-minus-1. This is the case you're concerned about, right? So would it be possible to make it abort only if the matched version has an invalid gemspec, and just raise a warning otherwise? Because if the gemfile is requiring 0.4.0, and it's 0.2.3 that's broken, then neither version will ever resolve to 0.2.3 and there is no potential for later problems.

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Mar 16, 2012

Member

The problem is that there's no way to know what the version of the broken spec is… because it's broken. Specs are ruby, and it's impossible to evaluate the spec, because it's invalid ruby.

On Mar 15, 2012, at 5:47 PM, Jani Patokallio wrote:

OK, I think I finally see where you're coming from: so if the Gemfile is written loosely, and the latest matching version of the gem has an invalid gemspec, then you don't want Bundler 1.0 to resolve to v.invalid and Bundler 1.1 to skip v.invalid and resolve to v.invalid-minus-1. This is the case you're concerned about, right? So would it be possible to make it abort only if the matched version has an invalid gemspec, and just raise a warning otherwise? Because if the gemfile is requiring 0.4.0, and it's 0.2.3 that's broken, then neither version will ever resolve to 0.2.3 and there is no potential for later problems.


Reply to this email directly or view it on GitHub:
#1715 (comment)

Member

indirect commented Mar 16, 2012

The problem is that there's no way to know what the version of the broken spec is… because it's broken. Specs are ruby, and it's impossible to evaluate the spec, because it's invalid ruby.

On Mar 15, 2012, at 5:47 PM, Jani Patokallio wrote:

OK, I think I finally see where you're coming from: so if the Gemfile is written loosely, and the latest matching version of the gem has an invalid gemspec, then you don't want Bundler 1.0 to resolve to v.invalid and Bundler 1.1 to skip v.invalid and resolve to v.invalid-minus-1. This is the case you're concerned about, right? So would it be possible to make it abort only if the matched version has an invalid gemspec, and just raise a warning otherwise? Because if the gemfile is requiring 0.4.0, and it's 0.2.3 that's broken, then neither version will ever resolve to 0.2.3 and there is no potential for later problems.


Reply to this email directly or view it on GitHub:
#1715 (comment)

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Mar 16, 2012

Isn't the version of the gem already available in the list of gems? It's already included in the error message output on encountering a broken gemspec: gem #{s[:name]} (#{s[:number]}) has an invalid gemspec

Isn't the version of the gem already available in the list of gems? It's already included in the error message output on encountering a broken gemspec: gem #{s[:name]} (#{s[:number]}) has an invalid gemspec

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Mar 16, 2012

Member

That may be true. To be honest, this bug just isn't a high priority at present, because bad gemspecs are no longer created by the latest versions of rubygems and rubygems.org.

Why not just get the gem yanked by the author or the rubygems.org team?

On Mar 15, 2012, at 6:14 PM, Jani Patokallioreply@reply.github.com wrote:

Isn't the version of the gem already available in the list of gems? It's already included in the error message output on encountering a broken gemspec: gem #{s[:name]} (#{s[:number]}) has an invalid gemspec


Reply to this email directly or view it on GitHub:
#1715 (comment)

Member

indirect commented Mar 16, 2012

That may be true. To be honest, this bug just isn't a high priority at present, because bad gemspecs are no longer created by the latest versions of rubygems and rubygems.org.

Why not just get the gem yanked by the author or the rubygems.org team?

On Mar 15, 2012, at 6:14 PM, Jani Patokallioreply@reply.github.com wrote:

Isn't the version of the gem already available in the list of gems? It's already included in the error message output on encountering a broken gemspec: gem #{s[:name]} (#{s[:number]}) has an invalid gemspec


Reply to this email directly or view it on GitHub:
#1715 (comment)

@soulcutter

This comment has been minimized.

Show comment
Hide comment
@soulcutter

soulcutter Apr 9, 2012

I ran into this today...

How many bad gemspecs are out there, though? It's got to be a huge number. Getting each one pulled is a bit onorous. I would be satisfied with a better error message AND patching update to be able to specify --full-index . I see this issue is a bit old and stale... should I open another issue for that? Or should this be reopened?

I ran into this today...

How many bad gemspecs are out there, though? It's got to be a huge number. Getting each one pulled is a bit onorous. I would be satisfied with a better error message AND patching update to be able to specify --full-index . I see this issue is a bit old and stale... should I open another issue for that? Or should this be reopened?

@indirect

This comment has been minimized.

Show comment
Hide comment
@indirect

indirect Apr 10, 2012

Member

It is practically zero, because the rubygems.org team has proactively gone through and wiped or corrected almost all of them. Plus, the latest versions of Rubygems and Rubygems.org automatically prevent bad gemspecs from being created or uploaded.

Member

indirect commented Apr 10, 2012

It is practically zero, because the rubygems.org team has proactively gone through and wiped or corrected almost all of them. Plus, the latest versions of Rubygems and Rubygems.org automatically prevent bad gemspecs from being created or uploaded.

@soulcutter

This comment has been minimized.

Show comment
Hide comment
@soulcutter

soulcutter Apr 10, 2012

Huh, strange how I still ran up against this. Maybe this is the wrong place to ask, but who do I contact about a problem like this? rake-hooks was the project that seems to have tripped bundler up for me FWIW.. it's not obvious to me which of the 7 versions on rubygems is creating the issue.

Huh, strange how I still ran up against this. Maybe this is the wrong place to ask, but who do I contact about a problem like this? rake-hooks was the project that seems to have tripped bundler up for me FWIW.. it's not obvious to me which of the 7 versions on rubygems is creating the issue.

@jpatokal

This comment has been minimized.

Show comment
Hide comment
@jpatokal

jpatokal Apr 10, 2012

No, it's not zero, because this bug report alone has four people who have hit the issue. And because Bundler chokes if any version of the gem has a bad spec, it makes no difference if the later versions are correct.

@soulcutter: The error message should say which version is causing the problem, eg. "Unfortunately, the gem exactish_target (0.2.1) has an invalid gemspec" means it's 0.2.1 that's causing the problem. There may be other broken versions as well though, since it aborts on the first...

No, it's not zero, because this bug report alone has four people who have hit the issue. And because Bundler chokes if any version of the gem has a bad spec, it makes no difference if the later versions are correct.

@soulcutter: The error message should say which version is causing the problem, eg. "Unfortunately, the gem exactish_target (0.2.1) has an invalid gemspec" means it's 0.2.1 that's causing the problem. There may be other broken versions as well though, since it aborts on the first...

@soulcutter

This comment has been minimized.

Show comment
Hide comment
@soulcutter

soulcutter Apr 10, 2012

Thanks @jpatokal - I didn't realize that was what the (1) in the message was :P

Thanks @jpatokal - I didn't realize that was what the (1) in the message was :P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment