Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Fix invalid specs resulting from incorrect syck time parsing #57

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet

dazuma commented Apr 26, 2011

Fix for a bug where invalid gemspecs are being generated when a gem is built with psych but read using syck.

To reproduce the issue:

  • Ensure psych is available in your ruby installation.
  • Build a gem. The gem's manifest should have a date of a format similar to "2011-04-26 00:00:00.000000000Z". If the date in the manifest has a format similar to "2011-04-26 00:00:00 Z", it means psych wasn't available and syck was used to generate the manifest.
  • Switch to a ruby installation that lacks psych (or manually remove psych from your ruby installation by removing both psych.rb and psych.{so,bundle}).
  • Install the gem you just built.
  • Check the generated gemspec for the installed gem. The date will appear formatted like "%q{2011-04-26 00:00:00.000000000Z}". This is incorrect; it should be formatted like "%q{2011-04-26}".
  • Now, any operation that reads the generated gemspec (for example, gem list) will fail. Indeed, it will not be possible to uninstall the gem until you manually fix the generated gemspec.

(These steps were reproduced on Snow Leopard, ruby 1.9.2-p180, with the built-in psych atop libyaml 0.1.3.)

The reason for this behavior is that syck is incapable of parsing certain timestamp formats, and unfortunately, among those is the format that psych outputs by default. Thus, a ruby using syck will fail to properly read the manifest of a gem built by a ruby using psych.

The fix involves two commits:

  • Commit e886aa4 fixes the problem at the source, preventing the generation of further bad gemspec files. It modifies the code that parses the manifest, and checks the @Date field to see if it failed to parse properly into a timestamp (in which case it will appear as a string instead). In this case, it manually converts it to the proper Time object.
  • Commit c3ac5a8 implements some damage control for bad gemspecs already written. It loosens the format restrictions for the gemspec date field, hence restoring operation for users already bitten by the bug.

I believe this is a critical issue to fix because it can put a user's gem installation in an inconsistent state that they cannot fix otherwise without manual edits to the generated gemspec files.

jgeiger commented Apr 28, 2011

Curious why this hasn't popped up before now. I'm having the same issue. I was force using 'syck' to get around this issue. http://crohr.me/2011/02/23/issue-with-ruby1.9.2-yaml-parser-and-merge-keys-in-yaml-configuration-file.html

Contributor

zenspider commented May 18, 2011

Dazuma, thanks. Could you write a test as well?

I'm not sold on where you attack the problem, but I'd like to see this under a reproduction in order to get it tackled.

I should also point out that tenderlove put out a gem of psych so it can be upgraded independent of ruby1.9. The fix is then just adding gem 'psych'.

Contributor

zenspider commented May 18, 2011

I think I got a repro:

def test_self_from_yaml_syck_bug
  # This is equivalent to (and totally valid) psych 1.0 output and
  # causes parse errors on syck.
  yaml = @a1.to_yaml
  yaml.sub!(/^date:.*/, "date: 2011-04-26 00:00:00.000000000Z")

  new_spec = with_syck do
    Gem::Specification.from_yaml yaml
  end

  assert_kind_of Time, @a1.date
  assert_kind_of Time, new_spec.date
end
Contributor

zenspider commented May 19, 2011

OK. I have a fix in normalize_yaml_input.

Thanks!

@zenspider zenspider closed this May 19, 2011

Here is the way I fix the "invalid date format in specification" error:

1.) Go to the specifications folder located at:

/usr/local/lib/ruby/gems/1.8/specifications/

2.) Find the spec that is causing the problem.

3.) Change "s.date = %q{2011-05-21 00:00:00.000000000Z}" to "s.date = %q{2011-05-21}"

That's a WIN for me! Good Luck

Per http://help.rubygems.org/discussions/problems/575-invalid-gemspec-with-invalid-date-format-in-specification, this was going to be fixed in 1.8.3 or 1.9.0... but 1.8.7 (or even 1.8.8) is out and it's still not fixed.

Any updates?

Thanks,
Andy

antdude commented Aug 23, 2011

Hi! I am seeing this a lot with Gem updates recently including today's:

gem install rest-client

Successfully installed rest-client-1.6.6
1 gem installed
Installing ri documentation for rest-client-1.6.6...
Installing RDoc documentation for rest-client-1.6.6...

gem update

Invalid gemspec in [/var/lib/gems/1.8/specifications/rest-client-1.6.6.gemspec]: invalid date format in specification: "2011-08-22 00:00:00.000000000Z"
Invalid gemspec in [/var/lib/gems/1.8/specifications/rest-client-1.6.6.gemspec]: invalid date format in specification: "2011-08-22 00:00:00.000000000Z"
Updating installed gems
Nothing to update

Still seeing this behavior w/ 1.8.8 as well

I'm getting this right now

ree 1.8.7 / gem 1.8.9

This has regressed.

Owner

drbrain commented Aug 24, 2011

Can you show gem env and gem list -d rest-client?

I can't reproduce:

$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 1.8.9
  - RUBY VERSION: 1.8.7 (2010-01-10 patchlevel 249) [universal-darwin11.0]
  - INSTALLATION DIRECTORY: /Library/Ruby/Gems/1.8
  - RUBY EXECUTABLE: /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
  - EXECUTABLE DIRECTORY: /usr/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - universal-darwin-11
  - GEM PATHS:
     - /Library/Ruby/Gems/1.8
     - /Users/drbrain/.gem/ruby/1.8
     - /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
     - "uninstall" => "--format-executable --no-user-install"
     - "install" => "--format-executable --no-user-install -P LowSecurity"
     - "gemcutter_key" => "********************************"
     - "update" => "--format-executable --no-user-install -P LowSecurity"
  - REMOTE SOURCES:
     - http://rubygems.org/
$ gem install -i ~/tmp/gems rest-client -v 1.6.6
Fetching: mime-types-1.16.gem (100%)
Fetching: rest-client-1.6.6.gem (100%)
Successfully installed mime-types-1.16
Successfully installed rest-client-1.6.6
2 gems installed
Installing ri documentation for mime-types-1.16...
Installing ri documentation for rest-client-1.6.6...
Installing RDoc documentation for mime-types-1.16...
Installing RDoc documentation for rest-client-1.6.6...
$ GEM_HOME=~/tmp/gems GEM_PATH=~/tmp/gems gem list

*** LOCAL GEMS ***

mime-types (1.16)
rest-client (1.6.6)
$ GEM_HOME=~/tmp/gems GEM_PATH=~/tmp/gems ruby -rubygems -e 'gem "rest-client"; p Gem.loaded_specs.include? "rest-client"'
true

ryw commented Aug 26, 2011

$ gem env
Invalid gemspec in [/Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/specifications/gherkin-2.4.15.gemspec]: invalid date format in specification: "2011-08-23 00:00:00.000000000Z"
Invalid gemspec in [/Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/specifications/rest-client-1.6.6.gemspec]: invalid date format in specification: "2011-08-22 00:00:00.000000000Z"
RubyGems Environment:
  - RUBYGEMS VERSION: 1.8.9
  - RUBY VERSION: 1.8.7 (2011-02-18 patchlevel 334) [i686-darwin10.8.0]
  - INSTALLATION DIRECTORY: /Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events
  - RUBY EXECUTABLE: /Users/ry/.rvm/rubies/ree-1.8.7-2011.03/bin/ruby
  - EXECUTABLE DIRECTORY: /Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86-darwin-10
  - GEM PATHS:
     - /Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events
     - /Users/ry/.rvm/gems/ree-1.8.7-2011.03@global
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - http://rubygems.org/
$ gem list -d rest-client
Invalid gemspec in [/Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/specifications/gherkin-2.4.15.gemspec]: invalid date format in specification: "2011-08-23 00:00:00.000000000Z"
Invalid gemspec in [/Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/specifications/rest-client-1.6.6.gemspec]: invalid date format in specification: "2011-08-22 00:00:00.000000000Z"

*** LOCAL GEMS ***

Invalid gemspec in [/Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/specifications/gherkin-2.4.15.gemspec]: invalid date format in specification: "2011-08-23 00:00:00.000000000Z"
Invalid gemspec in [/Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events/specifications/rest-client-1.6.6.gemspec]: invalid date format in specification: "2011-08-22 00:00:00.000000000Z"
rest-client (1.6.7)
    Authors: Adam Wiggins, Julien Kirch
    Homepage: http://github.com/archiloque/rest-client
    Installed at: /Users/ry/.rvm/gems/ree-1.8.7-2011.03@rm-events

    Simple HTTP and REST client for Ruby, inspired by microframework
    syntax for specifying actions.
Owner

drbrain commented Aug 26, 2011

@ryw You don't show gem install.

Do gem install rest-client -v 1.6.6 and gem install gherkin -v 2.4.15 fix the errors?

If I edit a gemspec manually to have a bad date and reinstall it the error is fixed:

$ gem list ZenTest
Invalid gemspec in [/Library/Ruby/Gems/1.8/specifications/ZenTest-4.6.1.gemspec]: invalid date format in specification: "2011-08-12 00:00:00.000000000Z"

*** LOCAL GEMS ***

Invalid gemspec in [/Library/Ruby/Gems/1.8/specifications/ZenTest-4.6.1.gemspec]: invalid date format in specification: "2011-08-12 00:00:00.000000000Z"

$ sudo gem install ZenTest -v 4.6.1
Invalid gemspec in [/Library/Ruby/Gems/1.8/specifications/ZenTest-4.6.1.gemspec]: invalid date format in specification: "2011-08-12 00:00:00.000000000Z"
Invalid gemspec in [/Library/Ruby/Gems/1.8/specifications/ZenTest-4.6.1.gemspec]: invalid date format in specification: "2011-08-12 00:00:00.000000000Z"
Successfully installed ZenTest-4.6.1
1 gem installed
Installing ri documentation for ZenTest-4.6.1...
Installing RDoc documentation for ZenTest-4.6.1...
$ gem list ZenTest

*** LOCAL GEMS ***

ZenTest (4.6.1)
Contributor

zenspider commented Aug 26, 2011

I do think that gem pristine should work without loading the specs and I'll add a separate ticket for that.

ryw commented Aug 29, 2011

Yes; re-installing those gems fixed. Thanks.

gorenje commented Sep 13, 2011

using gem --version ==> 1.8.10, i've got another date format causing issues:

 mongoid-2.1.2.gemspec]: invalid date format in specification: "2011-07-31 00:00:00.000000000 %:z"

which causes bundle to fail, e.g. doing "bundle list"

Invalid gemspec in [/var/app/.../gemset/specifications/mongoid-2.1.2.gemspec]: invalid date format in specification: "2011-07-31 00:00:00.000000000 %:z"
Invalid gemspec in [/var/app/.../gemset/specifications/mongoid-2.1.2.gemspec]: invalid date format in specification: "2011-07-31 00:00:00.000000000 %:z"
Gems included by the bundle:
Could not find mongoid-2.1.2 in any of the sources

but

$ bundle update mongoid
Invalid gemspec in [/var/app/.../gemset/specifications/mongoid-2.1.2.gemspec]: invalid date format in specification: "2011-07-31 00:00:00"
Invalid gemspec in [/var/app/.../gemset/specifications/mongoid-2.1.2.gemspec]: invalid date format in specification: "2011-07-31 00:00:00"
Using rake (0.8.7) 
....blah blah....
Installing mongoid (2.1.2) Invalid gemspec in [/var/app/.../gemset/specifications/mongoid-2.1.2.gemspec]: invalid date format in specification: "2011-07-31 00:00:00.000000000 %:z"
....blah blah....
Your bundle is updated! Use `bundle show [gemname]` to see where a bundled gem is installed.

And then bundle list will fail again, as above, with mongoid-2.1.2 not found. Also doing a gem install mongoid -v 2.1.2 did not help.

What eventually fixed the problem was emacs mongoid-2.1.2.gemspec and changing s.date to "s.date = %q{2011-08-09}" (some random but working date).

background on this issue (which per the article is a large issue):
http://blog.rubygems.org/2011/08/31/shaving-the-yaml-yacc.html

mkhezr commented Oct 11, 2011

I'm using rvm and ruby 1.9.2p180 and I was getting this error. ran the following command and the server starts up properly:

rvm rubygems latest

Hi, I still see this problem with term-ansicolor and a few others (but that one is vexing me today, so I must shout out to @flori). Is the problem now that gems uploaded to rubygems.org during the reign of the busted rubygems versions need to be rebuilt and uploaded again?

Are there client side workarounds that do not require newer rubygems on my production boxes? (Because they run Ubuntu 12.04, with rubygems 1.8.15, and that won't change unless you convince Ubuntu to update rubygems in their precise-updates repo, and my guess is that ain't happening.)

Could rubygems.org could run discovery and flag the offending gems?
Basically, for every gem:

for each gem $foo:
  tar -Oxf $foo.gem metadata.gz  | gunzip -dc - | grep date: | grep 00.000000000; if [ $? -eq 0 ]; then echo "Busted gem $foo"; fi

@sodabrew sodabrew referenced this pull request in rubygems/rubygems.org Nov 25, 2012

Closed

There are still gems with invalid date specs on rubygems.org #495

@nirvdrum nirvdrum pushed a commit to mogotest/css_parser that referenced this pull request Sep 16, 2014

@alexdunae alexdunae Rebuild gem with Rubygems 1.8.10
Deal with incorrect date format in the gemspec
See rubygems/rubygems#57
d6df775
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment