Double Loading Psych - Conflict with Ruby Bundled Version

[tadman]

Lately I've had nothing but trouble with Psych conflicting with the version that's built into Ruby. Using 1.9.3 and Psych 1.3.3 leads to the all too familiar " warning: already initialized constant" stream of errors that results when attempting to redefine what is already defined.

There was talk of introducing a fail-safe where Psych wouldn't load if already loaded, which you'd think would be as simple as adding something like this to the gem's main library file:

if (defined?(Psych))
  # Opportunity to warn or complain about already being loaded here...
end

Unlike most other gems where double-loading is almost always the fault of the developer, Psych is in the relatively unique position of being bundled with Ruby and being distributed as a commonly used gem. It's further complicated by the fact that it has C extensions that need to be loaded as well, and as far as I can tell, these extensions cannot be swapped out once loaded.

In short, allowing Psych to load twice is a bug in the library. It should, at the very least, warn when loaded twice. Ideally it should try and identify what caused it to be loaded prematurely, though this could be tricky to discern.

I'd also like to propose that the version of Psych shipped with Ruby be called something other than Psych so there's no risk of conflict in the future. It needs a different namespace so that it can co-exist with the gem version. This would side-step virtually all o the problems I've experienced and would eliminate a lot of confusion encountered by others if the issues filed here and questions asked in other places are any example.

Whatever YAML library is the default in Ruby should be given a name reflecting this status as "official".

[bluemont]

👍

I want to share my full stack trace because I think it may lead more lost souls here.

/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle: warning: already initialized constant ANY
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle: warning: already initialized constant UTF8
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle: warning: already initialized constant UTF16LE
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin11.4.0/psych.bundle: warning: already initialized constant UTF16BE
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/stream.rb:12: warning: already initialized constant ANY
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/stream.rb:15: warning: already initialized constant UTF8
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/stream.rb:18: warning: already initialized constant UTF16LE
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/stream.rb:21: warning: already initialized constant UTF16BE
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/sequence.rb:42: warning: already initialized constant ANY
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/sequence.rb:45: warning: already initialized constant BLOCK
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/sequence.rb:48: warning: already initialized constant FLOW
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/scalar.rb:9: warning: already initialized constant ANY
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/scalar.rb:12: warning: already initialized constant PLAIN
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/scalar.rb:15: warning: already initialized constant SINGLE_QUOTED
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/scalar.rb:18: warning: already initialized constant DOUBLE_QUOTED
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/scalar.rb:21: warning: already initialized constant LITERAL
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/scalar.rb:24: warning: already initialized constant FOLDED
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/mapping.rb:16: warning: already initialized constant ANY
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/mapping.rb:19: warning: already initialized constant BLOCK
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/nodes/mapping.rb:22: warning: already initialized constant FLOW
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/visitors/visitor.rb:10: warning: already initialized constant DISPATCH
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/scalar_scanner.rb:8: warning: already initialized constant TIME
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/scalar_scanner.rb:11: warning: already initialized constant FLOAT
/Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/parser.rb:33:in `': superclass mismatch for class Mark (TypeError)
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/parser.rb:32:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych/parser.rb:1:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych.rb:7:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/psych.rb:7:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/activesupport-3.2.6/lib/active_support/ordered_hash.rb:2:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/activesupport-3.2.6/lib/active_support/ordered_hash.rb:2:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/activesupport-3.2.6/lib/active_support/ordered_options.rb:1:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/activesupport-3.2.6/lib/active_support/ordered_options.rb:1:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/configuration.rb:2:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/configuration.rb:2:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/railtie.rb:2:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/railtie.rb:2:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/engine.rb:1:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/engine.rb:1:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/plugin.rb:1:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/plugin.rb:1:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/application.rb:3:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails/application.rb:3:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails.rb:10:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/railties-3.2.6/lib/rails.rb:10:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/actionpack-3.2.6/lib/action_controller/railtie.rb:1:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/actionpack-3.2.6/lib/action_controller/railtie.rb:1:in `'
    from /Users/david/dev/my_rails_app/config/application.rb:5:in `require'
    from /Users/david/dev/my_rails_app/config/application.rb:5:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /Users/david/dev/my_rails_app/config/environment.rb:2:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /Users/david/dev/my_rails_app/test/minitest_helper.rb:2:in `'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from /Users/david/.rbenv/versions/1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

[lacker]

+1. It also seems impossible to use Psych 1.3.4 with Ruby 1.9.2 and the bundler. With 1.9.2-p180, even:

gem install psych --version 1.3.4
bundle exec irb

leads to "warning: already initialized constant VERSION" errors. This is a shame since the default Psych 1.0.0 in Ruby 1.9.2 has memory leaks.

[tenderlove]

Has anyone tried to figure out why Psych is being loaded twice?

I don't want to add hacks around if it's loaded twice. That means you could have one version loaded when you expect a different one. I'd be fine raising an exception if Psych has already been loaded once, but that means your process will just quit.

I can help track this down, but I can't seem to reproduce the error on 1.9.3 (and I can't get 1.9.2 building on my machine). Could someone provide me with a way to reproduce this with 1.9.3? Or even better, figure out the root of the problem? Thanks.

[bluemont]

I'm not getting the error any longer. I'm not sure how to reproduce -- or when it "went away" or got fixed. I'm running Mountain Lion, rbenv, and ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12.0.0] now.

[lacker]

In 1.9.2-p180, AFAICT the bundler is requiring Psych before your gemfile is actually loaded. So it uses the 1.0.0 Psych version which ships with 1.9.2-p180, which memory leaks. This happens before the bundle is set up. So if I try to include a newer version of Psych via gemfile, it loads twice.

Arguably this is a bundler problem rather than a Psych problem. The bundler logic does not seem to work well with gems that the bundler itself uses that are also provided with the base Ruby distribution. However, I'm not aware of other gems besides Psych that are dual-distributed like this. If the Psych shipped with Ruby was separate from the Psych provided in gems, these conflicts between pre-bundler and post-bundler code wouldn't exist.

[tenderlove]

@lacker what version of bundler are you using? I sent a patch to them a while back that would this commit makes sure the latest gem is loaded (if it's on your system). That may fix the issue.

[lacker]

I'm using 1.1.5 and that patch is included in my bundler. However, that patch does not appear to have the desired effect. Perhaps psych is getting required before lib/bundler.rb. I will investigate this more - thanks for pointing out that patch.

[tenderlove]

I was able to reproduce this. IMO, it's a bug in bundler, so I filed an issue.

[tenderlove]

Supposedly this was fixed with bundler / rubygems, so I'm closing this issue.

[frahugo]

In case someone would be reading this. My double-loading issue with Psych was caused by rubygems-bundler gem, version > 1.1.0. From 1.1.1, it requires 'yaml', and there the psych.rb provided with my ruby install gets loaded. Downgrading to 1.1.0 fixed my Psych problem.

[zinkkrysty]

Thanks @frahugo for pointing out the culprit. It seems they fixed the issue in v1.3.4: rvm/rubygems-bundler#53