Unable to bundle viz if a dependency is satisfied by a prerelease

[aprescott]

This came out of #3217, which describes hitting a NoMethodError:

`block in _populate_relations': undefined method `runtime_dependencies' for nil:NilClass (NoMethodError)

I spent some time looking into this situation and have a potential fix (given below), but I didn't know whether I should PR it, since I'm unclear on whether bundler should change or whether rubygems should change, given the solution.

I figured I would give an explanation of what I think is happening, along with a fix, and then some discussion could happen.

I'm using bundler 1.9.5 here. Entire reproduction steps are given below.

Reproducing

The setup here is a gem, eco, which requires eco-source. If the eco-source dependency happens to be satisfied by a prerelease version, then you can't bundle viz on Rubygems 2.4.0+. This forces eco-source to be a prerelease, just to illustrate the point.

#2.3.0: WORKS
#2.4.0: FAILS
gem_version_for_repro=2.4.0

cd /tmp
rm -rf viz-dependency-error
mkdir viz-dependency-error
cd viz-dependency-error

echo "2.2.2" > .ruby-version

# get any .ruby-version reloading
cd ..
cd -
ruby -v # should be 2.2.2., just for isolating the version fully

cat <<EOF > Gemfile
gem "eco", "= 1.0.0"
gem "eco-source", "= 1.1.0.rc.1"
EOF

gem update --system $gem_version_for_repro
gem install bundler -v '= 1.9.5' --no-document
bundle install

bundle viz

The Gemfile.lock here is:

GEM
  specs:
    coffee-script (2.4.1)
      coffee-script-source
      execjs
    coffee-script-source (1.9.1.1)
    eco (1.0.0)
      coffee-script
      eco-source
      execjs
    eco-source (1.1.0.rc.1)
    execjs (2.5.2)

PLATFORMS
  ruby

DEPENDENCIES
  eco (= 1.0.0)
  eco-source (= 1.1.0.rc.1)

Note that eco requires any eco-source, but here it happens to be satisfied by a prerelease gem.

Outline

Let's say you have an application which relies on haml-rails. Suppose it resolves to haml-rails 0.7.0, which has this set of dependencies:

haml-rails (0.7.0)
  actionpack (>= 4.0.1)
  activesupport (>= 4.0.1)
  haml (>= 3.1, < 5.0)
  html2haml (>= 1.0.1)
  railties (>= 4.0.1)

Note the haml requirement of (>= 3.1, < 5.0).

It could be that the haml is satisfied by, say haml 4.1.0.beta.1, which is a prerelease gem.

When you run bundle viz, in this situation, eventually Bundler::Graph#_populate_relations is called, which involves this chunk of code:

tmp = Set.new
parent_dependencies.each do |dependency|
  child_dependencies = dependency.to_spec.runtime_dependencies.to_set
  @relations[dependency.name] += child_dependencies.map(&:name).to_set
  tmp += child_dependencies

  @node_options[dependency.name] = _make_label(dependency, :node)
  child_dependencies.each do |c_dependency|
    @edge_options["#{dependency.name}_#{c_dependency.name}"] = _make_label(c_dependency, :edge)
  end
end
parent_dependencies = tmp

The parent_dependencies.each iteration will eventually hit the haml dependency. It will then execute this line:

child_dependencies = dependency.to_spec.runtime_dependencies.to_set

dependency.to_spec in this specific case, for the haml dependency, will return nil on Rubygems 2.4.0+ (I think), because of rubygems/rubygems@84e8d9f.

When you dig into the to_spec method as it exists in Rubygems 2.4.6, there is some logic around prerelease?:

matches.delete_if { |spec| spec.version.prerelease? } unless prerelease?

In the Graph#_populate_relations method, which calls dependency.to_spec, for the specific haml version in this example, it looks like this in a binding.pry:

[20] pry(#<Bundler::Graph>)> dependency
=> Gem::Dependency.new("haml", Gem::Requirement.new(["< 5.0", ">= 3.1"]), :runtime)
[21] pry(#<Bundler::Graph>)> dependency.to_spec
=> nil
[22] pry(#<Bundler::Graph>)> dependency.to_specs.map { |spec| spec.version }
=> [Gem::Version.new("4.1.0.beta.1")]
[23] pry(#<Bundler::Graph>)> dependency.to_specs.map { |spec| spec.version.prerelease? }
=> [true]
[24] pry(#<Bundler::Graph>)> dependency.prerelease?
=> false

If you force dependency.prerelease to true, then things change so that to_spec now returns a Gem::Specificaton instead of nil:

[25] pry(#<Bundler::Graph>)> dependency.prerelease = true
=> true
[26] pry(#<Bundler::Graph>)> dependency.to_spec
=> Gem::Specification.new do |s|
  s.name = "haml"
  s.version = Gem::Version.new("4.1.0.beta.1")
  s.installed_by_version = Gem::Version.new("2.4.5")
  s.authors = ["Nathan Weizenbaum", "Hampton Catlin", "Norman Clarke"]
  s.date = Time.utc(2014, 1, 7)
  s.dependencies = [Gem::Dependency.new("tilt", Gem::Requirement.new([">= 0"]), :runtime),
   Gem::Dependency.new("rails", Gem::Requirement.new([">= 3.2.0"]), :development),
   Gem::Dependency.new("rbench", Gem::Requirement.new([">= 0"]), :development),
   Gem::Dependency.new("minitest", Gem::Requirement.new(["~> 4.0"]), :development),
   Gem::Dependency.new("nokogiri", Gem::Requirement.new(["~> 1.6.0"]), :development)]
  s.description = "Haml (HTML Abstraction Markup Language) is a layer on top of HTML or XML that's\ndesigned to express the structure of documents in a non-repetitive, elegant, and\neasy way by using indentation rather than closing tags and allowing Ruby to be\nembedded with ease. It was originally envisioned as a plugin for Ruby on Rails,\nbut it can function as a stand-alone templating engine.\n"
  s.email = ["haml@googlegroups.com", "norman@njclarke.com"]
  s.executables = ["haml"]
  s.files = ["bin/haml"]
  s.homepage = "http://haml.info/"
  s.licenses = ["MIT"]
  s.post_install_message = "\nHEADS UP! Haml 4.0 has many improvements, but also has changes that may break\nyour application:\n\n* Support for Ruby 1.8.6 dropped\n* Support for Rails 2 dropped\n* Sass filter now always outputs <style> tags\n* Data attributes are now hyphenated, not underscored\n* html2haml utility moved to the html2haml gem\n* Textile and Maruku filters moved to the haml-contrib gem\n\nFor more info see:\n\nhttp://rubydoc.info/github/haml/haml/file/CHANGELOG.md\n\n"
  s.require_paths = ["lib"]
  s.required_ruby_version = Gem::Requirement.new([">= 1.9.2"])
  s.required_rubygems_version = Gem::Requirement.new(["> 1.3.1"])
  s.rubygems_version = "2.4.5"
  s.specification_version = 4
  s.summary = "An elegant, structured (X)HTML/XML templating engine."
  end

So one fix I discovered is to have _populate_relations forcibly set prerelease = true. In other words, change

parent_dependencies.each do |dependency|
  child_dependencies = dependency.to_spec.runtime_dependencies.to_set

to

parent_dependencies.each do |dependency|
  dependency.prerelease = true
  child_dependencies = dependency.to_spec.runtime_dependencies.to_set

But that feels wrong because the dependency may not actually be a prerelease. And the meaning of prerelease = true seems to mean something different than what's happening here.

haml doesn't require a prerelease, but it happens to be one.

A better approach would be to check dependency.version.prerelease?, but Gem::Dependency doesn't expose a version. And the definition of prerelease? for the Gem::Dependency is about the dependency's requirements, unless manually set by the prerelease= writer:

# Does this dependency require a prerelease?

def prerelease?
  @prerelease || requirement.prerelease?
end

So here, we have

[37] pry(#<Bundler::Graph>)> dependency
=> Gem::Dependency.new("haml", Gem::Requirement.new(["< 5.0", ">= 3.1"]), :runtime)
[38] pry(#<Bundler::Graph>)> dependency.requirement
=> Gem::Requirement.new(["< 5.0", ">= 3.1"])

Again, haml doesn't actually require a prerelease version, it just happens to have been satisfied by one.

So the only way I can see to actually identify this more intelligently is to check to_specs:

parent_dependencies.each do |dependency|
  if dependency.to_specs.any? { |spec| spec.version.prerelease? }
    dependency.prerelease = true
  end

  child_dependencies = dependency.to_spec.runtime_dependencies.to_set

That seems to me like Rubygems should provide a simpler way of doing this? Or maybe I'm wrong and this is the right fix?

Potential fix

Here's a diff of the potential fix, given all of the above.

--- /tmp/orig   2015-05-06 16:58:22.000000000 -0400
+++ /tmp/fixed  2015-05-06 16:59:07.000000000 -0400
@@ -36,6 +36,10 @@
         else
           tmp = Set.new
           parent_dependencies.each do |dependency|
+            if dependency.to_specs.any? { |spec| spec.version.prerelease? }
+              dependency.prerelease = true
+            end
+
             child_dependencies = dependency.to_spec.runtime_dependencies.to_set
             @relations[dependency.name] += child_dependencies.map(&:name).to_set
             tmp += child_dependencies

With this change, the reproduction steps given above go from failing to working.

# without fix
$ bundle viz
# ...
# bunch of output
# ...

Error details

    NoMethodError: undefined method `runtime_dependencies' for nil:NilClass
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/graph.rb:43:in `block in _populate_relations'
    /Users/adamprescott/.rubies/ruby-2.2.2/lib/ruby/2.2.0/set.rb:283:in `each_key'
    /Users/adamprescott/.rubies/ruby-2.2.2/lib/ruby/2.2.0/set.rb:283:in `each'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/graph.rb:38:in `_populate_relations'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/graph.rb:20:in `initialize'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/cli/viz.rb:11:in `new'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/cli/viz.rb:11:in `run'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/cli.rb:336:in `viz'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/cli.rb:10:in `start'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/bin/bundle:20:in `block in <top (required)>'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/lib/bundler/friendly_errors.rb:7:in `with_friendly_errors'
    /Users/adamprescott/.gem/ruby/2.2.2/gems/bundler-1.9.5/bin/bundle:18:in `<top (required)>'
    /Users/adamprescott/.gem/ruby/2.2.2/bin/bundle:23:in `load'
    /Users/adamprescott/.gem/ruby/2.2.2/bin/bundle:23:in `<main>'
    /Users/adamprescott/.gem/ruby/2.2.2/bin/ruby_executable_hooks:15:in `eval'
    /Users/adamprescott/.gem/ruby/2.2.2/bin/ruby_executable_hooks:15:in `<main>'

Environment

    Bundler   1.9.5
    Rubygems  2.4.0
    Ruby      2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin14]
    GEM_HOME  /Users/adamprescott/.gem/ruby/2.2.2
    GEM_PATH  /Users/adamprescott/.gem/ruby/2.2.2:/Users/adamprescott/.rubies/ruby-2.2.2/lib/ruby/gems/2.2.0
    Git       2.3.0
――― TEMPLATE END ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

Unfortunately, an unexpected error occurred, and Bundler cannot continue.

First, try this link to see if there are any existing issue reports for this error:
https://github.com/bundler/bundler/search?q=undefined+method+%60runtime_dependencies%27+for+nil%3ANilClass&type=Issues

If there aren't any reports for this error yet, please create copy and paste the report template above into a new issue. Don't forget to anonymize any private data! The new issue form is located at:
https://github.com/bundler/bundler/issues/new

# with fix
$ bundle viz
# ...
# bunch of output
# ...

/private/tmp/viz-dependency-error/gem_graph.png

[indirect]

I think in this particular case it probably makes sense to just set dependency.prerelease = true, since it allows us to find prerelease specs when we need to find them. Thanks for the in-depth investigation!

[aprescott]

Sounds good! Want me to submit a PR?

[indirect]

Please, thanks!

On Thu, May 7, 2015 at 1:13 PM, Adam Prescott notifications@github.com
wrote:

Sounds good! Want me to submit a PR?

Reply to this email directly or view it on GitHub:
#3621 (comment)

[aprescott]

PR'd! #3629

Has some issues I could use help with, though.

[aprescott]

This has been resolved by e3e9f4a from #3629.