Skip to content
Permalink
Browse files
drop hash allocations and optimize for success cases
We rarely care what the conflicts on a spec are, just whether or not
there are conflicts.  This commit introduces an allocation-free method
that returns whether or not there are conflicts.  The only case we
actually need to know the conflicts are in the case that an exception is
raised, and in that case we can do the work twice since the program will
exit anyway.  This drops hash allocations.

Benchmark:

```ruby
require 'stackprof'
require 'allocation_tracer'
require 'rubygems/test_case'
require 'rubygems/ext'
require 'rubygems/specification'

class TestGemSpecification < Gem::TestCase
  def test_find_in_unresolved_tree_is_not_exponentiental
    save_loaded_features do
      num_of_pkg = 7
      num_of_version_per_pkg = 3
      packages = (0..num_of_pkg).map do |pkgi|
        (0..num_of_version_per_pkg).map do |pkg_version|
          deps = Hash[(pkgi..num_of_pkg).map { |deppkgi| ["pkg#{deppkgi}", ">= 0"] }]
          new_spec "pkg#{pkgi}", pkg_version.to_s, deps
        end
      end
      base = new_spec "pkg_base", "1", {"pkg0" => ">= 0"}

      Gem::Specification.reset
      install_specs base,*packages.flatten
      base.activate

      require 'benchmark'
      ObjectSpace::AllocationTracer.setup(%i{path line type})
      r = ObjectSpace::AllocationTracer.trace do
        assert_raises(LoadError) { require 'no_such_file_foo' }
      end
      r.sort_by { |k,v| v.first }.each do |k,v|
        p k => v
      end
      p hash_alloc: ObjectSpace::AllocationTracer.allocated_count_table[:T_HASH]
      p :TOTAL => ObjectSpace::AllocationTracer.allocated_count_table.values.inject(:+)
    end
  end
end
```

Before: 1562496 Hash allocations
After: 0 Hash allocations
  • Loading branch information
tenderlove committed Mar 12, 2015
1 parent 9cd7bdc commit c2aebd9b3af82c969711032f1a36ac35d55337cb
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 6 deletions.
@@ -105,7 +105,7 @@ def require path

# Ok, now find a gem that has no conflicts, starting
# at the highest version.
valid = found_specs.select { |s| s.conflicts.empty? }.last
valid = found_specs.reject { |s| s.has_conflicts? }.last

unless valid then
le = Gem::LoadError.new "unable to find a version of '#{names.first}' to activate"
@@ -963,7 +963,7 @@ def self.find_in_unresolved_tree path
specs.reverse_each do |spec|
trails = []
spec.traverse do |from_spec, dep, to_spec, trail|
next unless to_spec.conflicts.empty?
next if to_spec.has_conflicts?
trails << trail if to_spec.contains_requirable_file? path
end

@@ -1563,6 +1563,16 @@ def conflicts
conflicts
end

##
# Return true if there are possible conflicts against the currently loaded specs.

def has_conflicts?
self.runtime_dependencies.any? { |dep|
spec = Gem.loaded_specs[dep.name]
spec and not spec.satisfies_requirement? dep
}
end

##
# The date this gem was created. Lazily defaults to the current UTC date.
#
@@ -2154,10 +2164,8 @@ def check_version_conflict other # :nodoc:
# Check the spec for possible conflicts and freak out if there are any.

def raise_if_conflicts # :nodoc:
conf = self.conflicts

unless conf.empty? then
raise Gem::ConflictError.new self, conf
if has_conflicts? then
raise Gem::ConflictError.new self, conflicts
end
end

12 comments on commit c2aebd9

@indirect
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any chance that you will be available to do things this amazing for Bundler, sometime in the future? Because this is amazing.

@tenderlove
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd do it for bundler too, but someone else identified this bottleneck. I'm actually just doing superficial perf improvements until we can figure out a better algorithm. :(

@jeremy
Copy link

@jeremy jeremy commented on c2aebd9 Mar 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍 ======<() ~ ♪ ~♫

@indirect
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like saving 5 million allocations is more than superficial, but okay! I'd love a better algorithm, too. 😁

@drbrain
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If RubyGems is better at resolving why not have bundler use RubyGems for resolution?

@drbrain
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If -> When and if

@indirect
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stripe just funded a from-scratch, beautifully documented resolver that is now shared between Bundler and CocoaPods. Why not use it in Rubygems as well?

@drbrain
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not indeed!

@sferik
Copy link
Member

@sferik sferik commented on c2aebd9 Mar 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ 💛 💚 💙 💜

@arthurnn
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@schneems
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥 👍 ❤️

@parkr
Copy link
Contributor

@parkr parkr commented on c2aebd9 Mar 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 🎊 💗

Please sign in to comment.