Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add assertion for checking the contents of a collection, ignoring element order #93

Closed
wants to merge 5 commits into from

4 participants

@tkareine

No description provided.

tkareine added some commits
@tkareine tkareine Add assert_includes_all
This assertion is commonly needed when testing the contents of a
collection, but not caring about the order of the elements in the
collection.
dc0b808
@tkareine tkareine Add must_include_all counterpart for assert_includes_all 1aea6ef
@zenspider
Owner

I like the idea (with caveats, see below), but it looks like it only works with arrays, not collections in general. If it is just arrays, then this looks like it would suffice:

assert_empty a - b
assert_empty b - a

caveats: I'm not sure how much clarity this provides.

I have a number of tests that sort before doing an assert_equal, but they're pretty self-descriptive. Eg:

    deps = spec.dependencies.sort_by { |dep| dep.name }

    expected = [
      ["hoe",  :development, "~> #{Hoe::VERSION.sub(/\.\d+$/, '')}"],
      ["rdoc", :development, "~> 3.10"],
    ]

    assert_equal expected, deps.map { |dep|
      [dep.name, dep.type, dep.requirement.to_s]
    }

vs:

    deps = spec.dependencies

    expected = [
      ["hoe",  :development, "~> #{Hoe::VERSION.sub(/\.\d+$/, '')}"],
      ["rdoc", :development, "~> 3.10"],
    ]

    assert_equal_unordered expected, deps.map { |dep|
      [dep.name, dep.type, dep.requirement.to_s]
    }

Which one is clearer? They seem almost exactly the same to me.

Also, I don't like the name. assert_include_all is stunted at the very least. assert_include_all_of seems a bit better, but both of these names suggest the possibility of a superset, not that they're strictly the same except for order. That's why I named it assert_equal_unordered in my example above.

@zenspider zenspider was assigned
@phiggins
Collaborator

Also, I don't like the name.

FWIW, shoulda has this called assert_same_elements. I always thought that was a succinct name for this behavior.

@tkareine

I agree that assert_includes_all is not a good name. I don't think assert_same_elements is good either, because minitest already has assert_same, which would imply relationship. I like @zenspider's suggestion (assert_equal_unordered) best so far. I renamed the assertion.

assert_equal_unordered should work with any Enumerable. The assertion converts the collection to an Array for internal operation. Enumerable is guaranteed to convert the collection to Array with to_a. The call to dup is needed to avoid mutating the original collection, if the assertion is called with an Array as the first argument.

The elements of the collection need not implement <=>. I believe this is the most useful feature of the assertion, because a collection with custom elements does not need to be sorted externally before asserting its contents.

@zenspider
Owner
@tkareine
@tenderlove
Owner

This assertion seems really application specific. I can't imagine a common situation where I would need this. Is it really something that should be in minitest?

Could you provide real world examples where this would simplify and clarify the test cases? Maybe there is a different abstraction we need.

@phiggins
Collaborator

@tenderlove assert_equal expected, actual.sort is a pretty common idiom.

Ruby:

$ grep -R "assert_equal.*\.sort" test/ | wc -l
     150

Jruby (likely some overlap with ruby):

 grep -R "assert_equal.*\.sort" test/ | wc -l
     253

Rails:

$ grep -R "assert_equal.*\.sort" **/test/ | wc -l
     113
@tenderlove
Owner

@phiggins sure, but is adding a new assertion to minitest worth saving a call to sort?

[aaron@higgins rails (master)]$ grep -R "assert_equal.*\.sort" **/test/ | wc -l
     114
[aaron@higgins rails (master)]$ grep -R "assert_equal" **/test/ | wc -l
   11658
[aaron@higgins rails (master)]$

Less than 1%, and that's only counting calls to assert_equal. I'm still dubious that this is a "common idiom".

EDIT: I forgot to mention that I think adding the explicit sort may make the test more clear. I'm not sure this is common, nor am I sure it makes the tests read more clearly.

@phiggins
Collaborator

is adding a new assertion to minitest worth saving a call to sort?
Less than 1%

Probably not, when you put it that way.

@tenderlove
Owner

Oh @phiggins, you cave too easily. :trollface:

@phiggins
Collaborator

I don't really have horse in this race, just trying to help out. :cake:

@phiggins
Collaborator

Serious Business

@tkareine

@tenderlove, I think needing to sort a collection for asserting its contents is noise for test readability. Depending on how you get the expected elements, you might have to sort them as well.

It is even worse if the elements cannot be compared with <=>, like the booleans. Then you have to figure out a scheme for calculating a key for each element for ordering. And that scheme depends on the situation.

@zenspider
Owner
@tkareine

@zenspider, the example in your first comment doesn't get much better with assert_equal_unordered, and I don't think that can be improved.

However, let's presume I have a test class where I want to have the main expectation in one place, and use selected parts of that in the actual assertions. In particular, I want to write the expectation in the order what is shown to the user of the application. This helps maintaining the test. I have modified your original example to reflect this:

require "minitest/autorun"

class AssertEqualUnorderedJustification < MiniTest::Unit::TestCase
  # specified in the order as shown to the user of the application
  EXPECTED_DEPS = [
    ["hoe",  :runtime,     "~> 2.13.1"],
    ["rdoc", :runtime,     "= 3.9"],
    ["rdoc", :runtime,     "= 3.8"],
    ["hoe",  :development, "~> 2.13.3"],
    ["hoe",  :development, "~> 2.13.2"],
    ["hoe",  :development, "~> 2.13.0"],
    ["rdoc", :development, "= 3.12"]
  ]

  def EXPECTED_DEPS.type_of(type); self.select { |_, t| t == type }; end

  def setup
    # read from IO, order not guaranteed
    @deps = [
      ["rdoc", :runtime,     "= 3.9"],
      ["hoe",  :development, "~> 2.13.2"],
      ["hoe",  :development, "~> 2.13.3"],
      ["rdoc", :runtime,     "= 3.8"],
      ["hoe",  :runtime,     "~> 2.13.1"],
      ["hoe",  :development, "~> 2.13.0"],
      ["rdoc", :development, "= 3.12"],
    ]

    def @deps.type_of(type); self.select { |_, t| t == type }; end
  end

  def test_development_dependencies_with_assert_equal
    # requires both calls to #sort
    assert_equal @deps.type_of(:development).sort, EXPECTED_DEPS.type_of(:development).sort
  end

  def test_development_dependencies_with_assert_equal_unordered
    assert_equal_unordered @deps.type_of(:development), EXPECTED_DEPS.type_of(:development)
  end

  def test_runtime_dependencies_with_assert_equal
    # requires both calls to #sort
    assert_equal @deps.type_of(:runtime).sort, EXPECTED_DEPS.type_of(:runtime).sort
  end

  def test_runtime_dependencies_with_assert_equal_unordered
    assert_equal_unordered @deps.type_of(:runtime), EXPECTED_DEPS.type_of(:runtime)
  end
end

Granted, this does not happen often and you can work around it. But I have come across it every now and then, so that I dislike copying the assertion method to every project where I need it.

@tkareine

Any thoughts on this additional assertion and my arguments for it? In short, I think it allows writing tests for checking the contents of a collection easier, without requiring the collection (the actual or the excepted) to be sorted or the elements in it to be Comparable.

Most of the time, you can work around this by simply sorting both collections and using assert_equals. I like minitest to be small, so I understand if the addition is not worth it. What do you think?

@zenspider
Owner

  def test_assert_equal_unordered_keeps_equality_contract
    @assertion_count = 3

    @tc.assert_equal           [1.0, 2, 3], [1, 2, 3], "WOO"
    @tc.assert_equal_unordered [1.0, 2, 3], [1, 2, 3], "BOO"
  end

I don't like the idea of assert_equal being the contract maker. assert_equal is saying that the two objects respond true to #==. assert_equal_unordered is not saying that and is saying something completely different. This is the only test that fails on my non-destructive version of assert_equal_unordered and at this point I'm not sure whether I should punt on this test or my impl....

Open the debate...

@tkareine

I'd like that assert_equals_unordered would not surprise the user. To me, the name of the assertion relates it to assert_equals, giving the user impression that both assertions behave in a similar way. That's why I'm favoring my original destructive implementation here, because it relies on #== for equality.

That said, I like @zenspider's non-destructive implementation as well. If you choose that, you should document that equality is in terms of #eql?, not #==.

@zenspider
Owner
@zenspider
Owner

OK. After much discussion over here, we've decided to address the "surprise" issue with more documentation or to pull the feature all-together (none of us are convinced that it'll clean up our tests very much). The doco I just finished is:

+    # Fails unless +a+ contains the same contents as +b+, regardless
+    # of order.
+    #
+    #    assert_equal_unordered %w[a a b c], %w[a b c a] # pass
+    #
+    # NOTE: This uses Hash#== to determine collection equivalence, as
+    # such, do not expect it to behave the same as +assert_equal+.
+    #
+    #    assert_equal [1], [1.0]                         # pass
+    #    assert_equal({ 1 => true }, { 1.0 => true })    # fail
+    #    assert_equal_unordered [1], [1.0]               # fail
@tkareine

That's great, thank you! I'm fine with either option you take.

@zenspider
Owner

I just added seattlerb/minitest-unordered. Please poke at it. In particular, I didn't touch README.txt at all.

@zenspider zenspider closed this
@tkareine

Thanks. I already sent a pull request for updating README.txt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 9, 2012
  1. @tkareine

    Add assert_includes_all

    tkareine authored
    This assertion is commonly needed when testing the contents of a
    collection, but not caring about the order of the elements in the
    collection.
  2. @tkareine
Commits on Feb 10, 2012
  1. @tkareine
  2. @tkareine
  3. @tkareine
This page is out of date. Refresh to see the latest.
View
9 lib/minitest/spec.rb
@@ -306,6 +306,15 @@ module MiniTest::Expectations
infect_an_assertion :assert_includes, :must_include, :reverse
##
+ # See MiniTest::Assertions#assert_equal_unordered
+ #
+ # collection.must_equal_unordered elements
+ #
+ # :method: must_equal_unordered
+
+ infect_an_assertion :assert_equal_unordered, :must_equal_unordered, :reverse
+
+ ##
# See MiniTest::Assertions#assert_instance_of
#
# obj.must_be_instance_of klass
View
19 lib/minitest/unit.rb
@@ -248,6 +248,25 @@ def assert_includes collection, obj, msg = nil
end
##
+ # Fails unless +collection+ contains equal +elements+, ignoring the order
+ # of the elements.
+
+ def assert_equal_unordered collection, elements, msg = nil
+ msg = message(msg) {
+ "Expected #{mu_pp(collection)} to contain equal elements of #{mu_pp(elements)}"
+ }
+ assert_respond_to collection, :to_a
+ assert_respond_to elements, :size
+ remaining = collection.dup.to_a
+ assert remaining.size == elements.size, msg
+ elements.each do |e|
+ index = remaining.index e
+ remaining.delete_at index if index
+ end
+ assert remaining.empty?, msg
+ end
+
+ ##
# Fails unless +obj+ is an instance of +cls+.
def assert_instance_of cls, obj, msg = nil
View
17 test/test_minitest_spec.rb
@@ -64,6 +64,20 @@ def assert_triggered expected = "blah", klass = MiniTest::Assertion
end
end
+ it "needs to be sensible about must_equal_unordered order" do
+ @assertion_count += 8 # must_equal_unordered is 4 assertions
+
+ [1, 2, 3].must_equal_unordered([1, 2, 3]).must_equal true
+
+ assert_triggered "Expected [1, 2] to contain equal elements of [1, 2, 3]." do
+ [1, 2].must_equal_unordered [1, 2, 3]
+ end
+
+ assert_triggered "msg.\nExpected [1, 2, 4] to contain equal elements of [1, 2, 3]." do
+ [1, 2, 4].must_equal_unordered [1, 2, 3], "msg"
+ end
+ end
+
it "needs to catch an expected exception" do
@assertion_count = 2
@@ -120,6 +134,7 @@ def assert_triggered expected = "blah", klass = MiniTest::Assertion
must_be_within_delta
must_be_within_epsilon
must_equal
+ must_equal_unordered
must_include
must_match
must_output
@@ -128,7 +143,7 @@ def assert_triggered expected = "blah", klass = MiniTest::Assertion
must_send
must_throw)
- bad = %w[not raise throw send output be_silent]
+ bad = %w[not raise throw send output be_silent equal_unordered]
expected_wonts = expected_musts.map { |m| m.sub(/^must/, 'wont') }
expected_wonts.reject! { |m| m =~ /wont_#{Regexp.union(*bad)}/ }
View
82 test/test_minitest_unit.rb
@@ -883,6 +883,79 @@ def test_assert_includes_triggered
assert_equal expected, e.message
end
+ def test_assert_equal_unordered_when_comparable_elements
+ @assertion_count = 4
+
+ @tc.assert_equal_unordered [1, 2, 3], [2, 3, 1]
+ end
+
+ def test_assert_equal_unordered_when_not_comparable_elements
+ @assertion_count = 4
+
+ @tc.assert_equal_unordered [true, false, true], [true, true, false]
+ end
+
+ def test_assert_equal_unordered_keeps_equality_contract
+ @assertion_count = 4
+
+ # sanity checks
+ assert_equal (1 == 1.0), true
+ assert_equal 1.eql?(1.0), false
+
+ @tc.assert_equal_unordered [1.0, 2, 3], [1, 2, 3]
+ end
+
+ def test_assert_equal_unordered_when_enumerable_actual
+ @assertion_count = 4
+
+ es = Class.new do
+ include Enumerable
+
+ def initialize
+ @elems = [true, false, true]
+ end
+
+ def each
+ @elems.each { |e| yield e }
+ end
+ end.new
+
+ @tc.assert_equal_unordered es, [true, true, false]
+ end
+
+ def test_assert_equal_unordered_triggered_when_actual_has_more_elements_than_expected
+ @assertion_count = 4
+
+ e = @tc.assert_raises MiniTest::Assertion do
+ @tc.assert_equal_unordered [true, true], [true]
+ end
+
+ expected = "Expected [true, true] to contain equal elements of [true]."
+ assert_equal expected, e.message
+ end
+
+ def test_assert_equal_unordered_triggered_when_actual_has_less_elements_than_expected
+ @assertion_count = 4
+
+ e = @tc.assert_raises MiniTest::Assertion do
+ @tc.assert_equal_unordered [true], [true, true]
+ end
+
+ expected = "Expected [true] to contain equal elements of [true, true]."
+ assert_equal expected, e.message
+ end
+
+ def test_assert_equal_unordered_triggered_when_actual_has_different_elements_than_expected
+ @assertion_count = 5
+
+ e = @tc.assert_raises MiniTest::Assertion do
+ @tc.assert_equal_unordered [true, false, true], [false, false, true]
+ end
+
+ expected = "Expected [true, false, true] to contain equal elements of [false, false, true]."
+ assert_equal expected, e.message
+ end
+
def test_assert_instance_of
@tc.assert_instance_of String, "blah"
end
@@ -1260,10 +1333,11 @@ def test_class_asserts_match_refutes
methods = MiniTest::Assertions.public_instance_methods
methods.map! { |m| m.to_s } if Symbol === methods.first
- ignores = %w(assert_block assert_no_match assert_not_equal
- assert_not_nil assert_not_same assert_nothing_raised
- assert_nothing_thrown assert_output assert_raise
- assert_raises assert_send assert_silent assert_throws)
+ ignores = %w(assert_block assert_equal_unordered assert_no_match
+ assert_not_equal assert_not_nil assert_not_same
+ assert_nothing_raised assert_nothing_thrown
+ assert_output assert_raise assert_raises
+ assert_send assert_silent assert_throws)
asserts = methods.grep(/^assert/).sort - ignores
refutes = methods.grep(/^refute/).sort - ignores
Something went wrong with that request. Please try again.