Skip to content

Breaking change: Is.SupersetOf with ValueTuple requires IComparable in NUnit 3.13.2 #3841

Closed

Description

@ghelyar

The following test passed in 3.13.1 and earlier, but fails in 3.13.2.

This is a simplified repro, of course. ValueTuple equality is useful for simplifying more complex comparisons.

[Test]
public void Test()
{
    var a = new S { C = 'a' };
    var b = new S { C = 'b' };
    var c = new S { C = 'c' };
    var d = new S { C = 'd' };

    var actual = new[] { (a, 1), (b, 2), (c, 3), (d, 4) };
    var expected = new[] { (b, 2), (c, 3) };
    Assert.That(actual, Is.SupersetOf(expected));
}

class S
{
    public char C;
}

Here is the stack trace:

System.InvalidOperationException: Failed to compare two elements in the array.
 ---> System.ArgumentException: At least one object must implement IComparable.
   at System.Collections.Comparer.Compare(Object a, Object b)
   at System.Collections.Generic.ObjectComparer`1.Compare(T x, T y)
   at System.ValueTuple`2.CompareTo(ValueTuple`2 other)
   at System.ValueTuple`2.System.IComparable.CompareTo(Object other)
   at System.Collections.Comparer.Compare(Object a, Object b)
   at System.Array.SorterObjectArray.InsertionSort(Int32 lo, Int32 hi)
   at System.Array.SorterObjectArray.IntroSort(Int32 lo, Int32 hi, Int32 depthLimit)
   at System.Array.SorterObjectArray.IntrospectiveSort(Int32 left, Int32 length)
   --- End of inner exception stack trace ---
   at System.Array.SorterObjectArray.IntrospectiveSort(Int32 left, Int32 length)
   at System.Array.Sort(Array keys, Array items, Int32 index, Int32 length, IComparer comparer)
   at System.Collections.ArrayList.Sort(Int32 index, Int32 count, IComparer comparer)
   at System.Collections.ArrayList.Sort()
   at NUnit.Framework.Constraints.CollectionTally..ctor(NUnitEqualityComparer comparer, IEnumerable c)
   at NUnit.Framework.Constraints.CollectionItemsEqualConstraint.Tally(IEnumerable c)
   at NUnit.Framework.Constraints.CollectionSupersetConstraint.Matches(IEnumerable actual)
   at NUnit.Framework.Constraints.CollectionSupersetConstraint.ApplyTo[TActual](TActual actual)
   at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args)
   at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression)

Note that ValueTuple's CompareTo works as expected:

(a, 1).CompareTo((a, 1)) // 0

but trying to use this on the assertion does not help. This still fails, presumably because it's not using this delegate as the comparer for the sorting.

Assert.That(actual, Is.SupersetOf(expected).Using<(S, int)>((t1, t2) => t1.CompareTo(t2)));

Implementing IComparable on the inner type works, but having to implement this all over the place just to make an assertion in the unit tests work, when that assertion used to work fine until a patch change in NUnit is not very nice. The objects are already reference equal.

class S : IComparable
{
    public char C;
    public int CompareTo(object s) => C.CompareTo(((S)s).C);
}

Also note that HashSet<T>.IsSupersetOf still works as expected, so this works:

Assert.That(new HashSet<(S, int)>(actual).IsSupersetOf(new HashSet<(S, int)>(expected)));

but one of the reasons NUnit is so good is that Assert.That(actual, Is.SupersetOf(expected)) generates more useful messages when the assertion fails than "expected true but was false".

Using anonymous types rather than ValueTuples also still works as expected.

Activity

  1. mikkelbu commented on Apr 28, 2021

    @mikkelbu
    Member

    Note that ValueTuple's CompareTo works as expected:
    (a, 1).CompareTo((a, 1)) // 0

    It "only" works as expected, since a is the same reference, so it does not need to compare the values. E.g. (a, 1).CompareTo((b, 1) fails with At least one object must implement IComparable.

    So the logic we use to determine if we can sort the collection (i.e. whether it implements System.IComparable) is not good enough, as ValueTuple<T1,T2> implements IComparable, IComparable<(T1,T2)>, but then fail at runtime. Perhaps we also need to examine the type parameters to see whether they also support comparison (and perform this check recursively). Then in the case above we would not try to sort the lists, as the class is not comparable.

    Alternatively, and perhaps more safe, we should only sort the collection if it consists of primitive types (plus some selected types - like strings ...).

  2. added this to the 3.13.3 milestone on Apr 28, 2021
  3. mikkelbu commented on Jun 16, 2021

    @mikkelbu
    Member

    A possible quick-and-dirty approach could be to have a fallback to the old approach if ArrayList.Sort fails with a InvalidOperationException. Alternatively, we could be more strict about which types we could sider sortable by introducing a whitelist.

  4. stevenaw commented on Jun 20, 2021

    @stevenaw
    Member

    It looks like this will fail only if one of a ValueTuple's contained item's types don't implement IComparable. I like the idea of catching InvalidOperation here and falling back to the old approach rather than to special-case this particular scenario for value tuples.

    I'm going to assign this to myself. I hope to get a PR up in the next few days

  5. self-assigned this
    on Jun 20, 2021
  6. MichaelKetting commented on Jun 20, 2021

    @MichaelKetting

    @stevenaw please note that this also affected Tuple, not just ValueTuple.
    #3877

    I don't mind a quick fix to get this back into working order, but I would not call the lack of IComparable in a tuple's item an edge-case.

  7. stevenaw commented on Jun 20, 2021

    @stevenaw
    Member

    Thanks @MichaelKetting I've added a few tests to ensure it works with tuples.

    I want to clarify that my "special case" comment only refers to an internal optimization made possible when all items are sortable. Tuples and ValueTuples which reference a non-IComparable type should also work after this change in the same manner as in 3.13.1 and earlier

  8. mikkelbu commented on Jul 11, 2021

    @mikkelbu
    Member

    Closed by #3881

  9. added a commit that references this issue on Jul 23, 2021
  10. sherryyshi commented on Oct 4, 2022

    @sherryyshi

    #3881 seems to have been undone by #4048 and now I am seeing the same issue in 3.13.3.

  11. stevenaw commented on Oct 5, 2022

    @stevenaw
    Member

    Sorry to hear you're hitting this @sherryyshi can you give a bit more info and a short reproduction about what you're seeing?
    I'm particularly curious if you're hitting #4095 which affects ImmutableDictionary and other collections which convert the underlying type when ICollection.CopyTo(Array) is called.

  12. curr3nt commented on Jul 8, 2024

    @curr3nt

    I am sorry for posting on an old issue, but @sherryyshi is correct. The problem reappears when:

    • one of Tuple / ValueTuple values is a class instance (not tested with structs)
    • Tuple<Type1, Type2> is wrapped by another class.

    To reproduce, I modified a test from #3881:

    private class S
    {
        public char C;
    }
    
    class Wrapper : Tuple<S, int>
    {
        public Wrapper(S item1, int item2) : base(item1, item2)
        {
        }
    }
    
    [Test]
    public void DoesNotWorkOnWrappedTuples_OneTypeIsntIComparer()
    {
        var a = new S { C = 'a' };
        var b = new S { C = 'b' };
        var c = new S { C = 'c' };
        var d = new S { C = 'd' };
    
        var actual = new[] { new Wrapper(a, 1), new Wrapper(b, 2), new Wrapper(c, 3), new Wrapper(d, 4) };
        var expected = new[] { new Wrapper(b, 2), new Wrapper(c, 3) };
    
        var constraint = new CollectionSupersetConstraint(expected);
        var constraintResult = constraint.ApplyTo(actual);
    
        Assert.That(constraintResult.IsSuccess, Is.True);
    }

    A workaround is to create a new collection and unwrap each tuple back.

  13. manfred-brands commented on Jul 9, 2024

    @manfred-brands
    Member

    @curr3nt The problem here is that your type claims to be IComparable as it derives from Tuple<T1, T2>.

    public class Tuple<T1,T2> : IComparable, IStructuralComparable, IStructuralEquatable, ITuple

    However, because it is not a Tuple, the special code in place in NUnit for tuples isn't triggered.
    NUnit cannot assume just because you derive from it, you didn't override any functionality.

    If you instead aggregate the Tuple instead of deriving from it:

    private class Wrapper
    {
        public Tuple<S, int> MyTuple { get; }
    
        public Wrapper(S item1, int item2)
        {
            MyTuple = Tuple.Create(item1, item2);
        }
    }

    Then the SuperSet code detects your class is not IComparable.
    As it also doesn't implement IEquatable you need to use the UsingPropertiesComparer modifier on the constraint:

    var constraint = new CollectionSupersetConstraint(expected).UsingPropertiesComparer();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

is:bugpri:criticalUse this to indicate a hotfix may be necessary

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions