CollectionAssert.AreEquivalent is extremely slow

[nvborisenko]

var lines1 = new List<string> { };
var lines2 = new List<string> { };

for (int i = 0; i < 10000; i++)
{
  lines1.Add(i.ToString());
  lines2.Add(i.ToString());
}

CollectionAssert.AreEquivalent(lines1, lines2);

Verification of 10000 string elements takes 10 seconds. Verification of 100k elements may take more than 30 minutes (cannot wait until it finishes). This behavior takes a place in NUnit 3.10 and not reproducible in 3.9

[CharliePoole]

Several of us did changes to collection assert in that period. I'll take a look to see if one of mine caused this.

[mikkelbu]

Actually, I think that my change in #2600 has caused the problem (I've not had time to check it yet, as I'm halfway a VS upgrade). In the case above we will have to run through the entire remaining collection for every match we want to remove. Before my change we would find each match immediately as the first element.

[jnm2]

The ideal situation might be to implement IEqualityComparer in all our internal comparer classes so they can be used with HashSet and Dictionary. AreEquivalent would use HashSet.SetEquals with O(n + m) complexity (the counts of the two collections).

[CharliePoole]

I'm wondering if the various collection constraints should possibly be separated in their implementation, separately optimized and only then possibly recombined to remove duplication.

[jnm2]

That sounds like a good idea. Either way, the collection equivalence constraint itself will probably need to use HashSet.SetEquals eventually to get to the standard O(n+m) people expect. It won't be able to do that unless all our internal NUnit equality comparers implement IEqualityComparer.

[CharliePoole]

Implementing IEqualityComparer means either (1) forgetting about tolerance or (2) making tolerance a constructor argument for comparers and using a new one either for every comparison or perhaps caching a few of them. Could get complicated.

[jnm2]

That's an interesting question. Is it even meaningful to test for set equality with tolerance? I don't know what set equality means without transitive element equality comparisons, and if you have tolerance, you don't have transitive equality. My attempt at searching didn't turn anything up. It would be cool to know.

We'd only need to adapt to IEqualityComparer when using a hash table. We'd only use a hash table for operations that compare two collections. A hash table in theory requires transitive equality comparisons and transitive hash codes. This is what the IEqualityComparer contract demands, so if it's inherently possible to use a hash table for the comparison, it's inherently possible to adapt our element comparer to IEqualityComparer.

Let's say there is a hypothetical comparison we want NUnit to do between two collections which is not inherently possible to losslessly adapt to IEqualityComparer. If that is the case, that would mean the comparison cannot even in theory make use of a hash table. We would then have to fall back to something slower. But before we worry about that, does such a comparison meaningfully exist?

[herebebeasties]

I think set equality with tolerance does makes sense. Essentially, "is it possible to match every item in set A with one or more items in set B using equality function F such that no item in set B is left unmatched". But I think that's always going to be O(n*m) in the general case.

Even items that implement IComparable such that you could potentially sort them first and then compare the streams pairwise might have out-of-order matches between the sets, so that wouldn't be good enough. To really speed it up you'd need the ability to map your items into some kind of lower-precision ordered map (e.g. Math.Round() and search in adjacent buckets). None of that would really work with a general IEqualityComparer<T> and you'd still need to flag every item that's within tolerance and then see if there are any left over afterwards.

I don't think I'd expect NUnit to try solve any of that for me out of the box.

ETA: As @jnm2 says, all uses of "set" in this answer should really be replaced with "bag", as there may be duplicate items.

[jnm2]

@herebebeasties Good point. I was talking about set equality, but Is.EquivalentTo goes beyond set equality since that compares the number of repeated values. (The conceptual equivalent of comparing two Dictionary<T, int> countsByValue rather than comparing two HashSet<T> uniqueValues.)

We'd have to have an algorithm that not only did the O(m*n) initial mapping but also proceeded by trying every combination of overlaps to see if it could match the items 1:1. It seems like that would have factorial complexity on top of the n*m, depending on how connected the mapping is (worse with larger tolerances). This logic doesn't sound familiar, so I wonder if NUnit is currently returning wrong results for EquivalentTo in combination with Tolerance. Separate issue to fix, though.

How does it sound to implement the logic to first determine whether the element comparison is transitive, and if so, use HashSet<>.SetEquals Dictionary?

[jnm2]

Also, any takers? If this interests you, please jump in!
If not, I'll do it as one of my next PRs.