Combine Linq optimisations [WIP]

[JonHanna]

A few different optimisations to Linq added recently can be usefully combined with each other.

Combine IArrayProvider and IListProvider

Anything that can be one can be the other, so merge the two interfaces.

Add ToList support to OrderedPartition.

Have IList optimised result of Skip() partitionable.

Optimisation of Skip() for IList<T> sources from #4551 fits with optimisations of Skip() and Take() for other sources from #2401.

Combine the approaches, extending how the result of Skip() on a list handles subsequent operations.

Use partitioning on list-based select iterators.

The creation of SelectListIterator above allows for partitioning to be used with the list-based Select() iterators, improving some subsequent operations on them.

Remove unused paths in Set.

Set never has Add called after Remove. Remove paths and fields only necessary in that case.

Add debug-only check on this assumption, so any new uses that break this assumption are flagged to the developer.

Implement IIListProvider on Distinct() and Union().

With those changes to Set it's now easy to give these methods an optimised ToArray and ToList().

Tests of IIListProvider on empty GroupBy results.

Noticed in the process of the above that this case isn't covered by tests though it does have very slightly different behaviour.

[JonHanna]

Also added IIListProvider support to ReverseIterator. Not clear why I didn't do that before.

[JonHanna]

Further: Have Select() on an IPartition implement IPartition

With an IPartition-implementing result for Select, the advantages of IPartition results can carry through to results.

In particular the sequence .OrderBy(…).Select(…).Skip(pageOffset).Take(pageSize) can pass through the partial-sorting capability.

Introducing the TryGetElementAt, TryGetFirst and TryGetLast methods to IPartition this required allows for dropping ElementAt/OrDefault, First/OrDefault and Last/OrDefault methods from that interface. This reduces assembly size, number of methods to jit and repetition in code at a slight cost to execution speed. This is the opposite direction of much that's been done on this assembly but keeping the separate First() etc. gives only a small constant performance advantage that doesn't scale with sequence size and so seems hard to justify.

[JonHanna]

This has grown legs. Would it be handier to consider the first few commits first, and then submit those that build on them after that?

[stephentoub]

Thanks, @JonHanna. Yeah, I think it'd probably easier if it could be broken apart a bit, especially since GitHub isn't showing a diff view for the src changes to Enumerable.cs.

[JonHanna]

Well now that we seem to have a positive decision on #2363, the thing at the very start of the list should be handling that, which you're on, so I'm going to close this and wait until that is in to look at this again.