proposal: iter/sorted: functions on sorted iterators

[jba]

I propose adding a package of functions that operate on iterators whose values are sorted.

Sorted sequences can arise in a number of ways:

• From a slice that has been sorted.
• From the keys of an ordered map, such as github.com/google/btree or the one in proposal: container: add tree-based ordered.Map/Set #60630.
• From an external source, like a database with sorted keys.
• Naturally, as in the line numbers of a file or the times of a monotonic clock.

Common operations on pairs of sorted sequences include merging them into a single sorted sequence, as well as the set operations union, intersection and set difference. I propose a package with these operations, with the API given below.

As is common in packages like slices, there are two functions for each operation, one using the natural ordering of a type and one that accepts a comparison function. We could also consider adding functions that operate on the keys of iter.Seq2s, bringing along the corresponding values. I don't know if those sequences would arise enough to make that worthwhile. We could reconsider adding the Seq2 functions if and when we add ordered maps.

API

package sorted

Package sorted provides operations over iterators whose values are sorted.

FUNCTIONS

func Intersect[T cmp.Ordered](s1, s2 iter.Seq[T]) iter.Seq[T]
    Intersect returns an iterator over all elements of s1 that are also in s2,
    in the same order. Each element of s1 appears only once in the result.
    Both s1 and s2 must be sorted.

func IntersectFunc[T any](s1, s2 iter.Seq[T], cmp func(T, T) int) iter.Seq[T]
    IntersectFunc returns an iterator over all elements of s1 that are also in
    s2, in the same order. Each element of s1 appears only once in the result.
    Both s1 and s2 must be sorted according to cmp.

func Merge[T cmp.Ordered](s1, s2 iter.Seq[T]) iter.Seq[T]
    Merge returns an iterator over all elements of s1 and s2, in the same order.
    Both s1 and s2 must be sorted.

func MergeFunc[T any](s1, s2 iter.Seq[T], cmp func(T, T) int) iter.Seq[T]
    MergeFunc returns an iterator over all elements of s1 and s2, in the same
    order. Both s1 and s2 must be sorted according to cmp.

func Subtract[T cmp.Ordered](s1, s2 iter.Seq[T]) iter.Seq[T]
    Subtract returns an iterator over all elements of s1 that are not in s2,
    in the same order. Each element of s1 appears only once in the result.
    Both s1 and s2 must be sorted.

func SubtractFunc[T any](s1, s2 iter.Seq[T], cmp func(T, T) int) iter.Seq[T]
    SubtractFunc returns an iterator over all elements of s1 that are not in s2,
    in the same order. Each element of s1 appears only once in the result.
    Both s1 and s2 must be sorted according to cmp.

func Union[T cmp.Ordered](s1, s2 iter.Seq[T]) iter.Seq[T]
    Union returns an iterator over all elements of s1 and s2, in the same order.
    Each element appears only once in the result. Both s1 and s2 must be sorted.

func UnionFunc[T any](s1, s2 iter.Seq[T], cmp func(T, T) int) iter.Seq[T]
    UnionFunc returns an iterator over all elements of s1 and s2, in the same
    order. Each element appears only once in the result. Both s1 and s2 must be
    sorted according to cmp.

There is a working implementation at github.com/jba/sorted.

[gabyhelp]

Related Issues and Documentation

• proposal: container: add tree-based ordered.Map/Set #60630
• proposal: container/set: new package to provide a generic set type #69230
• proposal: x/exp/xiter: new package with iterator adapters #61898

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[jimmyfrasche]

Is there any difference between this Merge and the one in #61898?

If this package gets added, it seems like a good place for the sequence equivalent for Compact: #67441

[jba]

Is there any difference between this Merge and the one in #61898?

No, I just forgot about the other one.

If this package gets added, it seems like a good place for the sequence equivalent for Compact: #67441

Agreed.

[isgj]

I agree with the API but shouldn't this package come after #61898 (considering #61898 will go to exp first) ?

IMHO

1. the operators proposed in the other proposal are more common, used more frequently
2. the ones here will work only with a subset of sequences

[jub0bs]

I'm guessing that iter/sorted functions don't check that their iterator arguments are sorted, and I'm worried that some people will pass them unsorted iterators; the type system can't help us, in this case, since both sorted and unsorted iterators have type iter.Seq[T].

If this proposal gets accepted, its documentation should encourage other packages that provide sorted iterators to document them as such.

[jba]

I omitted that from the spec so that the implementation could choose to do it if we wanted it to. We don't want to document that it does, because that would force the functions to run slower.

In general, Go doesn't perform expensive checks like this. For example, sort functions that take a less or cmp function as argument don't check that those functions behave properly.

My hope is that by putting these in a separate package from other iterator functions, users would be reminded of the requirements at the call site: sorted.Union(a, b) and so on.

[Merovius]

You could also introduce package sorted; type Seq[E any] iter.Seq[E], requiring at least an explicit conversion.

[mateusz834]

@Merovius sadly you can still pass an func(yield(T) bool) to it without explicit conversion.

[Merovius]

Yes, I believe most iterators will be iter.Seq[E], though. Note that the general recommendation is to have methods All() iter.Seq[E], not All(yield func(E) bool), for example.

In any case, I’m not saying it’s perfect, or even that we necessarily should do it. But it’s IMO the strongest reasonable thing, has no runtime cost and will catch the overwhelming majority of mistakes.

(also, sorted.Seq[E] IMO reads nicely and it’s more appropriate to have the invariant of being sorted be part of the type than part of the function or package name)