Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

slices: new standard library package based on x/exp/slices #57433

Closed
ianlancetaylor opened this issue Dec 21, 2022 · 78 comments
Closed

slices: new standard library package based on x/exp/slices #57433

ianlancetaylor opened this issue Dec 21, 2022 · 78 comments
Milestone

Comments

@ianlancetaylor
Copy link
Contributor

In #45955 I proposed a new slices package in the standard library. After considerable discussion there and in discussion #47203 we settled on an API and added the experimental package golang.org/x/exp/slices.

We now have experience with this package in practice. pkg.go.dev reports that it is imported by 1,541 other packages. I propose that it is now time to move this package into the standard library for the 1.21 release.

Doing this doesn't mean that we can't add more functions to the standard library slices package later. Any such additions should be suggested in their own separate proposal.

The specific API that we will put into the standard library appears below.

This description of the API refers to constraints.Ordered in a few places; if the constraints package is not present in the standard library, that reference will be replaced by an unexported constraint with the same set of types.

Note that we should either accept or reject #57348 before we accept this proposal. If we accept #57348, the appropriate change will be made here.

I'm not aware of any other proposal to modify the existing slices package API. There are a few proposals to add new functions to the slices package (at least #52434, #53987, #54768, #56353); those proposals can move from x/exp/slices to just plain slices.

Proposed API:

// Equal reports whether two slices are equal: the same length and all
// elements equal. If the lengths are different, Equal returns false.
// Otherwise, the elements are compared in increasing index order, and the
// comparison stops at the first unequal pair.
// Floating point NaNs are not considered equal.
func Equal[E comparable](s1, s2 []E) bool

// EqualFunc reports whether two slices are equal using a comparison
// function on each pair of elements. If the lengths are different,
// EqualFunc returns false. Otherwise, the elements are compared in
// increasing index order, and the comparison stops at the first index
// for which eq returns false.
func EqualFunc[E1, E2 any](s1 []E1, s2 []E2, eq func(E1, E2) bool) bool

// Compare compares the elements of s1 and s2.
// The elements are compared sequentially, starting at index 0,
// until one element is not equal to the other.
// The result of comparing the first non-matching elements is returned.
// If both slices are equal until one of them ends, the shorter slice is
// considered less than the longer one.
// The result is 0 if s1 == s2, -1 if s1 < s2, and +1 if s1 > s2.
// Comparisons involving floating point NaNs are ignored.
func Compare[E constraints.Ordered](s1, s2 []E) int

// CompareFunc is like Compare but uses a comparison function
// on each pair of elements. The elements are compared in increasing
// index order, and the comparisons stop after the first time cmp
// returns non-zero.
// The result is the first non-zero result of cmp; if cmp always
// returns 0 the result is 0 if len(s1) == len(s2), -1 if len(s1) < len(s2),
// and +1 if len(s1) > len(s2).
func CompareFunc[E1, E2 any](s1 []E1, s2 []E2, cmp func(E1, E2) int) int

// Index returns the index of the first occurrence of v in s,
// or -1 if not present.
func Index[E comparable](s []E, v E) int

// IndexFunc returns the first index i satisfying f(s[i]),
// or -1 if none do.
func IndexFunc[E any](s []E, f func(E) bool) int

// Contains reports whether v is present in s.
func Contains[E comparable](s []E, v E) bool

// ContainsFunc reports whether at least one
// element e of s satisfies f(e).
func ContainsFunc[E any](s []E, f func(E) bool) bool

// Insert inserts the values v... into s at index i,
// returning the modified slice.
// In the returned slice r, r[i] == v[0].
// Insert panics if i is out of range.
// This function is O(len(s) + len(v)).
func Insert[S ~[]E, E any](s S, i int, v ...E) S

// Delete removes the elements s[i:j] from s, returning the modified slice.
// Delete panics if s[i:j] is not a valid slice of s.
// Delete modifies the contents of the slice s; it does not create a new slice.
// Delete is O(len(s)-j), so if many items must be deleted, it is better to
// make a single call deleting them all together than to delete one at a time.
// Delete might not modify the elements s[len(s)-(j-i):len(s)]. If those
// elements contain pointers you might consider zeroing those elements so that
// objects they reference can be garbage collected.
func Delete[S ~[]E, E any](s S, i, j int) S

// Replace replaces the elements s[i:j] by the given v, and returns the
// modified slice. Replace panics if s[i:j] is not a valid slice of s.
func Replace[S ~[]E, E any](s S, i, j int, v ...E) S

// Clone returns a copy of the slice.
// The elements are copied using assignment, so this is a shallow clone.
func Clone[S ~[]E, E any](s S) S

// Compact replaces consecutive runs of equal elements with a single copy.
// This is like the uniq command found on Unix.
// Compact modifies the contents of the slice s; it does not create a new slice.
// When Compact discards m elements in total, it might not modify the elements
// s[len(s)-m:len(s)]. If those elements contain pointers you might consider
// zeroing those elements so that objects they reference can be garbage collected.
func Compact[S ~[]E, E comparable](s S) S

// CompactFunc is like Compact but uses a comparison function.
func CompactFunc[S ~[]E, E any](s S, eq func(E, E) bool) S

// Grow increases the slice's capacity, if necessary, to guarantee space for
// another n elements. After Grow(n), at least n elements can be appended
// to the slice without another allocation. If n is negative or too large to
// allocate the memory, Grow panics.
func Grow[S ~[]E, E any](s S, n int) S

// Clip removes unused capacity from the slice, returning s[:len(s):len(s)].
func Clip[S ~[]E, E any](s S) S

// Sort sorts a slice of any ordered type in ascending order.
// Sort may fail to sort correctly when sorting slices of floating-point
// numbers containing Not-a-number (NaN) values.
// Use slices.SortFunc(x, func(a, b float64) bool {return a < b || (math.IsNaN(a) && !math.IsNaN(b))})
// instead if the input may contain NaNs.
func Sort[E constraints.Ordered](x []E)

// SortFunc sorts the slice x in ascending order as determined by the less function.
// This sort is not guaranteed to be stable.
//
// SortFunc requires that less is a strict weak ordering.
// See https://en.wikipedia.org/wiki/Weak_ordering#Strict_weak_orderings.
func SortFunc[E any](x []E, less func(a, b E) bool)

// SortStableFunc sorts the slice x while keeping the original order of equal
// elements, using less to compare elements.
func SortStableFunc[E any](x []E, less func(a, b E) bool)

// IsSorted reports whether x is sorted in ascending order.
func IsSorted[E constraints.Ordered](x []E) bool

// IsSortedFunc reports whether x is sorted in ascending order, with less as the
// comparison function.
func IsSortedFunc[E any](x []E, less func(a, b E) bool) bool

// BinarySearch searches for target in a sorted slice and returns the position
// where target is found, or the position where target would appear in the
// sort order; it also returns a bool saying whether the target is really found
// in the slice. The slice must be sorted in increasing order.
func BinarySearch[E constraints.Ordered](x []E, target E) (int, bool)

// BinarySearchFunc works like BinarySearch, but uses a custom comparison
// function. The slice must be sorted in increasing order, where "increasing" is
// defined by cmp. cmp(a, b) is expected to return an integer comparing the two
// parameters: 0 if a == b, a negative number if a < b and a positive number if
// a > b.
func BinarySearchFunc[E any](x []E, target E, cmp func(E, E) int) (int, bool)
@Merovius
Copy link
Contributor

Merovius commented Dec 21, 2022

Note that we should either accept or reject #57348 before we accept this proposal. If we accept #57348, the appropriate change will be made here.

Out of interest: Is there code which would compile with the current version but would stop compiling with my proposal? I can't really think of how that would be the case, because any code using slices.BinarySearchFunc should just be inferred to two identical type arguments, no? [edit] I mean, I said on that proposal that it's not backwards compatible, as if it's self-explanatory, so I must've thought of something, but I can't remember what… [/edit]

@fzipp
Copy link
Contributor

fzipp commented Dec 21, 2022

it is imported by 1,541 other packages

Usage statistics per function would be interesting. If there's a function that isn't used at all, that could be a sign that it's not necessary.

@Merovius
Copy link
Contributor

I don't know if it has been brought up, but Contains fits into the family None/Any/All - if we had all three, it would be Any. I've recently found myself needing All a couple of times. I bring this up not to suggest adding the others, but because I can't think of how I'd name the others if we stick with the name Contains.

I also find it a bit of a rough edge that it's sometimes less func(a, b T) bool and sometimes cmp func(a, b T) int. If I want a slice of things which can both be sorted and searched (and let's be honest, anything that needs to be searched will have to be sorted first) I'll have to write two, mostly redundant functions.

@fzipp
Copy link
Contributor

fzipp commented Dec 21, 2022

I don't know if it has been brought up, but Contains fits into the family None/Any/All - if we had all three, it would be Any.

I think it was discussed in #53983

@Merovius
Copy link
Contributor

Fair enough.

@ianlancetaylor
Copy link
Contributor Author

Is there code which would compile with the current version but would stop compiling with my proposal?

I had to think about it, but I think the answer is yes:

    f := slices.BinarySearchFunc[int]

We can't infer the second type argument in the absence of any actual arguments.

@ghost
Copy link

ghost commented Dec 22, 2022

It's a pity we can't have func SortFunc[E any, EP E | *E](x []E, less func(a, b EP) bool). It can be kind of expensive to pass largish elements by value.

@earthboundkid
Copy link
Contributor

Can we get a decision on #52427 before this is merged, to nail down the constraints package issue?

@andig
Copy link
Contributor

andig commented Dec 22, 2022

It's a pity we can't have func SortFunc[E any, EP E | *E](x []E, less func(a, b EP) bool). It can be kind of expensive to pass largish elements by value.

Isn‘t only the header passed by value? Then size shouldn‘t make a difference regarding performance.

@ericlagergren
Copy link
Contributor

It's a pity we can't have func SortFunc[E any, EP E | *E](x []E, less func(a, b EP) bool). It can be kind of expensive to pass largish elements by value.

Isn‘t only the header passed by value? Then size shouldn‘t make a difference regarding performance.

Not the slice, the comparison function.

@fzipp
Copy link
Contributor

fzipp commented Dec 22, 2022

Why is there a stable variant of SortFunc, but not of Sort?

Edit: I think I found the answer. It's because only value types can satisfy constraints.Ordered.

@Merovius
Copy link
Contributor

Merovius commented Dec 22, 2022

@fzipp Arguably there should be.

As usually, the special case are floating point numbers, where distinguishable elements can still compare as equal. Distinguishing the stable/unstable sorting result requires using math.Signbit in this case or might require math.Float64bits for NaN, but they can be distinguished.

But it seems very much a corner case and people who care can then still result to SortStableFunc.

For all other types satisfying constraints.Ordered, the stable/unstable sorting result will always be indistinguishable.

@gophun
Copy link

gophun commented Dec 22, 2022

We now have experience with this package in practice.

I would have liked to hear more experience reports from people who have used this package. But maybe the silence just means they're happy so far.

@Merovius
Copy link
Contributor

@gophun

I've been using the slices package over the last year or so whenever I thought of it. In particular, I've been using it a bunch for Advent of Code over the last three weeks. That's a small sample and it's code for a specific kind of problem, but OTOH it's all fresh code from scratch, so it's probably the most representative sample I have at hand for how I write code "right now". The TL;DR is that I use or could use almost all of its API and I use it quite frequently (which also feels true for my day job).

Stats/Details Grepping through the source, I call:
  4 Clone
  4 Index
  2 Sort
  3 SortFunc
  3 SortStableFunc

I would have used slices.BinarySearchFunc 5 times, if #57348 was accepted. That's ~20 calls in ~2700 SLOC, which is a decent amount. It's also one of the most-imported Go project packages in that codebase:

  1 	"crypto/sha1"
  1 	"encoding/base64"
  1 	"encoding/binary"
  1 	"encoding/json"
  1 	"golang.org/x/net/publicsuffix"
  1 	"image"
  1 	"image/color"
  1 	"image/png"
  1 	"io/fs"
  1 	"net/http"
  1 	"net/http/cookiejar"
  1 	"net/url"
  1 	"reflect"
  1 	"regexp"
  1 	"sort"
  1 	"sync"
  2 	"context"
  2 	"path/filepath"
  2 	"testing"
  2 	"time"
  2 	"unicode/utf8"
  3 	"bufio"
  4 	"bytes"
  5 	"flag"
  5 	"golang.org/x/exp/constraints"
  6 	"strconv"
  9 -->	"golang.org/x/exp/slices"
 10 	"io"
 10 	"strings"
 12 	"errors"
 21 	"log"
 22 	"os"
 30 	"fmt"

Looking at the API, I probably would have used Delete, Insert and Replace for one problem and Clip in at least one place, if I had thought of them. I also know that I've written versions of Compact and Grow a couple of times in the past.

APIs I haven't used and don't remember needing are Compare*, Contains* and IsSorted*, but there's also significant personal preference in that.

One thing I've been missing a couple of times are Map and Filter, but I think it makes sense to delay those until we solve iteration.

Overall, I find myself reaching for slices quite frequently and I use most of its API occasionally, with a decent bias towards the *Func variants. I think an interesting contrast is golang.org/x/exp/maps, which I don't seem to be using much, if at all.

@adonovan
Copy link
Member

Can we also add the in-place Reverse([]T) function? It's a common three-liner.

@adonovan
Copy link
Member

We could also produce a set of vet analyzers that can suggest fixes that would safely transform pre-generic code using explicit allocations/loops into calls to these new functions.

@DeedleFake
Copy link

We could also produce a set of vet analyzers that can suggest fixes that would safely transform pre-generic code using explicit allocations/loops into calls to these new functions.

That feels like it could get a bunch of false positives. That could be annoying if they show up in go test runs.

@adonovan
Copy link
Member

We could also produce a set of vet analyzers that can suggest fixes that would safely transform pre-generic code using explicit allocations/loops into calls to these new functions.

That feels like it could get a bunch of false positives. That could be annoying if they show up in go test runs.

Sorry, I meant vet analyzer here as a shorthand for golang.org/x/tools/go/analysis analyzers, along with a standalone tool that one could apply as desired to your codebase; I didn't mean they would actually be in the vet tool.

@AndrewHarrisSPU
Copy link

AndrewHarrisSPU commented Dec 22, 2022

I would have liked to hear more experience reports from people who have used this package. But maybe the silence just means they're happy so far.

I think the slices package has chosen the right focus. Go slices have quirks; it's really nice to have the operations slices provides, with implementation that is plainly tuned to (in-)variants particular to the slice data structure.

Eventually, I think it's really natural to want backed-by-slices, generics-oriented versions of what e.g. container/heap, container/ring do, or stacks/queues, or multi-dimensional slices, etc., etc., but it feels right that slices doesn't get into this.

@Aceix
Copy link

Aceix commented Dec 22, 2022

This is good but what of a C++ STL equivalent, but with chain-able methods for functional programming feel

@earthboundkid
Copy link
Contributor

The one function I find myself still regularly missing is the equivalent of maps.DeleteFunc. I understand why slices.Filter isn’t in the package (yet?), but an in place filter would be helpful.

@ianlancetaylor
Copy link
Contributor Author

I think it will be best if suggestions for new functions in the slices package go into separate proposals. Whether or not we add new functions to the slices package does not affect whether the slices package will go into the standard library. Similarly, whether or not the slices package goes into the standard library does not affect whether we add new functions. Thanks.

@dsnet
Copy link
Member

dsnet commented Dec 22, 2022

Usage statistics per function would be interesting. If there's a function that isn't used at all, that could be a sign that it's not necessary.

@fzipp, scanning the latest module of all publicly known modules on 2022-10-01, the usage stats are as follows:

Function Count
Contains 1481
Clone 531
Sort 433
Equal 404
SortFunc 336
Index 190
IndexFunc 166
Delete 145
Insert 82
BinarySearch 60
SortStableFunc 42
EqualFunc 41
Clip 38
Compact 37
IsSorted 36
Grow 29
BinarySearchFunc 26
Compare 20
IsSortedFunc 18
Replace 10
CompactFunc 5
CompareFunc 3
ContainsFunc 1

I find it ironic that the most used function is Contains, while the least used function is it's sibling ConrtainsFunc.

@ianlancetaylor
Copy link
Contributor Author

ContainsFunc was only added to the package on December 5, so it's not that surprising that it's not used very often.

@hherman1
Copy link

Replace seems borderline underused for inclusion here.

@bobg
Copy link

bobg commented Feb 9, 2023

Probably a little late for this suggestion, but how about Python-style negative indexes, as in https://pkg.go.dev/github.com/bobg/go-generics/slices?

@earthboundkid
Copy link
Contributor

@bobg, See #53510.

@seankhliao
Copy link
Member

negative indices would be a language change, most recently declined in #33359

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/467417 mentions this issue: slices: new package

@DeedleFake
Copy link

DeedleFake commented Feb 10, 2023

@seankhliao I'm apathetic about negative indices myself, but I think he meant just for the functions in this package, not in general. Then you could do things like slices.Delete(s, -3, -2) to remove the penultimate element, for example.

@beoran
Copy link

beoran commented Feb 11, 2023

I think negative indexes at're an example of wanting to be "too clever" . It's better to use the length of the slice.

l :=len(s)
slices.Delete(s, len-3, len-2)

Expresses more clearly what we mean

@natefinch
Copy link
Contributor

I just wanted to say thank you to whoever added the ,bool to the return value of the binary search functions. Figuring out how to use sort.Search correctly was an ordeal.

@ianlancetaylor
Copy link
Contributor Author

I believe it was first suggested by @rsc in #50340 (comment).

@ghost
Copy link

ghost commented Feb 18, 2023

// BinarySearch searches for target in a sorted slice and returns the position
// where target is found, or the position where target would appear in the
// sort order; it also returns a bool saying whether the target is really found
// in the slice. The slice must be sorted in increasing order.
func BinarySearch[E constraints.Ordered](x []E, target E) (int, bool)

increasing order

Non-decreasing, maybe?

johanbrandhorst pushed a commit to Pryz/go that referenced this issue Feb 22, 2023
This copies parts of x/exp/slices into the standard library.
We omit all functions that depend on constraints.Ordered,
and the Func variants of all such functions. In particular this
omits the various Sort and Search functions.

Fixes golang#57433

Change-Id: I3c28f4c2e6bd1e3c9ad70e120a0dd68065388f77
Reviewed-on: https://go-review.googlesource.com/c/go/+/467417
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Eli Bendersky <eliben@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/492576 mentions this issue: all: use slices.Equal to simplify code

gopherbot pushed a commit that referenced this issue May 10, 2023
#57433 added slices.Equal, using it can reduce the amount of code

Change-Id: I70d14b6c4c24da641a34ed36c900d9291033f526
Reviewed-on: https://go-review.googlesource.com/c/go/+/492576
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Run-TryBot: shuang cui <imcusg@gmail.com>
@dmitshur dmitshur modified the milestones: Backlog, Go1.21 May 30, 2023
eric pushed a commit to fancybits/go that referenced this issue Sep 7, 2023
This copies parts of x/exp/slices into the standard library.
We omit all functions that depend on constraints.Ordered,
and the Func variants of all such functions. In particular this
omits the various Sort and Search functions.

Fixes golang#57433

Change-Id: I3c28f4c2e6bd1e3c9ad70e120a0dd68065388f77
Reviewed-on: https://go-review.googlesource.com/c/go/+/467417
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Eli Bendersky <eliben@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
eric pushed a commit to fancybits/go that referenced this issue Sep 7, 2023
This copies parts of x/exp/slices into the standard library.
We omit all functions that depend on constraints.Ordered,
and the Func variants of all such functions. In particular this
omits the various Sort and Search functions.

Fixes golang#57433

Change-Id: I3c28f4c2e6bd1e3c9ad70e120a0dd68065388f77
Reviewed-on: https://go-review.googlesource.com/c/go/+/467417
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Eli Bendersky <eliben@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
@golang golang locked and limited conversation to collaborators May 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests