sort: make sorting easier, add Slice, SliceStable, SliceIsSorted, reflect.Swapper #16721

Closed
bradfitz opened this Issue Aug 16, 2016 · 158 comments

Comments

@bradfitz
Member

bradfitz commented Aug 16, 2016

EDIT: the proposal has changed later in this thread. See #16721 (comment) and #16721 (comment)

I propose we make sorting slices a first class concept in Go 1.8.

I think everybody appreciates the cute sort.Interface type at this point, but the reality is that:

  • nobody really sorts anything except slices (the handful of exceptions can keep doing what they're doing)
  • the Len and Swap functions are redundant. They're always the same for slices.
  • naming a new type (e.g. "widgetsByName") is tedious to many (including me; I still loathe it after 6+ years)
  • making a new type and three methods at the top-level (because it has to have methods, so it can't be a local type) is more difficult to read, since I'd prefer my Less function inline with my code, in context.

I think we should add to package sort at least:

package sort

// Slice sorts the provided slice using the function less.
// The sort is not stable.
// If slice is not a slice, Slice panics.
func Slice(slice interface{}, less func(i, j int) bool)

And maybe at the same time, or in the future:

// SliceSorter returns an sorter for the provided slice, suitable
// for with Reverse or Stable.
// If slice is not a slice, SliceSorter panics.
func SliceSorter(slice interface{}, less func(i, j int) bool) Interface

The assumption is that the user's less function would close over the data to be sorted.

For speed, this would not be purely reflect-based (except for one call to to get the slice length). The implementation would generate a custom swapper as needed for the type of the slice elements such that it's compatible with the GC & write barriers. I did a prototype of this which shows that it's as fast as the typical way.

I think there's plenty of evidence in the Github BigQuery dataset that we're making sort way too hard for users.

/cc @griesemer @robpike @ianlancetaylor @adg @broady

@bradfitz bradfitz added the Proposal label Aug 16, 2016

@bradfitz bradfitz added this to the Proposal milestone Aug 16, 2016

@griesemer

This comment has been minimized.

Show comment
Hide comment
@griesemer

griesemer Aug 16, 2016

Contributor

I'd be ok with something like this. It's pragmatic, small, and probably would reduce a lot of sort-related boilerplate.

Contributor

griesemer commented Aug 16, 2016

I'd be ok with something like this. It's pragmatic, small, and probably would reduce a lot of sort-related boilerplate.

@cznic

This comment has been minimized.

Show comment
Hide comment
@cznic

cznic Aug 16, 2016

Contributor

I think there's plenty of evidence in the Github BigQuery dataset that we're making sort way too hard to users.

I don't have access to the mentioned above dataset, but let me guess: The need-to-be-sorted type declarations and its associated methods declarations constitutes less than 1‰ of code in that corpus. Perhaps far less. If that is true then I don't think this proposal is justified. Also, using reflection in sorting, however small it is, feels to me like encouraging bad practices.

Contributor

cznic commented Aug 16, 2016

I think there's plenty of evidence in the Github BigQuery dataset that we're making sort way too hard to users.

I don't have access to the mentioned above dataset, but let me guess: The need-to-be-sorted type declarations and its associated methods declarations constitutes less than 1‰ of code in that corpus. Perhaps far less. If that is true then I don't think this proposal is justified. Also, using reflection in sorting, however small it is, feels to me like encouraging bad practices.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@cznic, you do have access: https://github.com/blog/2201-making-open-source-data-more-available

We'll have to disagree on the metrics at which this is subjectively worthwhile.

Member

bradfitz commented Aug 16, 2016

@cznic, you do have access: https://github.com/blog/2201-making-open-source-data-more-available

We'll have to disagree on the metrics at which this is subjectively worthwhile.

@josharian josharian changed the title from Proposal: first-class support for sorting slices to proposal: first-class support for sorting slices Aug 16, 2016

@atdiar

This comment has been minimized.

Show comment
Hide comment
@atdiar

atdiar Aug 16, 2016

No opinion on this.

Only thing I would say is that, sorting a slice is not exactly to be seen as simply sorting an array.
Reason being that if a slice has multiple subslices, sorting one slice will somehow change the others (in a deep value kind of way).

Or should we decide that sorting a slice returns a completely new slice ? (and thus allocate ?)

Just a data point. (I guess you discussed a similar issue when adding the append function but, just to make sure).

atdiar commented Aug 16, 2016

No opinion on this.

Only thing I would say is that, sorting a slice is not exactly to be seen as simply sorting an array.
Reason being that if a slice has multiple subslices, sorting one slice will somehow change the others (in a deep value kind of way).

Or should we decide that sorting a slice returns a completely new slice ? (and thus allocate ?)

Just a data point. (I guess you discussed a similar issue when adding the append function but, just to make sure).

@mackross

This comment has been minimized.

Show comment
Hide comment
@mackross

mackross Aug 16, 2016

As the primary force behind golang at our organization, this and no generic max/min function are the two things I find embarrassingly hard to explain. I'm not sure what it's like at Shoreline Blvd but without fail when an engineer joins our team their mouth drops through the floor at their first sort. Subjectively, I would love this in 1.8.

mackross commented Aug 16, 2016

As the primary force behind golang at our organization, this and no generic max/min function are the two things I find embarrassingly hard to explain. I'm not sure what it's like at Shoreline Blvd but without fail when an engineer joins our team their mouth drops through the floor at their first sort. Subjectively, I would love this in 1.8.

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Aug 16, 2016

Contributor

I'm fine with adding something to the sort package, although of course func (i, j int) bool is not the function signature that most people expect.

Contributor

ianlancetaylor commented Aug 16, 2016

I'm fine with adding something to the sort package, although of course func (i, j int) bool is not the function signature that most people expect.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@ianlancetaylor, the benefit of i, j int is that it lets you get at your []T's T or *T, if the values are too large to efficiently copy. And it fits with the existing Interface.Less signature. Plus it's not like we can do better, can we?

Member

bradfitz commented Aug 16, 2016

@ianlancetaylor, the benefit of i, j int is that it lets you get at your []T's T or *T, if the values are too large to efficiently copy. And it fits with the existing Interface.Less signature. Plus it's not like we can do better, can we?

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Aug 16, 2016

Contributor

Oh, sure, I understand why you suggested it. I'm just commenting that it will still surprise people.

Contributor

ianlancetaylor commented Aug 16, 2016

Oh, sure, I understand why you suggested it. I'm just commenting that it will still surprise people.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Aug 16, 2016

Member

As much as this is complained about and as many slice types as I have sorted this has never bothered me, so I don't see the point. Boilerplate is never fun, but this pittance is far below the threshold of annoyance.

A go generate command to spit out Len and Swap, or even just an editor macro, seems more inline with the Tao of Go here.

Member

jimmyfrasche commented Aug 16, 2016

As much as this is complained about and as many slice types as I have sorted this has never bothered me, so I don't see the point. Boilerplate is never fun, but this pittance is far below the threshold of annoyance.

A go generate command to spit out Len and Swap, or even just an editor macro, seems more inline with the Tao of Go here.

@ainar-g

This comment has been minimized.

Show comment
Hide comment
@ainar-g

ainar-g Aug 16, 2016

Contributor

I'm with @jimmyfrasche here. Why can't this be a go generateable tool?

Contributor

ainar-g commented Aug 16, 2016

I'm with @jimmyfrasche here. Why can't this be a go generateable tool?

@natefinch

This comment has been minimized.

Show comment
Hide comment
@natefinch

natefinch Aug 16, 2016

Contributor

If it's basically as fast as sort.Inferface, and is just an addition to the stdlib, not a language change.... why would anyone ever say no to this? Yes, it's annoying that it has to use interface{} that we try not to encourage people to use... but what other choice do we have? No one has come up with a better suggestion so far.

Put me down for a +1.

Contributor

natefinch commented Aug 16, 2016

If it's basically as fast as sort.Inferface, and is just an addition to the stdlib, not a language change.... why would anyone ever say no to this? Yes, it's annoying that it has to use interface{} that we try not to encourage people to use... but what other choice do we have? No one has come up with a better suggestion so far.

Put me down for a +1.

@artursapek

This comment has been minimized.

Show comment
Hide comment
@artursapek

artursapek Aug 16, 2016

How would SliceSorter work? Wouldn't it have to define a new type, and return it? That's not possible in Go afaik.

How would SliceSorter work? Wouldn't it have to define a new type, and return it? That's not possible in Go afaik.

@pbnjay

This comment has been minimized.

Show comment
Hide comment
@pbnjay

pbnjay Aug 16, 2016

I'd prefer usage of a []interface{} parameter (which would make the reflection unnecessary), but since that doesn't intuitively work on all slice types I know we're stuck with the proposed signature. +1 for saving the headaches anyway though.

To the go generate comments: It's still possible to use that method, this is only adding to the stdlib. But for newbies and rapid development use, go generate can be just as much of a pain...

pbnjay commented Aug 16, 2016

I'd prefer usage of a []interface{} parameter (which would make the reflection unnecessary), but since that doesn't intuitively work on all slice types I know we're stuck with the proposed signature. +1 for saving the headaches anyway though.

To the go generate comments: It's still possible to use that method, this is only adding to the stdlib. But for newbies and rapid development use, go generate can be just as much of a pain...

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

How would SliceSorter work?

It would be a private type implementing sort.Interface. It would return Interface, per the example above.

Member

bradfitz commented Aug 16, 2016

How would SliceSorter work?

It would be a private type implementing sort.Interface. It would return Interface, per the example above.

@riannucci

This comment has been minimized.

Show comment
Hide comment
@riannucci

riannucci Aug 16, 2016

@pbnjay I don't think []interface{} would be very convenient; it's not possible to go from e.g. []*MyThing -> []interface{} without allocating a new slice and copying every element (which would be way more annoying than writing out the Less/Len/Swap).

It's a bit unfortunate that it has to be func(i, j int) bool though, because frequently you'd have MyThing.Less(other *MyThing) bool, which means that you'd end up doing:

sort.Slice(mySlice, func(i, j int) bool { return mySlice[i].Less(mySlice[j]) })

But oh well :). I'd still take that over declaring a new type+3 methods just to be able to sort stuff.

@pbnjay I don't think []interface{} would be very convenient; it's not possible to go from e.g. []*MyThing -> []interface{} without allocating a new slice and copying every element (which would be way more annoying than writing out the Less/Len/Swap).

It's a bit unfortunate that it has to be func(i, j int) bool though, because frequently you'd have MyThing.Less(other *MyThing) bool, which means that you'd end up doing:

sort.Slice(mySlice, func(i, j int) bool { return mySlice[i].Less(mySlice[j]) })

But oh well :). I'd still take that over declaring a new type+3 methods just to be able to sort stuff.

@pbnjay

This comment has been minimized.

Show comment
Hide comment
@pbnjay

pbnjay Aug 16, 2016

@riannucci yup - that's what i meant by "doesn't intuitively work on all slice types" - maybe in Go 2 we can get a generic slice-interface type!

pbnjay commented Aug 16, 2016

@riannucci yup - that's what i meant by "doesn't intuitively work on all slice types" - maybe in Go 2 we can get a generic slice-interface type!

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@riannucci, in my experience, the Less method doesn't already exist somewhere. This is for ad-hoc sorts where you want to write the Less function inline, in context with your existing code:

   sort.Slice(people, func(i, j int) bool { return people[i].Name < people[j].Name })
Member

bradfitz commented Aug 16, 2016

@riannucci, in my experience, the Less method doesn't already exist somewhere. This is for ad-hoc sorts where you want to write the Less function inline, in context with your existing code:

   sort.Slice(people, func(i, j int) bool { return people[i].Name < people[j].Name })
@freeformz

This comment has been minimized.

Show comment
Hide comment
@freeformz

freeformz Aug 16, 2016

Contributor

How about func Slice(slice interface{}, l int, less func(i, j int) bool), where l is the length up to which the slice is sorted. This means the common usage would just be calling Slice with len([]T), but saves the reflect. I'd wager it's possibly not worth it?

Edit: Nevermind, my bad, there will still be reflection.

Contributor

freeformz commented Aug 16, 2016

How about func Slice(slice interface{}, l int, less func(i, j int) bool), where l is the length up to which the slice is sorted. This means the common usage would just be calling Slice with len([]T), but saves the reflect. I'd wager it's possibly not worth it?

Edit: Nevermind, my bad, there will still be reflection.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@freeformz, you already have to do the reflect a bit anyway, so it doesn't save you anything to pass the length explicitly.

Member

bradfitz commented Aug 16, 2016

@freeformz, you already have to do the reflect a bit anyway, so it doesn't save you anything to pass the length explicitly.

@myitcv

This comment has been minimized.

Show comment
Hide comment
@myitcv

myitcv Aug 16, 2016

Member

Following on from @jimmyfrasche's comment (but my follow up may indeed be the subject of another issue/a separate go-nuts thread)

A go generate command to spit out Len and Swap, or even just an editor macro, seems more inline with the Tao of Go here.

On the go generate suggestion.

This raises the question of whether to distribute core, oft-used programs that are go generate-ers.

As things stand, every go generate-er needs to be retrieved via go get (or some other means). If instead some core go generate-ers were distributed as part of a release then:

  • no further go get is required to do core things (like slice sorting)
  • we encourage people towards go generate
  • we avoid any reflection (per @cznic)
  • ...

There are of course some downsides here:

  • how to name these core-generators (to be relatively sure of avoiding PATH clashes with existing executable programs, we might need rather verbose names like goGenSort)
  • another step in the process of learning Go
  • ...
Member

myitcv commented Aug 16, 2016

Following on from @jimmyfrasche's comment (but my follow up may indeed be the subject of another issue/a separate go-nuts thread)

A go generate command to spit out Len and Swap, or even just an editor macro, seems more inline with the Tao of Go here.

On the go generate suggestion.

This raises the question of whether to distribute core, oft-used programs that are go generate-ers.

As things stand, every go generate-er needs to be retrieved via go get (or some other means). If instead some core go generate-ers were distributed as part of a release then:

  • no further go get is required to do core things (like slice sorting)
  • we encourage people towards go generate
  • we avoid any reflection (per @cznic)
  • ...

There are of course some downsides here:

  • how to name these core-generators (to be relatively sure of avoiding PATH clashes with existing executable programs, we might need rather verbose names like goGenSort)
  • another step in the process of learning Go
  • ...
@klingtnet

This comment has been minimized.

Show comment
Hide comment
@klingtnet

klingtnet Aug 16, 2016

@mackross +1 for the missing max/min but Go also lacks an absolute value function for integers, which is also hard to explain.

@mackross +1 for the missing max/min but Go also lacks an absolute value function for integers, which is also hard to explain.

@zephyr

This comment has been minimized.

Show comment
Hide comment
@zephyr

zephyr Aug 16, 2016

Shouldn’t there also be a stable counterpart? Maybe even with a variadic signature like

func SliceStable(slice interface{}, less ...func(i, j int) bool)

so that one could do

SliceStable(people, byName, byAge)

to sort people first by name and else (when they have the same name/are equivalent) by age?

zephyr commented Aug 16, 2016

Shouldn’t there also be a stable counterpart? Maybe even with a variadic signature like

func SliceStable(slice interface{}, less ...func(i, j int) bool)

so that one could do

SliceStable(people, byName, byAge)

to sort people first by name and else (when they have the same name/are equivalent) by age?

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Aug 16, 2016

Member

This only simplifies a trivial unimportant amount of code so I don't see it as more than, at best, a minor annoyance, regardless of frequency.

I'm sure Brad's implementation is a negligible hit on performance compared to the current way, but it would have to be used in function bodies to close over the slice like:

func something() {
  //...
  sort.Slice(things, func(i, j int) bool) {
     //...
   })
  //...
}

so you have to pay the cost of allocating the closure and building the generic sorter on each invocation of something, when you don't have to pay anything defining the methods yourself, other than the usual indirections interfaces involve. That's going to be negligible to the cost of sorting anything for all but the smallest of n, and, if it shows up in profiling, it's simple enough to rewrite it the old way, but this sort of code is generally discouraged by the stdlib.

If this is going for pure convenience, I'd prefer the version @ianlancetaylor hinted at where the comparator is an interface containing a func(a, b T) bool (though the runtime hit from that would certainly be less negligible, even modulo potential copying overhead, and it's even further afield of encouraged practice).

Maybe I'm just uncomfortable that this is at a weird level between convenient and practical that results in this awkward situation where you have to close over a slice that's being permuted under the covers.

All that said, if this goes in, I'll use it—I'm lazy and hypocritical!

I don't have any problem with this existing—it's the inclusion in the stdlib that gives me pause. I have absolutely zero reservations about this being a go gettable library: people could even start using it today. The excellent go4.org or even golang.org/x seem like fine places to put this, and if it gets used a lot without issues moving it to the stdlib would be fine with me.

Member

jimmyfrasche commented Aug 16, 2016

This only simplifies a trivial unimportant amount of code so I don't see it as more than, at best, a minor annoyance, regardless of frequency.

I'm sure Brad's implementation is a negligible hit on performance compared to the current way, but it would have to be used in function bodies to close over the slice like:

func something() {
  //...
  sort.Slice(things, func(i, j int) bool) {
     //...
   })
  //...
}

so you have to pay the cost of allocating the closure and building the generic sorter on each invocation of something, when you don't have to pay anything defining the methods yourself, other than the usual indirections interfaces involve. That's going to be negligible to the cost of sorting anything for all but the smallest of n, and, if it shows up in profiling, it's simple enough to rewrite it the old way, but this sort of code is generally discouraged by the stdlib.

If this is going for pure convenience, I'd prefer the version @ianlancetaylor hinted at where the comparator is an interface containing a func(a, b T) bool (though the runtime hit from that would certainly be less negligible, even modulo potential copying overhead, and it's even further afield of encouraged practice).

Maybe I'm just uncomfortable that this is at a weird level between convenient and practical that results in this awkward situation where you have to close over a slice that's being permuted under the covers.

All that said, if this goes in, I'll use it—I'm lazy and hypocritical!

I don't have any problem with this existing—it's the inclusion in the stdlib that gives me pause. I have absolutely zero reservations about this being a go gettable library: people could even start using it today. The excellent go4.org or even golang.org/x seem like fine places to put this, and if it gets used a lot without issues moving it to the stdlib would be fine with me.

@twotwotwo

This comment has been minimized.

Show comment
Hide comment
@twotwotwo

twotwotwo Aug 16, 2016

Taking a key func that's allowed to return one of a list of types ((u)int, []byte, string) makes it look a bit more like sort functions in other languages that take a key func, and would let you later drop in faster sorts specific to the key types without adding APIs. Lots of downsides make that unpalatable (keyFunc'd be an interface{} in the signature, trickier internally, overhead of Less calling keyFunc twice, and can't write a custom Less to sort by two fields or whatever). Maybe someone sees a better variation of that idea that I don't :) so throwing it out there anyhow.

Taking a key func that's allowed to return one of a list of types ((u)int, []byte, string) makes it look a bit more like sort functions in other languages that take a key func, and would let you later drop in faster sorts specific to the key types without adding APIs. Lots of downsides make that unpalatable (keyFunc'd be an interface{} in the signature, trickier internally, overhead of Less calling keyFunc twice, and can't write a custom Less to sort by two fields or whatever). Maybe someone sees a better variation of that idea that I don't :) so throwing it out there anyhow.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@jimmyfrasche,

so you have to pay the cost of allocating the closure

Even today with sort.Sort you have to pay the cost of putting a slice into an interface value, since a slice (1 pointer & two ints) is not 0 or 1 pointers (which is all that goes into an interface allocation-free). So it's not significantly different.

I'd prefer to keep this issue focused on the proposal and not the implementation, though.

Member

bradfitz commented Aug 16, 2016

@jimmyfrasche,

so you have to pay the cost of allocating the closure

Even today with sort.Sort you have to pay the cost of putting a slice into an interface value, since a slice (1 pointer & two ints) is not 0 or 1 pointers (which is all that goes into an interface allocation-free). So it's not significantly different.

I'd prefer to keep this issue focused on the proposal and not the implementation, though.

@broady

This comment has been minimized.

Show comment
Hide comment
@broady

broady Aug 16, 2016

Member

When I read the title, I thought you were going to propose something like a built-in:

var s []t
sort(s, func(a, b t) bool {
  return a < b
})

But this is certainly more pragmatic. Saying that 1% of programs touch sort isn't a very good argument against including this. This is scoped to the sort package, so programs that aren't sorting won't care about it. Even more reason to include it.

Member

broady commented Aug 16, 2016

When I read the title, I thought you were going to propose something like a built-in:

var s []t
sort(s, func(a, b t) bool {
  return a < b
})

But this is certainly more pragmatic. Saying that 1% of programs touch sort isn't a very good argument against including this. This is scoped to the sort package, so programs that aren't sorting won't care about it. Even more reason to include it.

@oyi812

This comment has been minimized.

Show comment
Hide comment
@oyi812

oyi812 Aug 16, 2016

I'd prefer to tolerate something more essential and less ugly bloating the standard packages.

oyi812 commented Aug 16, 2016

I'd prefer to tolerate something more essential and less ugly bloating the standard packages.

@nsf

This comment has been minimized.

Show comment
Hide comment
@nsf

nsf Aug 16, 2016

Can we teach compiler to handle both cases:

var s []int
sort(s, func(a, b int) bool { return a < b; })
var s []GiantType
sort(s, func(a, b *GiantType) bool { return a.prio < b.prio; })

Also not sure if it's worth worrying about copy mechanics. Perhaps compiler could elide copies somehow? Inline the closure? If it's a pure function, why not. And another point is that if you worry about copying, sort does a lot of swaps anyway, so maybe change data structure.

But anyways, I don't like the idea of having indices if we can do generics in the compiler itself. Most built-in functions act as generic functions.

nsf commented Aug 16, 2016

Can we teach compiler to handle both cases:

var s []int
sort(s, func(a, b int) bool { return a < b; })
var s []GiantType
sort(s, func(a, b *GiantType) bool { return a.prio < b.prio; })

Also not sure if it's worth worrying about copy mechanics. Perhaps compiler could elide copies somehow? Inline the closure? If it's a pure function, why not. And another point is that if you worry about copying, sort does a lot of swaps anyway, so maybe change data structure.

But anyways, I don't like the idea of having indices if we can do generics in the compiler itself. Most built-in functions act as generic functions.

@nsf

This comment has been minimized.

Show comment
Hide comment
@nsf

nsf Aug 16, 2016

Sorry for semicolons, I do a lot of javascript lately ("semicolon free" language).

nsf commented Aug 16, 2016

Sorry for semicolons, I do a lot of javascript lately ("semicolon free" language).

@nsf

This comment has been minimized.

Show comment
Hide comment
@nsf

nsf Aug 16, 2016

Another thought: recognize simple known types (strings, ints, floats) and use default comparison operator (less than):

var s []int
sort(s)

That's kind of obvious though.

nsf commented Aug 16, 2016

Another thought: recognize simple known types (strings, ints, floats) and use default comparison operator (less than):

var s []int
sort(s)

That's kind of obvious though.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche Aug 16, 2016

Member

@bradfitz I mentioned that, but you're right that I should have edited that portion out. I was trying to convey that the performance hit, while negligible, would have to be paid more often by design, but I ended up hemming and hawing to the point of incoherence. I apologize. It's not a terribly important concern anyway.

Biggest issue:

Why should this be in the stdlib? You could throw it up on go4 right now and move it to the stdlib later. People who want to use it could start using it today.

Member

jimmyfrasche commented Aug 16, 2016

@bradfitz I mentioned that, but you're right that I should have edited that portion out. I was trying to convey that the performance hit, while negligible, would have to be paid more often by design, but I ended up hemming and hawing to the point of incoherence. I apologize. It's not a terribly important concern anyway.

Biggest issue:

Why should this be in the stdlib? You could throw it up on go4 right now and move it to the stdlib later. People who want to use it could start using it today.

@twotwotwo

This comment has been minimized.

Show comment
Hide comment
@twotwotwo

twotwotwo Aug 16, 2016

@jimmyfrasche Something like it has been at https://github.com/bradfitz/slice for a while.

@jimmyfrasche Something like it has been at https://github.com/bradfitz/slice for a while.

@justinfx

This comment has been minimized.

Show comment
Hide comment
@justinfx

justinfx Aug 16, 2016

I really like this idea. I'm also one of those that have almost never written anything different for 2 of the 3 sort interface methods for slices. And also have found it a bit embarrassing having to explain to new Go devs how much boilerplate has to be written to sort a slice. Usually they then ask why they can't just call a sort function with a single key function, like in other languages.

I agree that because it's a localized addition to the sort package that only makes doing a thing more straightforward, that there is 100% benefit here. People can still use the existing approach or use the new shorter approach when it makes sense. Let's fix warts.

I really like this idea. I'm also one of those that have almost never written anything different for 2 of the 3 sort interface methods for slices. And also have found it a bit embarrassing having to explain to new Go devs how much boilerplate has to be written to sort a slice. Usually they then ask why they can't just call a sort function with a single key function, like in other languages.

I agree that because it's a localized addition to the sort package that only makes doing a thing more straightforward, that there is 100% benefit here. People can still use the existing approach or use the new shorter approach when it makes sense. Let's fix warts.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@twotwotwo, I don't recommend using github.com/bradfitz/slice. I never updated it to be compatible (safely) with Go 1.5's GC. But yes, essentially something like that. I should update it and move it to go4.org. Still, I believe this is common enough to warrant being in the standard library and would improve the readability of code.

@nsf, let's stay on topic. Nobody (except perhaps you) is proposing language changes. This bug is about a library change proposal only. If you want to discuss a hypothetical language change, the best place is probably golang-nuts@, since it's out of scope for Go 1.

Member

bradfitz commented Aug 16, 2016

@twotwotwo, I don't recommend using github.com/bradfitz/slice. I never updated it to be compatible (safely) with Go 1.5's GC. But yes, essentially something like that. I should update it and move it to go4.org. Still, I believe this is common enough to warrant being in the standard library and would improve the readability of code.

@nsf, let's stay on topic. Nobody (except perhaps you) is proposing language changes. This bug is about a library change proposal only. If you want to discuss a hypothetical language change, the best place is probably golang-nuts@, since it's out of scope for Go 1.

@nsf

This comment has been minimized.

Show comment
Hide comment
@nsf

nsf Aug 16, 2016

@bradfitz How is it offtopic? I was implying that this is a bad idea and you guys should add a built-in function instead. It's a pretty good place where custom code generation is justified. And saying that a language change is out of scope for Go 1 is not serious, there were a number of language changes in 1.x releases (1.2 added three-index slices for example). Adding a built-in function won't break anyone's code.

nsf commented Aug 16, 2016

@bradfitz How is it offtopic? I was implying that this is a bad idea and you guys should add a built-in function instead. It's a pretty good place where custom code generation is justified. And saying that a language change is out of scope for Go 1 is not serious, there were a number of language changes in 1.x releases (1.2 added three-index slices for example). Adding a built-in function won't break anyone's code.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2016

Member

@nsf, we can discuss on golang-nuts if you'd like.

Member

bradfitz commented Aug 16, 2016

@nsf, we can discuss on golang-nuts if you'd like.

@infogulch

This comment has been minimized.

Show comment
Hide comment
@infogulch

infogulch Aug 25, 2016

This might be a little off-topic, but one readability improvement might be to add one more type and method to the sort package to make it easier to compose a sort hierarchy:

package sort

type By func(i, j int) bool

// assumes !(a[i] < a[j]) && !(a[j] < a[i]) is equivalent to (a[i] == a[j])
func (a By) ThenBy(b By) By {
    return func(i, j int) bool {
        return a(i,j) || !a(j,i) && b(i,j)
    }
}

It could be used like this:

sort.With(len(s), reflect.Swapper(s), 
    sort.By(func(i, j int) bool { return s[i].Foo < s[j].Foo })
    .ThenBy(func(i, j int) bool { return s[i].Bar < s[j].Bar })
)

infogulch commented Aug 25, 2016

This might be a little off-topic, but one readability improvement might be to add one more type and method to the sort package to make it easier to compose a sort hierarchy:

package sort

type By func(i, j int) bool

// assumes !(a[i] < a[j]) && !(a[j] < a[i]) is equivalent to (a[i] == a[j])
func (a By) ThenBy(b By) By {
    return func(i, j int) bool {
        return a(i,j) || !a(j,i) && b(i,j)
    }
}

It could be used like this:

sort.With(len(s), reflect.Swapper(s), 
    sort.By(func(i, j int) bool { return s[i].Foo < s[j].Foo })
    .ThenBy(func(i, j int) bool { return s[i].Bar < s[j].Bar })
)
@joshlf

This comment has been minimized.

Show comment
Hide comment
@joshlf

joshlf Aug 25, 2016

@infogulch I really like this idea. I think, though, that it would be a natural extension if/when the existing proposal is adopted (since it assumes that we use func(i, j int) bool). I'd suggest proposing it then. But I do really like it. It's really clean.

joshlf commented Aug 25, 2016

@infogulch I really like this idea. I think, though, that it would be a natural extension if/when the existing proposal is adopted (since it assumes that we use func(i, j int) bool). I'd suggest proposing it then. But I do really like it. It's really clean.

@praetergeek

This comment has been minimized.

Show comment
Hide comment
@praetergeek

praetergeek Aug 26, 2016

I'm not sure I see the "simplicity" benefit of "ThenBy"; why not simply
write the cascading comparisons into the one callback function?

BTW, in a scalar field, !(a[i] < a[j]) && !(a[j] < a[i]) is equivalent
to (a[i] == a[j]) or !(a[i] != a[j]).

On 26 August 2016 at 02:04, Joe notifications@github.com wrote:

This might be a little off-topic, but one readability improvement might be
to add one more type and method to the sort package to make it easier to
compose a sort hierarchy:

package sort
type By func(i, j int) bool
// assumes !(a[i] < a[j]) && !(a[j] < a[i]) is equivalent to (a[i] != a[j])func (a By) ThenBy(b By) By {
return func(i, j int) bool {
return a(i,j) || !a(j,i) && b(i,j)
}
}

It could be used like this:

sort.With(len(s), reflect.Swapper(s),
sort.By(func(i, j int) bool { return s[i].Foo < s[j].Foo })
.ThenBy(func(i, j int) bool { return s[i].Bar < s[j].Bar })
)

I'm not sure I see the "simplicity" benefit of "ThenBy"; why not simply
write the cascading comparisons into the one callback function?

BTW, in a scalar field, !(a[i] < a[j]) && !(a[j] < a[i]) is equivalent
to (a[i] == a[j]) or !(a[i] != a[j]).

On 26 August 2016 at 02:04, Joe notifications@github.com wrote:

This might be a little off-topic, but one readability improvement might be
to add one more type and method to the sort package to make it easier to
compose a sort hierarchy:

package sort
type By func(i, j int) bool
// assumes !(a[i] < a[j]) && !(a[j] < a[i]) is equivalent to (a[i] != a[j])func (a By) ThenBy(b By) By {
return func(i, j int) bool {
return a(i,j) || !a(j,i) && b(i,j)
}
}

It could be used like this:

sort.With(len(s), reflect.Swapper(s),
sort.By(func(i, j int) bool { return s[i].Foo < s[j].Foo })
.ThenBy(func(i, j int) bool { return s[i].Bar < s[j].Bar })
)

@infogulch

This comment has been minimized.

Show comment
Hide comment
@infogulch

infogulch Aug 26, 2016

@praetergeek I'm not sure what your comment about boolean algebra in scalar fields is trying to convey. I understand that it's not exactly equivalent; that's why I made a comment about it. It should be true almost always, but if not for some case then it's easy to fall back to the other method.

Why? It's shorter (1 line per field vs 3n-2 lines), easy to understand (it says right there: sort by x then by y), very easy to visually parse (they line up until the field name), harder to get wrong (only 2 references to each field vs 4; all the logic is encapsulated in ThenBy), and individual field comparers can be reused if needed.

But as @joshlf recommended I think I'll save it for another proposal later if this one is accepted.

infogulch commented Aug 26, 2016

@praetergeek I'm not sure what your comment about boolean algebra in scalar fields is trying to convey. I understand that it's not exactly equivalent; that's why I made a comment about it. It should be true almost always, but if not for some case then it's easy to fall back to the other method.

Why? It's shorter (1 line per field vs 3n-2 lines), easy to understand (it says right there: sort by x then by y), very easy to visually parse (they line up until the field name), harder to get wrong (only 2 references to each field vs 4; all the logic is encapsulated in ThenBy), and individual field comparers can be reused if needed.

But as @joshlf recommended I think I'll save it for another proposal later if this one is accepted.

@Merovius

This comment has been minimized.

Show comment
Hide comment
@Merovius

Merovius Aug 26, 2016

I think @infogulch's idea boils down to introducing a DSL for sorting into the sort-package. I dislike the idea, because it can get out of hand quite quickly and I consider DSLs of this form a bad habit. They hide simple, readable code behind layers of abstractions that need to be unwound.

Also, it will, if used extensively, make code slower. While performance shouldn't be the ultimate goal, I think encouraging fast code by default is a good thing (and as I said, I believe it hurts readability too, though that's subjective).

I think @infogulch's idea boils down to introducing a DSL for sorting into the sort-package. I dislike the idea, because it can get out of hand quite quickly and I consider DSLs of this form a bad habit. They hide simple, readable code behind layers of abstractions that need to be unwound.

Also, it will, if used extensively, make code slower. While performance shouldn't be the ultimate goal, I think encouraging fast code by default is a good thing (and as I said, I believe it hurts readability too, though that's subjective).

@joshlf

This comment has been minimized.

Show comment
Hide comment
@joshlf

joshlf Aug 26, 2016

@Merovius I agree with the DSL point; I hadn't thought of it that way.

joshlf commented Aug 26, 2016

@Merovius I agree with the DSL point; I hadn't thought of it that way.

mpl pushed a commit to go4org/go4 that referenced this issue Sep 8, 2016

sort: add copy of std sort package with additional helpers
See golang/go#16721

Change-Id: I30b02ecfb1dea01da5f064e62e3c472a9867accd
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 8, 2016

Member

Status update on this proposal:

@ianlancetaylor, @griesemer, and @robpike are all neutral. They wanted to wait for @rsc to get back from vacation and chime in.

Because the default rejection answer to standard library addition proposals is "put it elsewhere", I've preemptively put it elsewhere:

https://godoc.org/go4.org/sort

The go4.org/sort package is now a pure superset of the standard library sort package.

I figure people can get some experience with the proposal there in the meantime. The reflect-based Swapper func creator is at https://godoc.org/go4.org/reflectutil and is used by go4.org/sort.

I still think the standard library should make sorting slices easier and permit writing sort functions inline without declaring a type, though, similar to http.HandlerFunc.

Member

bradfitz commented Sep 8, 2016

Status update on this proposal:

@ianlancetaylor, @griesemer, and @robpike are all neutral. They wanted to wait for @rsc to get back from vacation and chime in.

Because the default rejection answer to standard library addition proposals is "put it elsewhere", I've preemptively put it elsewhere:

https://godoc.org/go4.org/sort

The go4.org/sort package is now a pure superset of the standard library sort package.

I figure people can get some experience with the proposal there in the meantime. The reflect-based Swapper func creator is at https://godoc.org/go4.org/reflectutil and is used by go4.org/sort.

I still think the standard library should make sorting slices easier and permit writing sort functions inline without declaring a type, though, similar to http.HandlerFunc.

@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Sep 30, 2016

Contributor

Ping @rsc.

Contributor

adg commented Sep 30, 2016

Ping @rsc.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 30, 2016

Contributor

TL;DR: Can we do just sort.Slice and not all the general stuff below it?

I looked at the go4 code and the pending CL 27321. I agree that it would be good to make sort easier to use. I see a few different parts here:

  • sort.Slice, as in the go4 code.
  • sort.MakeInterface and reflect.Swapper.
  • sort.With.
  • sort.By from @infogulch above.

I would be happy to see sort.Slice in Go 1.8. It's a nice API and will be a significant improvement for the vast majority of sort.Sort uses.

The rest seem unfortunate. Specifically:

  • sort.MakeInterface can be written today, although it would execute a little less efficiently when used. It seems to exist only to implement sort.With, except that sort.With doesn't even use it.
  • Exposing reflect.Swapper may be a necessary evil. That's OK. Better to require other Go implementations to implement/port reflect.Swapper than to port unsafe code tucked into package sort.
  • sort.With can be written today (like sort.MakeInterface). As an API, it's not great because basically every call is going to use reflect.Swapper as the second argument, and it would be nice if reflect.Swapper were never mentioned again, especially not by Go users.
  • sort.By is a helper that others can provide easily enough. Unlike the others here, there's no obvious speedup behind the scenes if implemented inside package sort. Better to leave this one out for now.

Stepping back a bit, I guess the approach here is a few steps:

  1. Define a function-based sort API (sort.With)
  2. Define a converter from function API to interface API (sort.MakeInterface).
  3. Define a helper (reflect.Swapper) for sorting slices.

The alternative is one step: "add sort.Slice". Unless there is evidence that the intermediate steps have compelling uses to clean up existing code other than getting to slices, I'd rather not commit to the details of those steps in public API. sort.Slice this way ends up being a helper like sort.Ints, not a gateway to a whole parallel API.

I would also like to see sort.SliceStable added at the same time. It needs to be just as easy to do a stable sort as an unstable sort. I don't care much one way or the other about sort.SliceIsSorted but I would lean toward including it by analogy with the Ints functions.

I considered suggesting that sort.Slice return a sort.Interface, so that you'd write:

sort.Sort(sort.Slice(x, func(i, j int) bool { return x[i] < x[j] }))
sort.Stable(sort.Slice(x, func(i, j int) bool { return x[i] < x[j] }))
sort.IsSorted(sort.Slice(x, func(i, j int) bool { return x[i] < x[j] }))

but I think that's overkill. sort.Slice is a helper, so it should be helpful. The fact that the comparison takes indexes instead of values means that in basically every call there's going to be a closure, so we should avoid piling on too much other stutter. These are clearer:

sort.Slice(x, func(i, j int) bool { return x[i] < x[j] })
sort.SliceStable(x, func(i, j int) bool { return x[i] < x[j] })
sort.SliceIsSorted(x, func(i, j int) bool { return x[i] < x[j] }))
Contributor

rsc commented Sep 30, 2016

TL;DR: Can we do just sort.Slice and not all the general stuff below it?

I looked at the go4 code and the pending CL 27321. I agree that it would be good to make sort easier to use. I see a few different parts here:

  • sort.Slice, as in the go4 code.
  • sort.MakeInterface and reflect.Swapper.
  • sort.With.
  • sort.By from @infogulch above.

I would be happy to see sort.Slice in Go 1.8. It's a nice API and will be a significant improvement for the vast majority of sort.Sort uses.

The rest seem unfortunate. Specifically:

  • sort.MakeInterface can be written today, although it would execute a little less efficiently when used. It seems to exist only to implement sort.With, except that sort.With doesn't even use it.
  • Exposing reflect.Swapper may be a necessary evil. That's OK. Better to require other Go implementations to implement/port reflect.Swapper than to port unsafe code tucked into package sort.
  • sort.With can be written today (like sort.MakeInterface). As an API, it's not great because basically every call is going to use reflect.Swapper as the second argument, and it would be nice if reflect.Swapper were never mentioned again, especially not by Go users.
  • sort.By is a helper that others can provide easily enough. Unlike the others here, there's no obvious speedup behind the scenes if implemented inside package sort. Better to leave this one out for now.

Stepping back a bit, I guess the approach here is a few steps:

  1. Define a function-based sort API (sort.With)
  2. Define a converter from function API to interface API (sort.MakeInterface).
  3. Define a helper (reflect.Swapper) for sorting slices.

The alternative is one step: "add sort.Slice". Unless there is evidence that the intermediate steps have compelling uses to clean up existing code other than getting to slices, I'd rather not commit to the details of those steps in public API. sort.Slice this way ends up being a helper like sort.Ints, not a gateway to a whole parallel API.

I would also like to see sort.SliceStable added at the same time. It needs to be just as easy to do a stable sort as an unstable sort. I don't care much one way or the other about sort.SliceIsSorted but I would lean toward including it by analogy with the Ints functions.

I considered suggesting that sort.Slice return a sort.Interface, so that you'd write:

sort.Sort(sort.Slice(x, func(i, j int) bool { return x[i] < x[j] }))
sort.Stable(sort.Slice(x, func(i, j int) bool { return x[i] < x[j] }))
sort.IsSorted(sort.Slice(x, func(i, j int) bool { return x[i] < x[j] }))

but I think that's overkill. sort.Slice is a helper, so it should be helpful. The fact that the comparison takes indexes instead of values means that in basically every call there's going to be a closure, so we should avoid piling on too much other stutter. These are clearer:

sort.Slice(x, func(i, j int) bool { return x[i] < x[j] })
sort.SliceStable(x, func(i, j int) bool { return x[i] < x[j] })
sort.SliceIsSorted(x, func(i, j int) bool { return x[i] < x[j] }))
@mibk

This comment has been minimized.

Show comment
Hide comment
@mibk

mibk Sep 30, 2016

Contributor

Just a quick note. Isn't sort.IsSliceSorted better? Or is it more valuable to have the same prefix for all these related methods?

Contributor

mibk commented Sep 30, 2016

Just a quick note. Isn't sort.IsSliceSorted better? Or is it more valuable to have the same prefix for all these related methods?

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 30, 2016

Contributor

@mibk From a clean slate, maybe, but we already have IntsAreSorted, Float64sAreSorted, StringsAreSorted, not AreIntsSorted etc.

Contributor

rsc commented Sep 30, 2016

@mibk From a clean slate, maybe, but we already have IntsAreSorted, Float64sAreSorted, StringsAreSorted, not AreIntsSorted etc.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 30, 2016

Member

Exposing reflect.Swapper may be a necessary evil. That's OK. Better to require other Go implementations to implement/port reflect.Swapper than to port unsafe code tucked into package sort.

So you're cool with 4 API additions: reflect.Swapper, sort.Slice{,Stable,IsSorted}?

Member

bradfitz commented Sep 30, 2016

Exposing reflect.Swapper may be a necessary evil. That's OK. Better to require other Go implementations to implement/port reflect.Swapper than to port unsafe code tucked into package sort.

So you're cool with 4 API additions: reflect.Swapper, sort.Slice{,Stable,IsSorted}?

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 30, 2016

Contributor

So you're cool with 4 API additions: reflect.Swapper, sort.Slice{,Stable,IsSorted}?

SGTM

Contributor

rsc commented Sep 30, 2016

So you're cool with 4 API additions: reflect.Swapper, sort.Slice{,Stable,IsSorted}?

SGTM

@bradfitz bradfitz self-assigned this Sep 30, 2016

@bradfitz bradfitz modified the milestones: Go1.8, Proposal Sep 30, 2016

@bradfitz bradfitz changed the title from proposal: sort: make sorting easier to sort: make sorting easier, add Slice, SliceStable, SliceIsSorted, reflect.Swapper Sep 30, 2016

@gopherbot

This comment has been minimized.

Show comment
Hide comment

CL https://golang.org/cl/30088 mentions this issue.

@rsc rsc added the NeedsFix label Sep 30, 2016

gopherbot pushed a commit that referenced this issue Sep 30, 2016

reflect: add Swapper func
Swapper returns a func that swaps two elements in a slice.

Updates #16721

Change-Id: I7f2287a675c10a05019e02b7d62fb870af31216f
Reviewed-on: https://go-review.googlesource.com/30088
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@extemporalgenome

This comment has been minimized.

Show comment
Hide comment
@extemporalgenome

extemporalgenome Oct 1, 2016

@rsc considering that external packages may accept sort.Interface values, such as container/heap and presumably a subset of https://godoc.org/sort?importers (including a few of my own packages), I'd like to counter-argue that having no mechanism for creating a sort.Interface will be surprising and disappointing.

Even in the absence of such concerns, I'd like to argue that the proliferation of parallel functions (SliceStable) to be unfortunate -- I appreciate sort.StringSlice and friends, but see no justifiable benefit to sort.Strings and friends. Such functions bloat the package, saving users only a handful of keystrokes.

I feel that this may just boil down to a naming problem -- with a clearer name than MakeInterface, the usage of an interface builder may well become obvious.

extemporalgenome commented Oct 1, 2016

@rsc considering that external packages may accept sort.Interface values, such as container/heap and presumably a subset of https://godoc.org/sort?importers (including a few of my own packages), I'd like to counter-argue that having no mechanism for creating a sort.Interface will be surprising and disappointing.

Even in the absence of such concerns, I'd like to argue that the proliferation of parallel functions (SliceStable) to be unfortunate -- I appreciate sort.StringSlice and friends, but see no justifiable benefit to sort.Strings and friends. Such functions bloat the package, saving users only a handful of keystrokes.

I feel that this may just boil down to a naming problem -- with a clearer name than MakeInterface, the usage of an interface builder may well become obvious.

@suyash

This comment has been minimized.

Show comment
Hide comment
@suyash

suyash Oct 3, 2016

Contributor

sorry for bringing this up if this was resolved somewhere else, but any particular reason the sort.Search like syntax was not preferred?

I think that the fundamental argument of making reflect.Swapper function public is to allow users the option of using it or not, otherwise it can just be implemented inside the sort package itself.

another point is that getting the length of slice directly would probably be faster than getting it through reflection, though not sure by how much.

also, all the previous discussion about not encouraging generic programming through interface{}

Contributor

suyash commented Oct 3, 2016

sorry for bringing this up if this was resolved somewhere else, but any particular reason the sort.Search like syntax was not preferred?

I think that the fundamental argument of making reflect.Swapper function public is to allow users the option of using it or not, otherwise it can just be implemented inside the sort package itself.

another point is that getting the length of slice directly would probably be faster than getting it through reflection, though not sure by how much.

also, all the previous discussion about not encouraging generic programming through interface{}

@gopherbot gopherbot closed this in 22a2bdf Oct 3, 2016

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Oct 3, 2016

Contributor

@extemporalgenome It's trivial to write a converter from function triple to interface if someone needs that. That's true already today.

type funcs struct { n int; swap func(i, j int); less func(i, j int) bool }
func (x funcs) Len() int { return x.n }
func (x funcs) Swap(i, j int) { x.swap(i, j) }
func (x funcs) Less(i, j int) bool { return x.less(i, j) }
func FuncInterface(n int, swap func(i, j int), less func(i, j int) bool) Interface {
    return &funcs{n, swap, less}
}

If there is evidence that this comes up often in real programs, we can easily add it later. But we'd need to see that evidence first.

@suyash I assume that by sort.Search-like syntax you mean code like

sort.With(len(s), func(i, j int) { s[i], s[j] = s[j], s[i] }, func(i, j int) bool {
    if s[i].Foo != s[j].Foo {
        return s[i].Foo < s[j].Foo
    }
    return s[i].Bar < s[j].Bar
})

as in the comment above (Aug 21). The reason not to prefer this is that this entire proposal is about making sort easier to use. The above is not substantially less code than you have to write today. It does avoid the new type, but it is still full of boilerplate.

Compare to:

sort.Slice(s, func(i, j int) bool {
    if s[i].Foo != s[j].Foo {
        return s[i].Foo < s[j].Foo
    }
    return s[i].Bar < s[j].Bar
})

The new API is meant to be a helper, so it should be as helpful as possible - with as little boilerplate as possible - without restricting the scope of its help. My guess is that >99% of sorting happens on slices, in which case the generality added by the boilerplate serves no useful purpose. If someone has evidence otherwise, I'd be happy to see it.

Contributor

rsc commented Oct 3, 2016

@extemporalgenome It's trivial to write a converter from function triple to interface if someone needs that. That's true already today.

type funcs struct { n int; swap func(i, j int); less func(i, j int) bool }
func (x funcs) Len() int { return x.n }
func (x funcs) Swap(i, j int) { x.swap(i, j) }
func (x funcs) Less(i, j int) bool { return x.less(i, j) }
func FuncInterface(n int, swap func(i, j int), less func(i, j int) bool) Interface {
    return &funcs{n, swap, less}
}

If there is evidence that this comes up often in real programs, we can easily add it later. But we'd need to see that evidence first.

@suyash I assume that by sort.Search-like syntax you mean code like

sort.With(len(s), func(i, j int) { s[i], s[j] = s[j], s[i] }, func(i, j int) bool {
    if s[i].Foo != s[j].Foo {
        return s[i].Foo < s[j].Foo
    }
    return s[i].Bar < s[j].Bar
})

as in the comment above (Aug 21). The reason not to prefer this is that this entire proposal is about making sort easier to use. The above is not substantially less code than you have to write today. It does avoid the new type, but it is still full of boilerplate.

Compare to:

sort.Slice(s, func(i, j int) bool {
    if s[i].Foo != s[j].Foo {
        return s[i].Foo < s[j].Foo
    }
    return s[i].Bar < s[j].Bar
})

The new API is meant to be a helper, so it should be as helpful as possible - with as little boilerplate as possible - without restricting the scope of its help. My guess is that >99% of sorting happens on slices, in which case the generality added by the boilerplate serves no useful purpose. If someone has evidence otherwise, I'd be happy to see it.

@gopherbot

This comment has been minimized.

Show comment
Hide comment

CL https://golang.org/cl/30250 mentions this issue.

@suyash

This comment has been minimized.

Show comment
Hide comment
@suyash

suyash Oct 4, 2016

Contributor

@rsc lets take an example that comes up fairly often (for me), lets say you want to sort a set of numbers but you also want to track the original positions of the numbers.

So you had a slice like []int{5, 2, 3, 1, 4} and you sorted it to get []int{1, 2, 3, 4, 5} but at the same time, you also wanted to track positions, so in a separate slice you would want []int{3, 1, 2, 4, 0}.

With this addition, you would separately declare a pair type and a new pair array and copy original array's contents into the new array and initialize positions and then sort them, while defining the appropriate less closure.Something like

a := []int{...}

type pair struct {
    first, second int
}

p := make([]pair, len(a))
for i := 0 ; i < len(a) ; i++ {
    p[i] = pair{a[i], i}
}

sort.Slice(p, func (i, j int) bool {
    return p[i].first < p[j].first || (p[i].first == p[j].first && p[i].second < p[j].second)
})

while if we had the option to define a swapper, we can just define another slice, initialize that to positions (0, 1, 2...) and in the swap, swap in both slices.

something like

a := []int{...}

pos := make([]int, len(a))
for i := 0 ; i < len(a) ; i++ {
    pos[i] = i
}

sort.Slice(len(a), func(i, j int) {
    a[i], a[j] = a[j], a[i]
    pos[i], pos[j] = pos[j], pos[i]
}, func(i, j int) bool {
    return a[i] < a[j] || (a[i] == a[j] && pos[i] < pos[j])
})
  1. The first reason I am advocating the use of sort.Search like semantics is because sort.Search itself is defined only for slices, but the best part about it is that it can do 6 different operations simply by passing the appropriate closure (https://go-review.googlesource.com/#/c/29131/6/src/sort/example_search_test.go@58). One issue that comes up with doing things differently with sort.Slice is the difference from sort.Search in terms of semantics.

  2. More convenience (for me) means more flexibility, then less code.

  3. Again, the reason reflect.Swapper came up in the first place was to support the general case, so you could do

    sort.Slice(len(a), reflect.Swapper(a), func(i, j int) { ... })
    

    otherwise I don't think there was any need to have it as a public library function.

  4. another thing that came up in this discussion was the facilities provided by modern text editors
    w.r.t boilerplate code. editors these days have macros and snippets, so you can configure them to
    expand things, like you can define a swapper snippet that on a tabstop expands to func(i, j int) { a[i], a[j] = a[j], a[i] }

  5. @bradfitz raised a point about purity of the function irrespective of type errors (#16721 (comment))

  6. Again, and I don't know how much impact it will have, but getting the slice length directly will probably be faster than getting it through reflection

  7. You might think that something like sort.MakeInterface solves this, but @bradfitz himself did genzfunc.go for performance, the results of which are in the commit message at https://go-review.googlesource.com/#/c/27321/11. Also in case you didn't see this: https://github.com/nieksand/sortgenerics https://news.ycombinator.com/item?id=12602456

Contributor

suyash commented Oct 4, 2016

@rsc lets take an example that comes up fairly often (for me), lets say you want to sort a set of numbers but you also want to track the original positions of the numbers.

So you had a slice like []int{5, 2, 3, 1, 4} and you sorted it to get []int{1, 2, 3, 4, 5} but at the same time, you also wanted to track positions, so in a separate slice you would want []int{3, 1, 2, 4, 0}.

With this addition, you would separately declare a pair type and a new pair array and copy original array's contents into the new array and initialize positions and then sort them, while defining the appropriate less closure.Something like

a := []int{...}

type pair struct {
    first, second int
}

p := make([]pair, len(a))
for i := 0 ; i < len(a) ; i++ {
    p[i] = pair{a[i], i}
}

sort.Slice(p, func (i, j int) bool {
    return p[i].first < p[j].first || (p[i].first == p[j].first && p[i].second < p[j].second)
})

while if we had the option to define a swapper, we can just define another slice, initialize that to positions (0, 1, 2...) and in the swap, swap in both slices.

something like

a := []int{...}

pos := make([]int, len(a))
for i := 0 ; i < len(a) ; i++ {
    pos[i] = i
}

sort.Slice(len(a), func(i, j int) {
    a[i], a[j] = a[j], a[i]
    pos[i], pos[j] = pos[j], pos[i]
}, func(i, j int) bool {
    return a[i] < a[j] || (a[i] == a[j] && pos[i] < pos[j])
})
  1. The first reason I am advocating the use of sort.Search like semantics is because sort.Search itself is defined only for slices, but the best part about it is that it can do 6 different operations simply by passing the appropriate closure (https://go-review.googlesource.com/#/c/29131/6/src/sort/example_search_test.go@58). One issue that comes up with doing things differently with sort.Slice is the difference from sort.Search in terms of semantics.

  2. More convenience (for me) means more flexibility, then less code.

  3. Again, the reason reflect.Swapper came up in the first place was to support the general case, so you could do

    sort.Slice(len(a), reflect.Swapper(a), func(i, j int) { ... })
    

    otherwise I don't think there was any need to have it as a public library function.

  4. another thing that came up in this discussion was the facilities provided by modern text editors
    w.r.t boilerplate code. editors these days have macros and snippets, so you can configure them to
    expand things, like you can define a swapper snippet that on a tabstop expands to func(i, j int) { a[i], a[j] = a[j], a[i] }

  5. @bradfitz raised a point about purity of the function irrespective of type errors (#16721 (comment))

  6. Again, and I don't know how much impact it will have, but getting the slice length directly will probably be faster than getting it through reflection

  7. You might think that something like sort.MakeInterface solves this, but @bradfitz himself did genzfunc.go for performance, the results of which are in the commit message at https://go-review.googlesource.com/#/c/27321/11. Also in case you didn't see this: https://github.com/nieksand/sortgenerics https://news.ycombinator.com/item?id=12602456

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Oct 4, 2016

Contributor

@suyash For your example you could use rsc's FuncInterface function above. It's marginally more typing for you, and no additional API in the standard library. To get this into the standard library there needs to be clear and common use cases.

Contributor

ianlancetaylor commented Oct 4, 2016

@suyash For your example you could use rsc's FuncInterface function above. It's marginally more typing for you, and no additional API in the standard library. To get this into the standard library there needs to be clear and common use cases.

gopherbot pushed a commit that referenced this issue Oct 4, 2016

all: use sort.Slice where applicable
I avoided anywhere in the compiler or things which might be used by
the compiler in the future, since they need to build with Go 1.4.

I also avoided anywhere where there was no benefit to changing it.

I probably missed some.

Updates #16721

Change-Id: Ib3c895ff475c6dec2d4322393faaf8cb6a6d4956
Reviewed-on: https://go-review.googlesource.com/30250
TryBot-Result: Gobot Gobot <gobot@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Oct 4, 2016

Contributor

@suyash What Ian said, but also "text editors can macro-generate your code for you" is a particularly bad rationale for an API decision. Code has to be read and maintained, not just written. Also, if you are OK with text editor macros, one could generate the old, more flexible sort.Interface type+methods for you if you insist.

The time it takes to compute the slice length is not going to show up.

Contributor

rsc commented Oct 4, 2016

@suyash What Ian said, but also "text editors can macro-generate your code for you" is a particularly bad rationale for an API decision. Code has to be read and maintained, not just written. Also, if you are OK with text editor macros, one could generate the old, more flexible sort.Interface type+methods for you if you insist.

The time it takes to compute the slice length is not going to show up.

@groob groob referenced this issue in kolide/fleet Oct 15, 2016

Merged

Add ordering options for List* methods #318

chromium-infra-bot pushed a commit to luci/luci-go that referenced this issue Jan 25, 2017

Add sort function helper library.
Inspired by comments on golang/go#16721.

This will be particularly handy when we get to use a version of go (1.8) which
has `sort.Slice`. But in the mean time, it's at least clearer and quicker than
hand-rolling Less methods (and potentially safer too!).

R=dnj@chromium.org, nodir@chromium.org, vadimsh@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2657733002

snsinfu added a commit to snsinfu/learn-go that referenced this issue May 13, 2017

Add 07: Magic square solver
Initial dumb implementation.

- CPU profiling. To see the result, run 07-magic_square, execute
  `go tool pprof 07-magic_square path/to/cpu.pprof` and input
  `top[Enter]` in the prompt.

- Generic algorithm. The trick is the Float64Slice adaptor; see [1].
  Maybe I will use the technique used in the sort library[2][3]
  which allows simpler API.

[1]: http://stackoverflow.com/a/6256127/5266681
[2]: golang/go#16721
[3]: golang/go@22a2bdf

@golang golang locked and limited conversation to collaborators Oct 4, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.