slices: use dual-pivot quicksort instead of pdqsort #61027
Comments
Dual-pivot quicksort has smaller recursion depth. Some pattern-defeating technique is added to make this variant. Replace standalone rotate function in symmerge with rotateLeft. Benchmark on Xeon-8374C name old time/op new time/op delta SlicesSortInts 7.67ms ± 0% 6.23ms ± 0% -18.82% (p=0.000 n=10+10) SlicesSortInts_Sorted 113µs ± 1% 88µs ± 2% -22.31% (p=0.000 n=10+10) SlicesSortInts_Reversed 148µs ± 0% 126µs ± 1% -15.04% (p=0.000 n=10+9) SlicesSortStrings 28.0ms ± 1% 27.3ms ± 4% -2.35% (p=0.022 n=9+10) SlicesSortStrings_Sorted 628µs ± 1% 567µs ± 0% -9.80% (p=0.000 n=10+9) SortFuncStructs 14.1ms ± 2% 13.4ms ± 2% -5.53% (p=0.000 n=10+10) SortStableFuncStructs 32.2ms ± 2% 33.3ms ± 1% +3.43% (p=0.000 n=9+10) Benchmark on EPYC-7K83 name old time/op new time/op delta SlicesSortInts 7.10ms ± 0% 5.69ms ± 0% -19.82% (p=0.000 n=9+10) SlicesSortInts_Sorted 96.7µs ± 1% 90.8µs ± 1% -6.06% (p=0.000 n=10+10) SlicesSortInts_Reversed 133µs ± 2% 113µs ± 2% -15.20% (p=0.000 n=10+10) SlicesSortStrings 26.9ms ± 4% 26.0ms ± 2% -3.36% (p=0.000 n=10+9) SlicesSortStrings_Sorted 658µs ± 1% 644µs ± 0% -2.20% (p=0.000 n=10+8) SortFuncStructs 13.5ms ± 8% 12.9ms ± 2% -4.42% (p=0.002 n=10+8) SortStableFuncStructs 35.5ms ± 7% 31.6ms ± 7% -11.18% (p=0.000 n=9+10) For golang#61027
|
CC @eliben |
ianlancetaylor
added
NeedsInvestigation
|
I think we already used this decade's "change to a new sort algorithm" coupon. |
Dual-pivot quicksort is not a new sort algorithm. It can be found in Java standard library many years ago. New algorithm like pdqsort is not always best. |
|
My point is we just put the ecosystem through the churn of a sort change in Go 1.19 (2022). I don't see a strong argument that slices.Sort is somehow fundamentally different from sort.Slice and merits a different algorithm, nor do I see a strong argument for invalidating our decision just last year to switch to pdqsort. We could spend lots of time flip-flopping between different sort algorithms in the standard library, or we could do other work. Having just changed the sort algorithm last year, I think we owe it to ourselves and our users not to change it again for at least five years (that is, Go 1.29 at the earliest) unless we have very good reasons. We should definitely not make a change for Go 1.21 - it is too late in the cycle. |
I agree that it’s not a good idea to make changes frequently. People now can use different sort algorithms with third-party libraries. Just let it fly. |
|
To add to @rsc 's point, we're also putting the community through a "sorting change" right now, in the upcoming Go 1.21, where the pdqsort was a large performance improvement for some cases. The |
Patern-defeating techique just makes fastest cases faster. It helps little in real hybrid workload. New apis are needed to achieve significant speedup. But I don't think it's worthy to change standard library apis for speed now. |
|
I didn't really care which sort algorithm is used in go standard library until someone claimed pdqsort is fastest algorithm in Go. That's not the truth, but got credit from go tream. @eliben @rsc |
|
I'm the author of this article, and I don't think I claimed that "PDQSORT is the fastest sorting algorithm in Go". We can see the summary of this article(translated from Chinese directly):
The title of this article is "Building the fastest sorting algorithm"(it is ongoing), PDQSORT is just an important milestone on the road to building the fastest sorting algorithm. There are still many other languages or libraries that use this algorithm, and I still haven't found any algorithm that is faster than PDQSORT in all cases for now. PDQSORT is still one of the fastest hybrid sorting algorithms in some cases. In addition, I am not the author of the PDQSORT algorithm itself. This implementation is first used internally at my work and then contributed to the community. This article is more to let everyone understand the principles of the algorithm (in fact, this article is part of a course for college students within the company), and to attract more people to participate in community building, if you have a better idea for sorting algorithms, feel free. |
|
Pattern-defeating technique can be found in timsort. Branch-eliminating technique comes from blockquicksort. Cache-exploiting technique comes from dual-pivot quicksort. Most worryingly, go team seems to support this mistake. |
|
In the process of implementation, we often need some engineering compromises, which means that the optimization of the paper wouldn't always work. For example, the implementation in the standard library is about 10%~ 20% slower than the original implementation I used. This is because it needs to support both interface-sorting and integer-sorting, and these algorithms need to be generated using the same file. If the algorithm only needs to support integer sorting or doesn't need to be generated via the same file, many other optimizations can be used, but it doesn't mean these engineering compromises are a mistake. |
Reinplement #52789 based on go 1.21rc2. Here is new benchmark result.
Xeon-8374C
EPYC-7K83
Dual-pivot quicksort usually has better performance than current pdqsort. It can be even faster with api change.
Xeon-8374C (X86-64)
Yitian-710 (ARM64)
By the way, this new implement is also shorter and simpler.
The text was updated successfully, but these errors were encountered: