GH-14975: [Python] Dataset.sort_by

[amol-]

• Closes: [Python] Add Dataset.sort_by #14975

• Proof of concept using an ExecPlan
• Add test to filter and then sort to confirm lazy filtering works with sorting.

[github-actions]

• Closes: [Python] Add Dataset.sort_by #14975

[github-actions]

⚠️ GitHub issue #14975 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

Is there any advantage is using the exec plan version here, instead of the current version? (but I suppose also no disadvantage)

[amol-]

I didn't benchmark it, I can if you want to have numbers, but I guessed that it's faster to do the whole operation in C++ once than having to sort indices and create a temporary arrays of indices just to then take the actual values from the table.

[jorisvandenbossche]

For future reference, at the time I replaced this with a manual sort_indices call in a follow-up PR (#15268, for enabling this in a minimal build). But so it also appears that the Acero-based implementation is currently actually slower, see #34249 (comment) for some analysis (so as long as that is not solved, eg by allowing configuring the batch size, we should keep this manual implementation for Table/RecordBatch.sort_by).

[ursabot]

Benchmark runs are scheduled for baseline = 0f5b8dd and contender = 387e95a. 387e95a is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.26% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.58% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 387e95ad ec2-t3-xlarge-us-east-2
[Failed] 387e95ad test-mac-arm
[Finished] 387e95ad ursa-i9-9960x
[Finished] 387e95ad ursa-thinkcentre-m75q
[Finished] 0f5b8dda ec2-t3-xlarge-us-east-2
[Failed] 0f5b8dda test-mac-arm
[Finished] 0f5b8dda ursa-i9-9960x
[Finished] 0f5b8dda ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[kou]

@github-actions crossbow submit example-python-minimal-build-*

[github-actions]

Revision: 3dbe53b

Submitted crossbow builds: ursacomputing/crossbow @ actions-78101888a5

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv

[kou]

@amol- It seems that this breaks example-python-minimal-build-* nightly CI jobs:

https://github.com/ursacomputing/crossbow/actions/runs/3772771994/jobs/6413886549#step:3:7216

=================================== FAILURES ===================================
______________________________ test_table_sort_by ______________________________

    def test_table_sort_by():
        table = pa.table([
            pa.array([3, 1, 4, 2, 5]),
            pa.array(["b", "a", "b", "a", "c"]),
        ], names=["values", "keys"])
    
>       assert table.sort_by("values").to_pydict() == {
            "keys": ["a", "a", "b", "b", "c"],
            "values": [1, 2, 3, 4, 5]
        }

pyarrow/tests/test_table.py:2198: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   res = _pc()._exec_plan._sort_source(self, output_type=Table,
E   AttributeError: module 'pyarrow.compute' has no attribute '_exec_plan'

pyarrow/table.pxi:4761: AttributeError

Could you fix this?

[jorisvandenbossche]

Fixing this in #15268