GH-35515: [C++][Python] Add non decomposable aggregation UDF

[icexelloss]

Rationale for this change

Non decomposable aggregation is aggregation that cannot be split into consume/merge/finalize. This is often when the logic rewritten with external python libraries (numpy, pandas, statmodels, etc) and those either cannot be decomposed or not worthy the effect (these are often one-off function instead of reusable one). This PR implements the support for non decomposable aggregation UDFs.

The major issue with non decomposable UDF is that the UDF needs to see all data at once, unlike scalar UDF where UDF only needs to see a batch at a time. This makes non decomposable not so useful as it is same as collect all the data to a pd.DataFrame and apply the UDF on it. However, one very application of non decomposable UDF is with segmented aggregation. To refresh, segmented aggregation works on ordered data and passed one logic chunk at a time (e.g., all data with the same date). With segmented aggregation and non decomposable aggregation UDF, the user can apply any custom aggregation logic over large stream of ordered data, with the memory overhead of a single segment.

What changes are included in this PR?

This PR is currently WIP and not ready for review.

So far I have implemented the minimal amount of code to make a basic test working but needs clean up, error handling etc.

• First round of self review
• Second round of self review
• Implement and test unary
• Implement and test varargs
• Implement and test Acero support with segmented aggregation

Are these changes tested?

Added new test calling with compute and acero.

The compute tests calls the aggregation on the full array. The acero test callings the aggregation with segmented aggregation.

Are there any user-facing changes?

• Closes: [C++][Python] Non decomposable aggregation UDF #35515

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

• Closes: [C++][Python] Non decomposable aggregation UDF #35515

[github-actions]

⚠️ GitHub issue #35515 has been automatically assigned in GitHub to PR creator.

[icexelloss]

"Scalar" as supposed to the "grouped" aggregator which has difference interface:
https://github.com/apache/arrow/blob/main/cpp/src/arrow/compute/kernels/hash_aggregate.cc#L66

[icexelloss]

@westonpace I believe this PR is good to go. The failed CI seems unrelated. I have checked the Py refcount and it seems OK (I will add details in the comment thread above)

[westonpace]

@icexelloss I'll take another look through today.

[icexelloss]

@icexelloss I'll take another look through today.

Thank you!

[westonpace]

A few more very minor suggestions but, overall, I think this is fine.

[westonpace]

Ok, so the test case is verifying that the python function can take in *args if needed (even though it still lists the args when registering)?

[westonpace]

This increment seems redundant given you already have one here.

[icexelloss]

Admitted there could be some redundancy here. I created an follow up to take a closer look:

#36000

[icexelloss]

I checked failed CI jobs and those seem unrelated.

[ursabot]

Benchmark runs are scheduled for baseline = e920bed and contender = 8b5919d. 8b5919d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️1.15% ⬆️0.06%] test-mac-arm
[Finished ⬇️10.46% ⬆️6.21%] ursa-i9-9960x
[Finished ⬇️0.3% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 8b5919d8 ec2-t3-xlarge-us-east-2
[Finished] 8b5919d8 test-mac-arm
[Finished] 8b5919d8 ursa-i9-9960x
[Finished] 8b5919d8 ursa-thinkcentre-m75q
[Finished] e920bed4 ec2-t3-xlarge-us-east-2
[Finished] e920bed4 test-mac-arm
[Finished] e920bed4 ursa-i9-9960x
[Finished] e920bed4 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java