GH-34388: [C++] Build core compute kernels unconditionally

[benibus]

This includes the core compute machinery in libarrow by default - in addition to all cast kernels and several other kernels that are either dependencies of cast (take) or utilized in libarrow/libparquet (unique, filter). The remaining kernels won't be built/registered unless ARROW_COMPUTE=ON (note that this would slightly change the option's meaning, as currently, nothing in arrow/compute is built unless it's set).

Initially this was more substantial as the original goal was to build the extra kernels as a shared library (suggested in the orginal issue). After some discussion in the issue thread, I opted not to do that - primarily because I can't personally see the utility of a separate lib here, even ignoring the complexity it introduces. However, there may be a good reason that simply hasn't occured to me.

• Closes: [C++] Build core compute kernels unconditionally #34388

[github-actions]

⚠️ GitHub issue #25025 has been automatically assigned in GitHub to PR creator.

[benibus]

One thing I haven't decided is how to deal with the compute unit tests since most of them make heavy use of the extra kernels, so a good chunk of them will fail without them. Easiest option would be force ARROW_COMPUTE=ON if ARROW_BUILD_TESTS=ON (not the worst idea in the world, i guess - as this is a packaging-focused feature). Alternatively, we could just not build the tests in question - although that would include most of the tests in compute/exec.

(Also, I'll look into the unity build failures - not quite sure what's going wrong there...)

[benibus]

cc @felipecrv

[felipecrv]

This isn't enough to close the issue, right? Do you want to associate this one with a sub-issue of #25025 so you can merge it before working on the shared library setup?

I've read the PR description more carefully now.

[benibus]

Well to be fair, that might still be the right move as it'd be easy to make that a follow-up PR if we decide to go down that road (assuming we get the library boundaries right in this one).

[westonpace]

The goal is to eventually reduce the size of the "core" library right? Do we have any idea how slim this set of baseline kernels is compared to the full set?

[westonpace]

Seems like a reasonable set of minimal kernels. Do you want to add a CI job (or does one already exist?) that builds without ARROW_COMPUTE to ensure basic functionality (e.g. parquet reading/writing and csv reading/writing) still works?

[benibus]

Do we have any idea how slim this set of baseline kernels is compared to the full set?

Here's what would be present in the default build:

array_filter
array_take
cast
dictionary_encode
drop_null
filter
indices_nonzero
take
unique
value_counts

There are currently 240 kernels in the full set, so it's a pretty deep cut.

Do you want to add a CI job (or does one already exist?) that builds without ARROW_COMPUTE to ensure basic functionality (e.g. parquet reading/writing and csv reading/writing) still works?

That would be a good idea, yes. AFAIK none of the existing jobs build without ARROW_COMPUTE. Even if they did, the CSV writer/STL tests wouldn't be included and libparquet wouldn't be built at all.

[westonpace]

This failure looks related: https://github.com/apache/arrow/actions/runs/4247620924/jobs/7385865305

Perhaps just a changing of include orders has angered the unity build gremlins in some way. Given the two implementations are identical maybe we could put them in util_internal.h?

[github-actions]

• Closes: [C++] Build core compute kernels unconditionally #34388

[github-actions]

⚠️ GitHub issue #34388 has been automatically assigned in GitHub to PR creator.

[benibus]

I still need to add the CI job, but in preparation, I set things up so that certain tests won't be built without the complete kernel registry - so we wouldn't need any special ctest flags to avoid expected failures.

The unity build redefinition errors should be fixed now. Most of the problematic code in scalar_round.cc was actually completely unused, so I just removed it.

[westonpace]

@felipecrv / @lidavidm I haven't been following the discussion on #25025 very closely. However, this change seems good. I assume we want to proceed with it?

[lidavidm]

Yes, either way, I think this is a necessary first step before we can unbundle the kernels + I hope it is easier to review this way

[ursabot]

Benchmark runs are scheduled for baseline = 4b31aa4 and contender = be7a763. be7a763 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.4% ⬆️0.06%] test-mac-arm
[Finished ⬇️0.51% ⬆️0.26%] ursa-i9-9960x
[Finished ⬇️0.25% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] be7a7634 ec2-t3-xlarge-us-east-2
[Finished] be7a7634 test-mac-arm
[Finished] be7a7634 ursa-i9-9960x
[Finished] be7a7634 ursa-thinkcentre-m75q
[Finished] 4b31aa4f ec2-t3-xlarge-us-east-2
[Failed] 4b31aa4f test-mac-arm
[Finished] 4b31aa4f ursa-i9-9960x
[Finished] 4b31aa4f ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java