[C++] Split non-cast compute kernels into a separate shared library

[asfimport]

Since we are going to implement a lot more precompiled kernels, I am not sure it makes sense to require all of them to be compiled unconditionally just to get access to compute::Cast, which is needed in many different contexts.

After ARROW-8792 is merged, I would suggest creating a plugin hook for adding a bundle of kernels from a shared library outside of libarrow.so, and then moving all the object code outside of Cast to something like libarrow_compute.so. Then we can change the CMake flags to compile Cast kernels always (?) and then opt in to building the additional kernels package separately

Reporter: Wes McKinney / @wesm
Watchers: Rok Mihevc / @rok

Related issues:

• [C++] C++ array kernels framework and execution buildout (umbrella issue) (relates to)
• [Python] Allow disabling more components (relates to)

_{Note: This issue was originally created as ARROW-8891. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
I glanced at this, but since dictionary casts use Take, either we have to restore the non-Take implementation and duplicate code paths, include Take libarrow.so, or relegate dictionary casts to libarrow_compute.so.

[benibus]

@lidavidm @westonpace @zeroshade @felipecrv
I've been working on the PR for this - just wanted to give an update and get some design opinions.

Currently, I've set things up in the build system so that everything outside of compute/kernels is built into libarrow unconditionally - along with:

• scalar_cast_*.cc
• vector_selection.cc for take (used by dictionary casts, parquet)
• vector_hash.cc for unique (used by parquet)

If ARROW_COMPUTE is enabled, then the remaining kernel sources are built into libarrow_compute, which links against libarrow. Alternatively, we could introduce a new flag for this, but as it stands, -DARROW_COMPUTE=ON still gives you all the kernels (and we could unconditionally compile code that uses casts - i.e. the CSV writer, STL tests, etc).

Assuming that sounds reasonable (if not, let me know), I just need to determine how registration will work for the extra kernels. Currently, this is done on the first invocation of compute::GetFunctionRegistry in libarrow.so. However, it'll no longer be possible to register the extra kernels in this way given that their registration functions are in a different object that libarrow doesn't link against. As I see it, there are a couple possibilities:

• Load the kernels from libarrow_compute at runtime (more flexible/forward-looking, but platform-dependent)
• Circularly link libarrow and libarrow_compute (would require fancy linker flags). You might be able to get the same effect with an intermediate library but I'm not sure if/how it would work in practice.

Any suggestions? Perhaps I'm missing a more obvious solution. @wesm's original post suggests creating a "plugin hook" for loading kernels from an external lib. To me, that sounds like a dlopen type of deal, but I'm not positive.

[lidavidm]

You could require an explicit call to register the functions; the bindings could hide this call in their own machinery (it would be more inconvenient for C++ users, though you could make that call in the various wrappers we have in places like api_scalar.h). You could also play games with static initializers to call this function for you.

[benibus]

I did play around with that idea, although I'm a little hesitant to commit to explicit initialization since I fear the C++ breakage would be severe - primarily because we wouldn't be able to access the extended registry from within libarrow itself. For instance, ExecContext's default constructor wouldn't be able to implicitly register those kernels outside of libarrow_compute, so any user/test code that utilizes equal or add would now need to proactively load the registry. You could get around that in the wrappers to some extent, but it wouldn't be complete.

I should mention, another alternative would be to not build the separate lib at all - instead, just optionally include those kernels in libarrow. That might be the approach I take for the initial PR (we'd have a more concrete starting point, at least).

[lidavidm]

I like that option. That solves the main problem, which is just being able to assume availability of Cast.

[westonpace]

I'm fine with an explicit registration call. Pyarrow (and R) could easily call this for their users. This just leaves those using the C++ API directly and it is a single call when setting up.

Yes, it would be rather annoying for tests. However, we can probably do something like this.

[jorisvandenbossche]

I should mention, another alternative would be to not build the separate lib at all - instead, just optionally include those kernels in libarrow. That might be the approach I take for the initial PR (we'd have a more concrete starting point, at least).

That solves the main problem, which is just being able to assume availability of Cast.

We can argue about what is the "main" problem, but this certainly doesn't solve the use case of being able to install the compute kernels optionally/separately, which a separate shared library would enable (eg a package manager like conda could package libarrow_compute separately from libarrow, so that the user can determine if they need all compute or not. While with the above solution, it's the developer/packager that needs to make this decision upfront if they include all compute kernels or not at compile time)

[lidavidm]

Sure, it would be nice to be able to do that.

That would require a much bigger overhaul of our build system though, since you can't currently even build, say libparquet.so separately from Arrow itself.

[felipecrv]

We can leave this issue open and associate #34295 with a sub issue concerned with the ability of configuring the build via ARROW_COMPUTE (which is what that PR fixes).

[jorisvandenbossche]

That would require a much bigger overhaul of our build system though, since you can't currently even build, say libparquet.so separately from Arrow itself.

I don't think for packagers that is necessarily a problem, as long as you can install both so's separately? (which you can already do now AFAIK, eg you can install just libarrow.so without also installing libparquet.so)

[lidavidm]

Fair. For conda-forge though, are you able to do that? It seems to be one build = one package (I suppose you could game the build steps).