[Python] Improve the pyarrow build system

[jorisvandenbossche]

The current build system of pyarrow consists of using setuptools with a custom setup.py that calls cmake to do the actual build. While this works, there are nowadays newer python "build backends" that we could benefit from to make this easier to maintain and use.
Specifically build backends that are meant to work with CMake projects, such as scikit-build, might be worth exploring.

[raulcd]

Thanks @jorisvandenbossche !

[h-vetinari]

Specifically build backends that are meant to work with CMake projects, such as scikit-build, might be worth exploring.

Scipy has pioneered meson as the backend (in the wider python ecosystem at least), which is now being used by numpy & pandas as well, and is being picked up by more and more projects. For a regular python project it's probably a better choice than CMake-based builds (the syntax is already nice and pythonic too), though if you have substantial CMake infrastructure then that integrates more easily with scikit-build.

[jorisvandenbossche]

though if you have substantial CMake infrastructure then that integrates more easily with scikit-build.

We do have that, yes: https://github.com/apache/arrow/blob/main/python/CMakeLists.txt

[h-vetinari]

I'm aware, I've been building arrow for a while now. ;-)
It was more a rhetorical statement - depends a bit on what one considers "substantial" too. :)

[jorisvandenbossche]

Also relevant issue: #36410

[jorisvandenbossche]

Major wish-list item: proper incremental / cached cython builds.

[vyasr]

I'll note that if scikit-build is still of interest, the modern solution here would be to use scikit-build-core