ARROW-16340: [C++][Python] Move all Python related code into PyArrow #13311

AlenkaF · 2022-06-03T13:37:50Z

This PR moves src/arrow/python directory into pyarrow and arranges PyArrow to build it. The build on the Python side is made in two steps:

_run_cmake_pyarrow_cpp() where the C++ part of the pyarrow is build first (the part that was moved in the refactoring)
_run_cmake() where pyarrow is built as before

No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with:

>>> pushd python/build/dist/temp 
>>> ctest

github-actions · 2022-06-03T13:38:19Z

https://issues.apache.org/jira/browse/ARROW-16340

github-actions · 2022-06-03T13:38:22Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

python/pyarrow/src_arrow/CMakeLists.txt

amol- · 2022-06-10T13:28:38Z

cpp/cmake_modules/FindArrowPython.cmake

  arrow_find_package(ARROW_PYTHON
-                     "${ARROW_HOME}"
+                     "${CPYARROW_HOME}"


This smells a bit hackish. Feels a bit like we are building two totally different projects.

On long term we should probably have libarrow_python.so just be one of the shared objects constituting pyarrow. libarrow_python.so should probably be possible to integrate with the rest of the built files using add_subdirectory or something equivalent.

On long term we should probably have libarrow_python.so just be one of the shared objects constituting pyarrow.

That's actually what is already happening, I think. The libarrow_python.so gets copied into the pyarrow directory (either the repo itself for an inplace build, or the build directory, before that one gets copied to site-packages for a normal install).
But so for building the cython extensions, we need to point to where libarrow_python.so can be found (which is now a different location as where libarrow.so can be found). And doing that with a argument to cmake seems a good way to do that? (alternatively we might need to edit the arrow_find_package to work with this new situation?)

python/pyarrow/src_arrow/CMakeLists.txt

AlenkaF · 2022-06-28T10:16:43Z

@kou there is one more issue I am struggling with and would be happy to get your opinion.

On the CI I get linker errors for python flight module (not locally on M1):
https://github.com/apache/arrow/runs/7087113174?check_suite_focus=true

I am facing similar issue locally when trying to build the tests for C++ part of PyArrow with GTest:

[100%] Linking CXX executable arrow-python-test
ld: library not found for -larrow_testing_shared
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [arrow-python-test] Error 1
make[1]: *** [CMakeFiles/arrow-python-test.dir/all] Error 2
make: *** [all] Error 2
error: command '/usr/bin/make' failed with exit code 2
~/repos/arrow-cmake

What could be the setup work I am missing so that the libraries would be linked correctly?

kou · 2022-06-28T20:18:14Z

On the CI I get linker errors for python flight module (not locally on M1):
https://github.com/apache/arrow/runs/7087113174?check_suite_focus=true

It seems that this link is wrong. (This doesn't have the error message.)

What could be the setup work I am missing so that the libraries would be linked correctly?

We need find_package(ArrowTesting REQUIRED) to use arrow_testing_shared target.

AlenkaF · 2022-06-29T06:09:45Z

On the CI I get linker errors for python flight module (not locally on M1):
https://github.com/apache/arrow/runs/7087113174?check_suite_focus=true

It seems that this link is wrong. (This doesn't have the error message.)

Maybe this way would be easier:

Error with make build of C PyArrow

-- Running cmake for C pyarrow
  cmake -DCMAKE_INSTALL_PREFIX=/arrow/python/build/dist -DCMAKE_BUILD_TYPE=debug -DPYARROW_WITH_DATASET=on -DPYARROW_WITH_PARQUET_ENCRYPTION=on -DPYARROW_WITH_HDFS=on /arrow/python/pyarrow/src_arrow
  -- CMAKE_MODULE_PATH /arrow/python/cmake_modules;/opt/conda/envs/arrow/lib/cmake/arrow
  -- The C compiler identification is GNU 10.3.0
  -- The CXX compiler identification is GNU 10.3.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found PkgConfig: /opt/conda/envs/arrow/bin/pkg-config (found version "0.29.2")
  -- Found Arrow: /opt/conda/envs/arrow/include (found version "9.0.0")
  -- Arrow version: 9.0.0 (HOME: /opt/conda/envs/arrow)
  -- Arrow SO and ABI version: 900
  -- Arrow full SO version: 900.0.0
  -- Found the Arrow core shared library: /opt/conda/envs/arrow/lib/libarrow.so
  -- Found the Arrow core import library: /opt/conda/envs/arrow/lib/libarrow.so
  -- Found the Arrow core static library: ARROW_static_lib-NOTFOUND
  -- Found Python3: /opt/conda/envs/arrow/bin/python3.9 (found suitable version "3.9.13", minimum required is "3.7") found components: Interpreter Development.Module NumPy
  -- Found Python3Alt: /opt/conda/envs/arrow/bin/python3.9 (Required is at least version "3.7")
  CMake Warning (dev) at CMakeLists.txt:55 (option):
    Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
    --help-policy CMP0077" for policy details.  Use the cmake_policy command to
    set the policy and suppress this warning.
    For compatibility with older versions of CMake, option is clearing the
    normal variable 'ARROW_BUILD_SHARED'.
  This warning is for project developers.  Use -Wno-dev to suppress it.
  -- Found ArrowDataset: /opt/conda/envs/arrow/include (found version "9.0.0")
  -- Found the Arrow Dataset by HOME: /opt/conda/envs/arrow
  -- Found the Arrow Dataset shared library: /opt/conda/envs/arrow/lib/libarrow_dataset.so
  -- Found the Arrow Dataset import library: /opt/conda/envs/arrow/lib/libarrow_dataset.so
  -- Found the Arrow Dataset static library: ARROW_DATASET_static_lib-NOTFOUND
  -- Performing Test CXX_LINKER_SUPPORTS_VERSION_SCRIPT
  -- Performing Test CXX_LINKER_SUPPORTS_VERSION_SCRIPT - Success
  -- Found ArrowFlight: /opt/conda/envs/arrow/include (found version "9.0.0")
  -- Found the Arrow Flight by HOME: /opt/conda/envs/arrow
  -- Found the Arrow Flight shared library: /opt/conda/envs/arrow/lib/libarrow_flight.so
  -- Found the Arrow Flight import library: /opt/conda/envs/arrow/lib/libarrow_flight.so
  -- Found the Arrow Flight static library: ARROW_FLIGHT_static_lib-NOTFOUND
  -- Found ZLIB: /opt/conda/envs/arrow/lib/libz.so (found version "1.2.12")
  -- Found Protobuf: /opt/conda/envs/arrow/lib/libprotobuf.so;-lpthread (found version "3.19.4")
  -- Found OpenSSL: /opt/conda/envs/arrow/lib/libcrypto.so (found version "1.1.1o")
  -- Found c-ares: /opt/conda/envs/arrow/lib/cmake/c-ares/c-ares-config.cmake (found version "1.18.1")
  -- Found Threads: TRUE
  -- Check if compiler accepts -pthread
  -- Check if compiler accepts -pthread - yes
  -- Found RE2 via CMake.
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /arrow/python/build/dist/temp
  -- Finished cmake for C pyarrow
  -- Running make build and install for C pyarrow
  make -j4
  [  3%] Generating /arrow/cpp/build/src/arrow/flight/Flight.pb.cc, /arrow/cpp/build/src/arrow/flight/Flight.pb.h, /arrow/cpp/build/src/arrow/flight/Flight.grpc.pb.cc, /arrow/cpp/build/src/arrow/flight/Flight.grpc.pb.h
  make[2]: I/arrow/format: No such file or directory
  make[2]: I/arrow/format: No such file or directory
  [  7%] Building CXX object CMakeFiles/arrow_python_objlib.dir/arrow_to_pandas.cc.o
  [ 11%] Building CXX object CMakeFiles/arrow_python_objlib.dir/benchmark.cc.o
  [ 15%] Building CXX object CMakeFiles/arrow_python_objlib.dir/common.cc.o
  [ 15%] Built target flight_grpc_gen
  [ 19%] Building CXX object CMakeFiles/arrow_python_objlib.dir/datetime.cc.o
  [ 23%] Building CXX object CMakeFiles/arrow_python_objlib.dir/decimal.cc.o
  [ 26%] Building CXX object CMakeFiles/arrow_python_objlib.dir/deserialize.cc.o
  [ 30%] Building CXX object CMakeFiles/arrow_python_objlib.dir/extension_type.cc.o
  [ 34%] Building CXX object CMakeFiles/arrow_python_objlib.dir/gdb.cc.o
  [ 38%] Building CXX object CMakeFiles/arrow_python_objlib.dir/helpers.cc.o
  [ 42%] Building CXX object CMakeFiles/arrow_python_objlib.dir/inference.cc.o
  [ 46%] Building CXX object CMakeFiles/arrow_python_objlib.dir/init.cc.o
  [ 50%] Building CXX object CMakeFiles/arrow_python_objlib.dir/io.cc.o
  [ 53%] Building CXX object CMakeFiles/arrow_python_objlib.dir/ipc.cc.o
  [ 57%] Building CXX object CMakeFiles/arrow_python_objlib.dir/numpy_convert.cc.o
  [ 61%] Building CXX object CMakeFiles/arrow_python_objlib.dir/numpy_to_arrow.cc.o
  [ 65%] Building CXX object CMakeFiles/arrow_python_objlib.dir/python_to_arrow.cc.o
  [ 69%] Building CXX object CMakeFiles/arrow_python_objlib.dir/pyarrow.cc.o
  [ 73%] Building CXX object CMakeFiles/arrow_python_objlib.dir/serialize.cc.o
  [ 76%] Building CXX object CMakeFiles/arrow_python_objlib.dir/udf.cc.o
  [ 80%] Building CXX object CMakeFiles/arrow_python_objlib.dir/csv.cc.o
  [ 84%] Building CXX object CMakeFiles/arrow_python_objlib.dir/filesystem.cc.o
  [ 88%] Building CXX object CMakeFiles/arrow_python_objlib.dir/parquet_encryption.cc.o
  [ 88%] Built target arrow_python_objlib
  [ 92%] Linking CXX shared library libarrow_python.so
  /opt/conda/envs/arrow/bin/../lib/gcc/x86_64-conda-linux-gnu/10.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lparquet_shared
  collect2: error: ld returned 1 exit status
  make[2]: *** [CMakeFiles/arrow_python_shared.dir/build.make:126: libarrow_python.so.900.0.0] Error 1
  make[1]: *** [CMakeFiles/Makefile2:202: CMakeFiles/arrow_python_shared.dir/all] Error 2
  make: *** [Makefile:136: all] Error 2
  error: command '/opt/conda/envs/arrow/bin/make' failed with exit code 2
  error: subprocess-exited-with-error

kou · 2022-06-29T06:17:10Z

We need find_package(Parquet REQUIRED) for parquet_shared.

AlenkaF · 2022-06-29T06:27:05Z

Thank you @kou!

AlenkaF · 2022-07-01T04:39:29Z

I would like to put this PR ready for review.

There are still two CI failures I am investigating (see ⬇️ ) plus test_cython_api() from test_cython.py is broke, so I am researching it at the moment. But I think this shouldn't block this PR to get initial reviews 🙏 @kou @pitrou @kszucs @jorisvandenbossche

Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy)
I have seen this error in some other failing builds and am not sure if it is connected to this PR or not.

  CMake Error at /opt/conda/envs/arrow/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found
    suitable version "3.9.13", minimum required is "3.7")
  Call Stack (most recent call first):
    /opt/conda/envs/arrow/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
    /opt/conda/envs/arrow/share/cmake-3.23/Modules/FindPython/Support.cmake:3192 (find_package_handle_standard_args)
    /opt/conda/envs/arrow/share/cmake-3.23/Modules/FindPython3.cmake:490 (include)
    /arrow/python/cmake_modules/FindPython3Alt.cmake:56 (find_package)
    CMakeLists.txt:64 (find_package)

fatal error: arrow/flight/Flight.pb.h: No such file or directory
This error is for sure connected to this PR. I have bumped into it locally for python flight build but it dissapeared after I continued making changes to the code.

  In file included from /arrow/cpp/src/arrow/flight/serialization_internal.h:22,
                   from /arrow/python/pyarrow/src_arrow/flight.cc:21:
  /arrow/cpp/src/arrow/flight/protocol_internal.h:23:10: fatal error: arrow/flight/Flight.pb.h: No such file or directory
     23 | #include "arrow/flight/Flight.pb.h"  // IWYU pragma: export
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  make[2]: *** [CMakeFiles/arrow_python_flight_objlib.dir/build.make:76: CMakeFiles/arrow_python_flight_objlib.dir/flight.cc.o] Error 1
  make[1]: *** [CMakeFiles/Makefile2:255: CMakeFiles/arrow_python_flight_objlib.dir/all] Error 2
  make: *** [Makefile:136: all] Error 2
  error: command '/opt/conda/envs/arrow/bin/make' failed with exit code 2
  error: subprocess-exited-with-error

cpp/cmake_modules/FindArrowPython.cmake

cpp/src/arrow/public_api_test.cc

python/pyarrow/src_arrow/CMakeLists.txt

python/setup.py

pitrou · 2022-07-01T08:33:11Z

Hmm, we should try to make sure the Python Flight bindings can compile without including some internal headers.

AlenkaF · 2022-07-01T09:36:57Z

Hmm, we should try to make sure the Python Flight bindings can compile without including some internal headers.

Are you thinking in terms of this PR or in general?

pitrou · 2022-07-01T10:05:25Z

I mean in general, though that could be done as part of this PR (unless I'm missing something that makes internal headers mandatory here, but that would be surprising).

lidavidm · 2022-07-01T12:00:40Z

I think it should be easy enough to remove the need for serialization_internal.h inside python/flight.cc (use Make instead of directly constructing the structure)

AlenkaF · 2022-07-01T12:10:55Z

Thanks for the comments!
In that case I will try to remove the internal headers for Python Flight and see how it goes.

AlenkaF · 2022-07-04T08:45:24Z

Is there something similar I can do with this part:

arrow/cpp/src/arrow/python/flight.cc

Lines 385 to 393 in 38918ef

    
           Status CreateSchemaResult(const std::shared_ptr<arrow::Schema>& schema, 
        
                                     std::unique_ptr<arrow::flight::SchemaResult>* out) { 
        
             std::string schema_in; 
        
             RETURN_NOT_OK(arrow::flight::internal::SchemaToString(*schema, &schema_in)); 
        
             arrow::flight::SchemaResult value(schema_in); 
        
             *out = std::unique_ptr<arrow::flight::SchemaResult>( 
        
                 new arrow::flight::SchemaResult(value)); 
        
             return Status::OK(); 
        
           }

that also uses arrow::flight::internal::SchemaToString? @lidavidm

I have successfully changed the construction of FlightInfo for Status CreateFlightInfo in python/flight.cc.

lidavidm · 2022-07-04T12:24:04Z

Hmm, looks like we need to add a factory method for SchemaResult. There isn't anything available already

AlenkaF · 2022-08-25T08:05:37Z

I have rebased the PR and corrected the arrow.soec.in file, hope I got it right @kou? The changes can be found here.
If yes I can run the nightly checks again.

kou · 2022-08-25T08:19:47Z

Yes. Thanks.
Sorry, I miss one more thing. Could you also remove python*-devel and python*-numpy from dev/tasks/linux-packages/apache-arrow/yum/*/Dockerfiles?

…apache-arrow/yum/*/Dockerfile

AlenkaF · 2022-08-25T08:29:54Z

@github-actions crossbow submit -g nightly-tests -g nightly-packaging -g nightly-release

github-actions · 2022-08-25T10:31:36Z

Revision: 0860306

Submitted crossbow builds: ursacomputing/crossbow @ actions-58857300ac

Task	Status
almalinux-8-amd64
almalinux-8-arm64
almalinux-9-amd64
almalinux-9-arm64
amazon-linux-2-amd64
amazon-linux-2-arm64
centos-7-amd64
centos-8-stream-amd64
centos-8-stream-arm64
centos-9-stream-amd64
centos-9-stream-arm64
conan-maximum
conan-minimum
conda-clean
conda-linux-gcc-py310-arm64
conda-linux-gcc-py310-cpu
conda-linux-gcc-py310-cuda
conda-linux-gcc-py310-ppc64le
conda-linux-gcc-py37-arm64
conda-linux-gcc-py37-cpu-r40
conda-linux-gcc-py37-cpu-r41
conda-linux-gcc-py37-cuda
conda-linux-gcc-py37-ppc64le
conda-linux-gcc-py38-arm64
conda-linux-gcc-py38-cpu
conda-linux-gcc-py38-cuda
conda-linux-gcc-py38-ppc64le
conda-linux-gcc-py39-arm64
conda-linux-gcc-py39-cpu
conda-linux-gcc-py39-cuda
conda-linux-gcc-py39-ppc64le
conda-osx-arm64-clang-py310
conda-osx-arm64-clang-py38
conda-osx-arm64-clang-py39
conda-osx-clang-py310
conda-osx-clang-py37-r40
conda-osx-clang-py37-r41
conda-osx-clang-py38
conda-osx-clang-py39
conda-win-vs2017-py310
conda-win-vs2017-py37-r40
conda-win-vs2017-py37-r41
conda-win-vs2017-py38
conda-win-vs2017-py39
debian-bookworm-amd64
debian-bookworm-arm64
debian-bullseye-amd64
debian-bullseye-arm64
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
homebrew-cpp
homebrew-r-autobrew
homebrew-r-brew
java-jars
nuget
python-sdist
r-binary-packages
test-alpine-linux-cpp
test-build-cpp-fuzz
test-build-vcpkg-win
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.7
test-conda-python-3.7-hdfs-2.9.2
test-conda-python-3.7-hdfs-3.2.1
test-conda-python-3.7-kartothek-latest
test-conda-python-3.7-kartothek-master
test-conda-python-3.7-pandas-0.24
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8
test-conda-python-3.8-hypothesis
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9
test-conda-python-3.9-dask-latest
test-conda-python-3.9-dask-master
test-conda-python-3.9-pandas-master
test-conda-python-3.9-spark-master
test-debian-10-cpp-amd64
test-debian-10-cpp-i386
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-debian-11-go-1.16
test-debian-11-python-3
test-debian-c-glib
test-debian-ruby
test-fedora-35-cpp
test-fedora-35-python-3
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rocker-r-base-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-skyhook-integration
test-ubuntu-18.04-cpp
test-ubuntu-18.04-cpp-release
test-ubuntu-18.04-cpp-static
test-ubuntu-18.04-r-sanitizer
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-14
test-ubuntu-20.04-cpp-17
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-20.04-python-3
test-ubuntu-22.04-cpp
test-ubuntu-c-glib
test-ubuntu-default-docs
test-ubuntu-ruby
ubuntu-bionic-amd64
ubuntu-bionic-arm64
ubuntu-focal-amd64
ubuntu-focal-arm64
ubuntu-jammy-amd64
ubuntu-jammy-arm64
verify-rc-source-cpp-linux-almalinux-8-amd64
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-cpp-linux-ubuntu-18.04-amd64
verify-rc-source-cpp-linux-ubuntu-20.04-amd64
verify-rc-source-cpp-linux-ubuntu-22.04-amd64
verify-rc-source-cpp-macos-amd64
verify-rc-source-cpp-macos-arm64
verify-rc-source-cpp-macos-conda-amd64
verify-rc-source-csharp-linux-almalinux-8-amd64
verify-rc-source-csharp-linux-conda-latest-amd64
verify-rc-source-csharp-linux-ubuntu-18.04-amd64
verify-rc-source-csharp-linux-ubuntu-20.04-amd64
verify-rc-source-csharp-linux-ubuntu-22.04-amd64
verify-rc-source-csharp-macos-amd64
verify-rc-source-csharp-macos-arm64
verify-rc-source-go-linux-almalinux-8-amd64
verify-rc-source-go-linux-conda-latest-amd64
verify-rc-source-go-linux-ubuntu-18.04-amd64
verify-rc-source-go-linux-ubuntu-20.04-amd64
verify-rc-source-go-linux-ubuntu-22.04-amd64
verify-rc-source-go-macos-amd64
verify-rc-source-go-macos-arm64
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-18.04-amd64
verify-rc-source-integration-linux-ubuntu-20.04-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-integration-macos-amd64
verify-rc-source-integration-macos-arm64
verify-rc-source-integration-macos-conda-amd64
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-18.04-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64
verify-rc-source-js-linux-almalinux-8-amd64
verify-rc-source-js-linux-conda-latest-amd64
verify-rc-source-js-linux-ubuntu-18.04-amd64
verify-rc-source-js-linux-ubuntu-20.04-amd64
verify-rc-source-js-linux-ubuntu-22.04-amd64
verify-rc-source-js-macos-amd64
verify-rc-source-js-macos-arm64
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-18.04-amd64
verify-rc-source-python-linux-ubuntu-20.04-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64
verify-rc-source-ruby-linux-almalinux-8-amd64
verify-rc-source-ruby-linux-conda-latest-amd64
verify-rc-source-ruby-linux-ubuntu-18.04-amd64
verify-rc-source-ruby-linux-ubuntu-20.04-amd64
verify-rc-source-ruby-linux-ubuntu-22.04-amd64
verify-rc-source-ruby-macos-amd64
verify-rc-source-ruby-macos-arm64
verify-rc-source-windows
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-high-sierra-cp37-amd64
wheel-macos-high-sierra-cp38-amd64
wheel-macos-high-sierra-cp39-amd64
wheel-macos-mavericks-cp310-amd64
wheel-macos-mavericks-cp37-amd64
wheel-macos-mavericks-cp38-amd64
wheel-macos-mavericks-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

AlenkaF · 2022-08-26T04:35:20Z

Lots of green 😍 😆
@kou @pitrou the failures do not look related, all issues from before in the nightlies should be fixed 👍

kou

+1

The centos-9-stream-amd64 failure is a new failure but I think that it's not related to this pull request. We can fix it as a follow-up task.

https://github.com/ursacomputing/crossbow/runs/8027904112?check_suite_focus=true#step:6:4839

FAILED: gandiva-glib/Gandiva-1.0.gir 
/usr/bin/meson --internal exe --unpickle /build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/meson-private/meson_exe_g-ir-scanner_f62efc76b62635482d998451ca9762c4e34e9776.dat
while executing ['/usr/bin/g-ir-scanner', '--no-libtool', '--namespace=Gandiva', '--nsversion=1.0', '--warn-all', '--output', 'gandiva-glib/Gandiva-1.0.gir', '--c-include=gandiva-glib/gandiva-glib.h', '--warn-all', '--include-uninstalled=./arrow-glib/Arrow-1.0.gir', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/gandiva-glib', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/.', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/.', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/../cpp/redhat-linux-build/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../cpp/redhat-linux-build/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/../cpp/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../cpp/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/.', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/.', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/../cpp/redhat-linux-build/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../cpp/redhat-linux-build/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/../cpp/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../cpp/src', '--filelist=/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib/libgandiva-glib.so.1000.0.0.p/Gandiva_1.0_gir_filelist', '--include=Arrow-1.0', '--symbol-prefix=ggandiva', '--identifier-prefix=GGandiva', '--pkg-export=gandiva-glib', '--cflags-begin', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/.', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/.', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/../cpp/redhat-linux-build/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../cpp/redhat-linux-build/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/../cpp/src', '-I/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../cpp/src', '-I/usr/include/glib-2.0', '-I/usr/lib64/glib-2.0/include', '-I/usr/include/sysprof-4', '-I/usr/include/gobject-introspection-1.0', '--cflags-end', '--add-include-path=/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/arrow-glib', '--add-include-path=/usr/share/gir-1.0', '-L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib', '--library', 'gandiva-glib', '-L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/arrow-glib', '-L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../../cpp/redhat-linux-build/release', '--extra-library=gobject-2.0', '--extra-library=glib-2.0', '--extra-library=girepository-1.0', '--sources-top-dirs', '/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/subprojects/', '--sources-top-dirs', '/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/subprojects/', '--warn-error']
--- stdout ---
Package arrow was not found in the pkg-config search path.
Perhaps you should add the directory containing `arrow.pc'
to the PKG_CONFIG_PATH environment variable
Package 'arrow', required by 'arrow-glib-uninstalled', not found

g-ir-scanner: link: gcc -o /build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/tmp-introspecttrutbm6g/Gandiva-1.0 -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection /build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/tmp-introspecttrutbm6g/Gandiva-1.0.o -L. -Wl,-rpath,. -Wl,--no-as-needed -L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib -Wl,-rpath,/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib -L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/arrow-glib -Wl,-rpath,/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/arrow-glib -L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../../cpp/redhat-linux-build/release -Wl,-rpath,/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../../cpp/redhat-linux-build/release -lgandiva-glib -lgobject-2.0 -lglib-2.0 -lgirepository-1.0 -lgio-2.0 -lgobject-2.0 -Wl,--export-dynamic -lgmodule-2.0 -pthread -lglib-2.0 -lglib-2.0 -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1

--- stderr ---
/usr/bin/ld: /build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../../cpp/redhat-linux-build/release/libgandiva.so.1000: undefined reference to `std::__glibcxx_assert_fail(char const*, int, char const*, char const*)'
collect2: error: ld returned 1 exit status
linking of temporary binary failed: Command '['gcc', '-o', '/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/tmp-introspecttrutbm6g/Gandiva-1.0', '-O2', '-flto=auto', '-ffat-lto-objects', '-fexceptions', '-g', '-grecord-gcc-switches', '-pipe', '-Wall', '-Werror=format-security', '-Wp,-D_FORTIFY_SOURCE=2', '-Wp,-D_GLIBCXX_ASSERTIONS', '-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1', '-fstack-protector-strong', '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1', '-m64', '-march=x86-64-v2', '-mtune=generic', '-fasynchronous-unwind-tables', '-fstack-clash-protection', '-fcf-protection', '/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/tmp-introspecttrutbm6g/Gandiva-1.0.o', '-L.', '-Wl,-rpath,.', '-Wl,--no-as-needed', '-L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib', '-Wl,-rpath,/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/gandiva-glib', '-L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/arrow-glib', '-Wl,-rpath,/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/arrow-glib', '-L/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../../cpp/redhat-linux-build/release', '-Wl,-rpath,/build/rpmbuild/BUILD/apache-arrow-9.0.0.dev783/c_glib/build/../../cpp/redhat-linux-build/release', '-lgandiva-glib', '-lgobject-2.0', '-lglib-2.0', '-lgirepository-1.0', '-lgio-2.0', '-lgobject-2.0', '-Wl,--export-dynamic', '-lgmodule-2.0', '-pthread', '-lglib-2.0', '-lglib-2.0', '-Wl,-z,relro', '-Wl,--as-needed', '-Wl,-z,now', '-specs=/usr/lib/rpm/redhat/redhat-hardened-ld', '-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1']' returned non-zero exit status 1.

kou · 2022-08-26T04:46:16Z

Could you update the description of this pull request before we merge this?

AlenkaF · 2022-08-26T06:38:25Z

Will also make a JIRA for centos-9-stream-amd64 failure.

AlenkaF · 2022-08-26T06:54:11Z

@kou Description is updated and the JIRA issue for the centos-9-stream-amd64 failure can be found here: https://issues.apache.org/jira/browse/ARROW-17536.

kou · 2022-08-26T07:27:13Z

Thanks!
I merge this.

ursabot · 2022-08-26T10:01:18Z

Benchmark runs are scheduled for baseline = 7e7b8e1 and contender = b832853. b832853 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Failed ⬇️6.03% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.25% ⬆️0.39%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] b832853b ec2-t3-xlarge-us-east-2
[Failed] b832853b test-mac-arm
[Failed] b832853b ursa-i9-9960x
[Finished] b832853b ursa-thinkcentre-m75q
[Finished] 7e7b8e1f ec2-t3-xlarge-us-east-2
[Finished] 7e7b8e1f test-mac-arm
[Failed] 7e7b8e1f ursa-i9-9960x
[Finished] 7e7b8e1f ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-08-26T10:01:28Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

…pache#13311) This PR moves `src/arrow/python` directory into `pyarrow` and arranges PyArrow to build it. The build on the Python side is made in two steps: 1. `_run_cmake_pyarrow_cpp()` where the C++ part of the pyarrow is build first (the part that was moved in the refactoring) 2. `_run_cmake()` where pyarrow is built as before No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with: ```shell >>> pushd python/build/dist/temp >>> ctest ``` Lead-authored-by: Alenka Frim <frim.alenka@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…es (#14275) This PR is a follow-up of #13311 where it was decided to change the base directory for PyArrow C++ headers to avoid top level inclusions. See #13311 (comment) Authored-by: Alenka Frim <frim.alenka@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

…pache#13311) This PR moves `src/arrow/python` directory into `pyarrow` and arranges PyArrow to build it. The build on the Python side is made in two steps: 1. `_run_cmake_pyarrow_cpp()` where the C++ part of the pyarrow is build first (the part that was moved in the refactoring) 2. `_run_cmake()` where pyarrow is built as before No changes are needed in the build process from the user side to successfully build pyarrow after this refactoring. The test for PyArrow CPP will however be moved into Cython and can currently be run with: ```shell >>> pushd python/build/dist/temp >>> ctest ``` Lead-authored-by: Alenka Frim <frim.alenka@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…es (apache#14275) This PR is a follow-up of apache#13311 where it was decided to change the base directory for PyArrow C++ headers to avoid top level inclusions. See apache#13311 (comment) Authored-by: Alenka Frim <frim.alenka@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

…14498) Remove all instances of `ARROW_BUILD_DIR` environment variable that was aded in the PyArrow refactoring PR #13311. Authored-by: Alenka Frim <frim.alenka@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions bot added Component: C++ Component: Python labels Jun 3, 2022

kou mentioned this pull request Jun 4, 2022

ARROW-16609: [C++] xxhash not installed into dist/lib/include when building C++ #13282

Merged

AlenkaF force-pushed the ARROW-16340 branch from 759307c to 9ef0936 Compare June 8, 2022 10:06

raulcd reviewed Jun 10, 2022

View reviewed changes

python/pyarrow/src_arrow/CMakeLists.txt Outdated Show resolved Hide resolved

amol- reviewed Jun 10, 2022

View reviewed changes

python/pyarrow/src_arrow/CMakeLists.txt Outdated Show resolved Hide resolved

amol- reviewed Jun 10, 2022

View reviewed changes

AlenkaF force-pushed the ARROW-16340 branch from 87f81d1 to 8b6159c Compare June 15, 2022 06:16

AlenkaF force-pushed the ARROW-16340 branch 2 times, most recently from abbdece to b2f150c Compare June 23, 2022 08:15

AlenkaF commented Jun 27, 2022

View reviewed changes

python/pyarrow/src_arrow/CMakeLists.txt Outdated Show resolved Hide resolved

AlenkaF marked this pull request as ready for review July 1, 2022 04:39

kou reviewed Jul 1, 2022

View reviewed changes

AlenkaF force-pushed the ARROW-16340 branch from 13ab0cd to 997a9b9 Compare July 1, 2022 07:49

AlenkaF force-pushed the ARROW-16340 branch from 997a9b9 to ea1da51 Compare July 1, 2022 08:34

AlenkaF force-pushed the ARROW-16340 branch from 153a1cb to 767dda9 Compare August 25, 2022 08:03

Remove python*-devel and python*-numpy from dev/tasks/linux-packages/…

0860306

…apache-arrow/yum/*/Dockerfile

kou approved these changes Aug 26, 2022

View reviewed changes

kou changed the title ~~ARROW-16340: [Python] Move all Python related code into PyArrow~~ ARROW-16340: [C++][Python] Move all Python related code into PyArrow Aug 26, 2022

kou merged commit b832853 into apache:master Aug 26, 2022

AlenkaF deleted the ARROW-16340 branch September 16, 2022 06:35

AlenkaF mentioned this pull request Sep 29, 2022

ARROW-17160: [C++] Create a base directory for PyArrow CPP header files #14275

Merged

AlenkaF mentioned this pull request Oct 25, 2022

ARROW-18126: [Python] Remove ARROW_BUILD_DIR in building pyarrow C++ #14498

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-16340: [C++][Python] Move all Python related code into PyArrow #13311

ARROW-16340: [C++][Python] Move all Python related code into PyArrow #13311

AlenkaF commented Jun 3, 2022 •

edited

github-actions bot commented Jun 3, 2022

github-actions bot commented Jun 3, 2022

amol- Jun 10, 2022

jorisvandenbossche Jun 16, 2022

AlenkaF commented Jun 28, 2022

kou commented Jun 28, 2022

AlenkaF commented Jun 29, 2022

kou commented Jun 29, 2022

AlenkaF commented Jun 29, 2022

AlenkaF commented Jul 1, 2022

pitrou commented Jul 1, 2022

AlenkaF commented Jul 1, 2022

pitrou commented Jul 1, 2022

lidavidm commented Jul 1, 2022

AlenkaF commented Jul 1, 2022

AlenkaF commented Jul 4, 2022

lidavidm commented Jul 4, 2022

AlenkaF commented Aug 25, 2022

kou commented Aug 25, 2022

AlenkaF commented Aug 25, 2022

github-actions bot commented Aug 25, 2022

AlenkaF commented Aug 26, 2022

kou left a comment

kou commented Aug 26, 2022

AlenkaF commented Aug 26, 2022

AlenkaF commented Aug 26, 2022

kou commented Aug 26, 2022

ursabot commented Aug 26, 2022

ursabot commented Aug 26, 2022

ARROW-16340: [C++][Python] Move all Python related code into PyArrow #13311

ARROW-16340: [C++][Python] Move all Python related code into PyArrow #13311

Conversation

AlenkaF commented Jun 3, 2022 • edited

github-actions bot commented Jun 3, 2022

github-actions bot commented Jun 3, 2022

amol- Jun 10, 2022

Choose a reason for hiding this comment

jorisvandenbossche Jun 16, 2022

Choose a reason for hiding this comment

AlenkaF commented Jun 28, 2022

kou commented Jun 28, 2022

AlenkaF commented Jun 29, 2022

kou commented Jun 29, 2022

AlenkaF commented Jun 29, 2022

AlenkaF commented Jul 1, 2022

pitrou commented Jul 1, 2022

AlenkaF commented Jul 1, 2022

pitrou commented Jul 1, 2022

lidavidm commented Jul 1, 2022

AlenkaF commented Jul 1, 2022

AlenkaF commented Jul 4, 2022

lidavidm commented Jul 4, 2022

AlenkaF commented Aug 25, 2022

kou commented Aug 25, 2022

AlenkaF commented Aug 25, 2022

github-actions bot commented Aug 25, 2022

AlenkaF commented Aug 26, 2022

kou left a comment

Choose a reason for hiding this comment

kou commented Aug 26, 2022

AlenkaF commented Aug 26, 2022

AlenkaF commented Aug 26, 2022

kou commented Aug 26, 2022

ursabot commented Aug 26, 2022

ursabot commented Aug 26, 2022

AlenkaF commented Jun 3, 2022 •

edited