[Python] wheel cmake error on Raspberry PI

[sdy72]

Describe the usage question you have. Please include as many useful details as possible.

Dear Arrow

When I try to install pyarrow on my Raspberry PI 3 using pip I get this error

  CMake Error at /usr/local/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found
    suitable version "3.9.2", minimum required is "3.7")
  Call Stack (most recent call first):
    /usr/local/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
    /usr/local/share/cmake-3.25/Modules/FindPython/Support.cmake:3245 (find_package_handle_standard_args)
    /usr/local/share/cmake-3.25/Modules/FindPython3.cmake:490 (include)
    /tmp/pip-install-h0q930xi/pyarrow_e68ff656d2a84a388a1cc31acfc338a9/cmake_modules/FindPython3Alt.cmake:60 (find_package)

Full output attached. I didn't know where to ask or if it should be a bug report instead. So I start here.

System is "Raspbian GNU/Linux 11 (bullseye) on Raspberry PI 3 armhf.
I tried both with numpy installed from apt and pip, and cmake install from apt and latest build from source.
Same error message.

Please let me know if I should provide info or how I can help you help me.
My main objective is to install streamlit which has a dependency to pyarrow via pip.

pyarrow_install_stdout.txt

Component(s)

Python

[kou]

Could you show the output of /usr/bin/python3 -c 'import numpy; print(numpy.get_include())'?

[sdy72]

Sure:
pi@raspberrypi:~ $ /usr/bin/python3 -c 'import numpy; print(numpy.get_include())'
/usr/lib/python3/dist-packages/numpy/core/include

There are 26 .h files in
/usr/lib/python3/dist-packages/numpy/core/include/numpy/

[kou]

Hmm. there is no problem for it...

Can you add debug prints (message("...")) to /usr/local/share/cmake-3.25/Modules/FindPython/Support.cmake (line 3145-3226) https://gitlab.kitware.com/cmake/cmake/-/blob/v3.25.1/Modules/FindPython/Support.cmake#L3154-3226 ?

[sdy72]

Thanks kou
I played around with messages you suggested and found something that could be the issue.
I attached a little cmake toy example and I was able to reproduce an error on both my Ubuntu/22.04 LTS (cmake 3.22.1)
as well as my Raspberry PI described above.

I looked at this:

arrow/cpp/cmake_modules/FindPython3Alt.cmake

Line 60 in 16d40c8

find_package(Python3 ${Python3Alt_FIND_PACKAGE_OPTIONS}

It makes a difference whether you put this
find_package(Python3 ...
or
find_package(Python ...

Please check this toy example:
CMakeLists.txt

[kou]

Thanks! Could you show the output of the following CMakeLists.txt?

cmake_minimum_required(VERSION 3.20)
project(hello LANGUAGES CXX)

## this works:
find_package(Python 3.7 REQUIRED COMPONENTS Interpreter Development NumPy)

message(STATUS "#### Python_NumPy_VERSION = ${Python_NumPy_VERSION} ")
message(STATUS "#### Python_NumPy_FOUND = ${Python_NumPy_FOUND} ")
message(STATUS "#### Python_NumPy_INCLUDE_DIRS = ${Python_NumPy_INCLUDE_DIRS} ")
message(STATUS "#### Python_EXECUTABLE = ${Python_EXECUTABLE} ")

## this leaves Python_NumPy_INCLUDE_DIRS and the other variables empty
find_package(Python3 3.7 REQUIRED COMPONENTS Interpreter Development NumPy)

message(STATUS "#### Python3_NumPy_VERSION = ${Python3_NumPy_VERSION} ")
message(STATUS "#### Python3_NumPy_FOUND = ${Python3_NumPy_FOUND} ")
message(STATUS "#### Python3_NumPy_INCLUDE_DIRS = ${Python3_NumPy_INCLUDE_DIRS} ")
message(STATUS "#### Python3_EXECUTABLE = ${Python3_EXECUTABLE} ")

add_executable(hello "hello.cpp")

[sdy72]

My bad. That works both places.

Another observation:
After https://gitlab.kitware.com/cmake/cmake/-/blob/v3.25.1/Modules/FindPython/Support.cmake
lines 3170-3176

_Python3_NumPy_PATH is empty when pyarrow installer runs.
_Python3_NumPy_PATH = /usr/lib/python3/dist-packages/numpy/core/include when I run the toy example.

I'll try to put more message output and share here.
Do you have suggestions to something specific you wanna see?

[sdy72]

New finding. I removed the try: ... except: pass in
https://gitlab.kitware.com/cmake/cmake/-/blob/v3.25.1/Modules/FindPython/Support.cmake#L3172
and got this error:

      from numpy.core._multiarray_umath import (
  ImportError: libcblas.so.3: cannot open shared object file: No such file or directory

I then followed the recommendation in https://numpy.org/devdocs/user/troubleshooting-importerror.html
and installed sudo apt-get install libatlas-base-dev

Then find_package(Python3... ran with the success.
_Python3_NumPy_PATH = /tmp/pip-build-env-rlrroqfr/overlay/lib/python3.9/site-packages/numpy/core/include

Now I get the following error instead. Attached output.
pyarrow_install_make_error.txt

---

Btw, I also installed arrow from source apache-arrow-10.0.1/cpp
cmake -> make -> sudo make install
Prior to all this, otherwise I got this error

CMake Error at CMakeLists.txt:63 (find_package):
    By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
    asked CMake to find a package configuration file provided by "Arrow", but
    CMake did not find one.

Is this intended?

[kou]

Great!
Could you show the cmake command line used for Apache Arrow C++?

[sdy72]

cmake .
install_manifest.txt

[kou]

OK. We need to specify some options to enable Apache Arrow C++ components that are used by pyarrow.

Could you try the following?

cmake --preset=ninja-release-python-minimal .
cmake --build .
sudo cmake --build . --target install

See also: https://arrow.apache.org/docs/dev/developers/python.html#build-and-test

[sdy72]

Thanks.
I also got some hints about dependencies here:
https://gist.github.com/renjieliu/b86f14e648cac7ddf582cb6ceb164d32

I had to fetch xsimd source manually and set this prior to the cmake commands:

export xsimd_DIR=/home/pi/xsimd/xsimd

But I could not get rid of this error:

[1/233] Creating directories for 'jemalloc_ep'
FAILED: jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-mkdir /home/pi/arrow/apache-arrow-10.0.1/cpp/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-mkdir
cd /home/pi/arrow/apache-arrow-10.0.1/cpp && /usr/local/bin/cmake -Dcfgdir= -P /home/pi/arrow/apache-arrow-10.0.1/cpp/jemalloc_ep-prefix/tmp/jemalloc_ep-mkdirs.cmake && /usr/local/bin/cmake -E touch /home/pi/arrow/apache-arrow-10.0.1/cpp/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-mkdir
cmake -E touch: failed to update "/home/pi/arrow/apache-arrow-10.0.1/cpp/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-mkdir".
ninja: build stopped: subcommand failed.

[kou]

Could you share the output of ls -la /home/pi/arrow/apache-arrow-10.0.1/cpp/jemalloc_ep-prefix/src/jemalloc_ep-stamp/?

[sdy72]

Thanks. My bad. I wiped and re-unpacked arrow source code and the above cmake commands completed with success.
(I also went back to use cmake 3.18 installed with apt).

Now I get this error when I try to install pyarrow with pip:

  -- Build files have been written to: /tmp/pip-install-lmhsbdp3/pyarrow_0897d7658651493cb858488552932daa/build/cpp
  -- Finished CMake for PyArrow C++
  -- Running CMake build and install for PyArrow C++
  cmake --build . --config release --target install
  Scanning dependencies of target arrow_python_objlib
  [  4%] Building CXX object CMakeFiles/arrow_python_objlib.dir/arrow/python/arrow_to_pandas.cc.o
  /tmp/pip-install-lmhsbdp3/pyarrow_0897d7658651493cb858488552932daa/pyarrow/src/arrow/python/arrow_to_pandas.cc:49:10: fatal error: arrow/compute/api.h: No such file or directory
     49 | #include "arrow/compute/api.h"
        |          ^~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  gmake[2]: *** [CMakeFiles/arrow_python_objlib.dir/build.make:82: CMakeFiles/arrow_python_objlib.dir/arrow/python/arrow_to_pandas.cc.o] Error 1
  gmake[1]: *** [CMakeFiles/Makefile2:174: CMakeFiles/arrow_python_objlib.dir/all] Error 2
  gmake: *** [Makefile:149: all] Error 2
  error: command '/usr/bin/cmake' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

[kou]

Could you the show full output of pip install pyarrow?

[sdy72]

Sure
pyarrow-pip-output.txt

[kou]

Thanks.
Could you show find /usr/local/include/arrow/?

[sdy72]

find_arrow_include.txt

[kou]

Could you show the output of sudo cmake --build . --target install that is mentioned in #14920 (comment) ?

[sdy72]

cmake_build_output.txt

[kou]

Hmm. Did you really use cmake --preset=ninja-release-python-minimal .?
Could you also show the output of cmake --preset=ninja-release-python-minimal .

[sdy72]

I just verified it, yes. Output attached.

I just tried with this option:
-DARROW_COMPUTE=ON
but got out of memory error during build. I'll try adding more swap space

cmake_ninja_output.txt

[kou]

I got it.
Your CMake is 3.18.4 but cmake --preset is available since 3.19.

Could you use cmake -DARROW_BUILD_STATIC=OFF -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_FILESYSTEM=ON -DARROW_JSON=ON -DCMAKE_BUILD_TYPE=Release -GNinja . instead?

[sdy72]

That worked. It least after I also added dataset on:
cmake -DARROW_BUILD_STATIC=OFF -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_FILESYSTEM=ON -DARROW_JSON=ON -DARROW_DATASET=ON -DCMAKE_BUILD_TYPE=Release -GNinja .

I successfully installed pyarrow. Thank you :-)

Then when I try to import pyarrow I get this error:

>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: /usr/local/lib/libarrow.so.1000: undefined symbol: __atomic_fetch_add_8

Could that be related to this?
pytorch/pytorch#22898 (comment)

[westonpace]

Yes, arrow will need to be linked with -latomic. However, that should be happening here:

arrow/cpp/src/arrow/CMakeLists.txt

Lines 577 to 581 in a0d1630

	# Need -latomic on Raspbian.
	# See also: https://issues.apache.org/jira/browse/ARROW-12860
	if(${CMAKE_SYSTEM_NAME} STREQUAL "Linux" AND ${CMAKE_SYSTEM_PROCESSOR} MATCHES "armv7")
	string(APPEND ARROW_PC_LIBS_PRIVATE " -latomic")
	endif()

Is there any reason this check might not be working?

[sdy72]

Could it be this?
message(STATUS "#### CMAKE_SYSTEM_PROCESSOR = ${CMAKE_SYSTEM_PROCESSOR} ")
outputs:
#### CMAKE_SYSTEM_PROCESSOR = armv7l

[westonpace]

I think that's ok. MATCHES should still match true if the pattern is a substring in the variable. Perhaps the issue is that this only applies to static builds? @kou might know more here.

[kou]

Ah, we need to add -latomic to our CMake package because PyArrow uses CMake package not pkg-config package. Could you try the following patch?

diff --git a/cpp/src/arrow/CMakeLists.txt b/cpp/src/arrow/CMakeLists.txt
index 3bf5ebb9b9..4c784f2bb8 100644
--- a/cpp/src/arrow/CMakeLists.txt
+++ b/cpp/src/arrow/CMakeLists.txt
@@ -578,6 +578,8 @@ endif()
 # See also: https://issues.apache.org/jira/browse/ARROW-12860
 if(${CMAKE_SYSTEM_NAME} STREQUAL "Linux" AND ${CMAKE_SYSTEM_PROCESSOR} MATCHES "armv7")
   string(APPEND ARROW_PC_LIBS_PRIVATE " -latomic")
+  list(APPEND ARROW_SHARED_INSTALL_INTERFACE_LIBS "atomic")
+  list(APPEND ARROW_STATIC_INSTALL_INTERFACE_LIBS "atomic")
 endif()
 
 # If libarrow.a is only built, "pkg-config --cflags --libs arrow"

[sdy72]

It works now all the way. Thank you kou.

[kou]

Thanks for confirming it.
I've opened a pull request: #15251