"arrow/python/platform.h: No such file or directory" error

[nasica88]

I am trying to install pyarrow on Redhat 7.5ALT ppc64le with pip 10.0.1, but I end up with "arrow/python/platform.h: No such file or directory" error.

What did I do wrong ?

At first, I got "No package 'arrow' found" error, even though I did install arrow.

$ pip list | grep arrow
arrow 0.12.1

So I had to build arrow from source (from this github), and installed libarrow.so in /usr/local/lib64 as below and set the PKG_CONFIG_PATH accordingly.

$ find /usr -name libarrow.*
/usr/local/lib64/libarrow.so.10.0.0
/usr/local/lib64/libarrow.so.10
/usr/local/lib64/libarrow.so
/usr/local/lib64/libarrow.a

$ export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig

Still, I end up with "arrow/python/platform.h: No such file or directory" error.

$ pip --version
pip 10.0.1 from /opt/anaconda2/lib/python2.7/site-packages/pip (python 2.7)

$ pip install pyarrow
Collecting pyarrow
Using cached https://files.pythonhosted.org/packages/be/2d/11751c477e4e7f4bb07ac7584aafabe0d0608c170e4bff67246d695ebdbe/pyarrow-0.9.0.tar.gz
...
[ 66%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
/tmp/pip-install-kil31a/pyarrow/build/temp.linux-ppc64le-2.7/lib.cxx:592:35: fatal error: arrow/python/platform.h: No such file or directory
#include "arrow/python/platform.h"
^
compilation terminated.
make[2]: *** [CMakeFiles/lib.dir/lib.cxx.o] Error 1
make[1]: *** [CMakeFiles/lib.dir/all] Error 2
make: *** [all] Error 2
error: command 'make' failed with exit status 2

[xhochy]

$ pip list | grep arrow
arrow 0.12.1

This is a different Arrow, not the project here.

To debug this problems, it would be good to have answers to the following questions:

• Does /usr/local/include/arrow/python/platform.h exist?
• Does setting export ARROW_HOME=/usr/local help?
• I see in your path, that you have anaconda. What about using our conda recipes (https://github.com/conda-forge/arrow-cpp-feedstock, https://github.com/conda-forge/pyarrow-feedstock/, …)?

[nasica88]

@xhochy

1. No, platform.h does not exist, nor the /usr/local/include/arrow/python directory. I see /usr/local/include/arrow only.

2. No, it doesn't work.

3. It doesn't work, either, at least not for ppc64le.

$ conda install arrow-cpp
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

• arrow-cpp

Current channels:

• https://conda.anaconda.org/conda-forge/linux-ppc64le
• https://conda.anaconda.org/conda-forge/noarch
• https://repo.anaconda.com/pkgs/main/linux-ppc64le
• https://repo.anaconda.com/pkgs/main/noarch
• https://repo.anaconda.com/pkgs/free/linux-ppc64le
• https://repo.anaconda.com/pkgs/free/noarch
• https://repo.anaconda.com/pkgs/r/linux-ppc64le
• https://repo.anaconda.com/pkgs/r/noarch
• https://repo.anaconda.com/pkgs/pro/linux-ppc64le
• https://repo.anaconda.com/pkgs/pro/noarch

[wesm]

@nasica88 you are missing libarrow_python (which is built with -DARROW_PYTHON=ON) -- did you build and install the C++ libraries from source? See instructions in https://github.com/apache/arrow/blob/master/python/doc/source/development.rst

[wesm]

Note we have not packaged and tested Arrow on ppc64le because we don't have access to such a system; your help testing things out and reporting issues would be appreciated

[nasica88]

@wesm Thank you for the link anove. It helps. I succeeds in building pyarrow as below.

$ MAKEFLAGS=-j16 python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet --with-plasma --inplace
...
[100%] Built target lib
-- Finished cmake --build for pyarrow
Bundling includes: release/include
Moving generated C++ source lib.cpp to build path /home/data/arrow/python/pyarrow/lib.cpp
Moving built C-extension release/lib.cpython-36m-powerpc64le-linux-gnu.so to build path /home/data/arrow/python/pyarrow/lib.cpython-36m-powerpc64le-linux-gnu.so
Moving generated C++ source _parquet.cpp to build path /home/data/arrow/python/pyarrow/_parquet.cpp
Moving built C-extension release/_parquet.cpython-36m-powerpc64le-linux-gnu.so to build path /home/data/arrow/python/pyarrow/_parquet.cpython-36m-powerpc64le-linux-gnu.so
release/_orc.cpython-36m-powerpc64le-linux-gnu.so
Cython module _orc failure permitted
Moving generated C++ source _plasma.cpp to build path /home/data/arrow/python/pyarrow/_plasma.cpp
Moving built C-extension release/_plasma.cpython-36m-powerpc64le-linux-gnu.so to build path /home/data/arrow/python/pyarrow/_plasma.cpython-36m-powerpc64le-linux-gnu.so

---

However, when I tried to build whl file, it gives me a very weird error. I say weird because /home/data/dist/lib64/libarrow.so does exist !

$ ls -l /home/data/dist/lib64/libarrow.so
lrwxrwxrwx 1 root root 14 Jul 20 16:10 /home/data/dist/lib64/libarrow.so -> libarrow.so.10
$ ls -l /home/data/dist/lib64/libarrow.so.10
lrwxrwxrwx 1 root root 18 Jul 20 16:10 /home/data/dist/lib64/libarrow.so.10 -> libarrow.so.10.0.0
$ ls -l /home/data/dist/lib64/libarrow.so.10.0.0
-rwxr-xr-x 1 root root 4785528 Jul 20 16:10 /home/data/dist/lib64/libarrow.so.10.0.0

---

Below is the error I get when I try to build whl file. Can you pls advise ?

$ python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel
running build_ext
-- Runnning cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/root/anaconda3/bin/python -DPYARROW_BUILD_PARQUET=on -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUILD_PLASMA=on -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release /home/data/arrow/python
INFOCompiler command: /usr/bin/c++
INFOCompiler version: Using built-in specs.
COLLECT_GCC=/usr/bin/c++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/ppc64le-redhat-linux/4.8.5/lto-wrapper
Target: ppc64le-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-ppc64le-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-ppc64le-redhat-linux/cloog-install --enable-gnu-indirect-function --enable-secureplt --with-long-double-128 --enable-targets=powerpcle-linux --disable-multilib --with-cpu-64=power8 --with-tune-64=power8 --build=ppc64le-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)

INFOCompiler id: GNU
Selected compiler gcc 4.8.5
Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory: /home/data/arrow/python/build/temp.linux-ppc64le-3.6/release
-- Searching for Python libs in /root/anaconda3/lib64;/root/anaconda3/lib;/root/anaconda3/lib/python3.6/config-3.6m-powerpc64le-linux-gnu
-- Looking for python3.6m
-- Found Python lib /root/anaconda3/lib/libpython3.6m.so
-- Searching for Python libs in /root/anaconda3/lib64;/root/anaconda3/lib;/root/anaconda3/lib/python3.6/config-3.6m-powerpc64le-linux-gnu
-- Looking for python3.6m
-- Found Python lib /root/anaconda3/lib/libpython3.6m.so
-- Checking for module 'arrow'
-- No package 'arrow' found
-- Found the Arrow core library: /home/data/dist/lib64/libarrow.so
-- Found the Arrow Python library: /home/data/dist/lib64/libarrow_python.so
CMake Error: File /home/data/dist/lib64/libarrow.so. does not exist.
CMake Error at CMakeLists.txt:230 (configure_file):
configure_file Problem configuring file
Call Stack (most recent call first):
CMakeLists.txt:271 (bundle_arrow_lib)

[wesm]

Did you set $ARROW_HOME to /home/data/dist?

[nasica88]

@wesm Yes I did.

[wesm]

hi @nasica88 can you please open a JIRA with the details from this issue? I'm not sure how to proceed as I do not have access to a ppc64le machine to test. Perhaps you can also open a request with Anaconda to get their assistance with packaging pyarrow as part of the Anaconda distribution

[nasica88]

@wesm By the way, you can always access a ppc64le (IBM POWER8) machine with the linux OS of your choice for free. Visit the following site for IBM Power Development Platform. You get two weeks of a small ppc64le VM for free, and store it after two weeks, and request another two weeks for free perpetually. They do not give you a GPU, though.

https://www-356.ibm.com/partnerworld/wps/servlet/ContentHandler/stg_com_sys_power-development-platform

[nasica88]

@wesm I could work around that "/home/data/dist/lib64/libarrow.so. does not exist" error with the following.

$ ln -s /usr/local/lib64/libarrow.so.10.0.0 /usr/local/lib64/libarrow.so.
$ ln -s /usr/local/lib64/libarrow_python.so.10.0.0 /usr/local/lib64/libarrow_python.so.
$ ln -s /usr/local/lib64/libplasma.so.10.0.0 /usr/local/lib64/libplasma.so.

In short, CMakeLists.txt tries to concatenate ARG_ABI_VERSION behind libarrow.so so that the file name should be something like libarrow.so.10.0.0. It simply didn't work. I do not know if this is due to a bug or some misconfigured environment. See the following line # 230 in CMakeLists.txt.

configure_file(${${library_path}}.${ARG_ABI_VERSION}

Any way, I just made a soft link with the name of /usr/local/lib64/libarrow.so., with the dot (.) in the end. That solved it.

...
creating build/bdist.linux-ppc64le/wheel/pyarrow/tensorflow
copying build/lib.linux-ppc64le-3.6/pyarrow/tensorflow/build.sh -> build/bdist.linux-ppc64le/wheel/pyarrow/tensorflow
copying build/lib.linux-ppc64le-3.6/pyarrow/tensorflow/plasma_op.cc -> build/bdist.linux-ppc64le/wheel/pyarrow/tensorflow
running install_egg_info
Copying pyarrow.egg-info to build/bdist.linux-ppc64le/wheel/pyarrow-0.9.1.dev386+g7d2fbeb.d20180723-py3.6.egg-info
running install_scripts
creating build/bdist.linux-ppc64le/wheel/pyarrow-0.9.1.dev386+g7d2fbeb.d20180723.dist-info/WHEEL

$ ls -l ./dist/pyarrow-0.9.1.dev386+g7d2fbeb.d20180723-cp36-cp36m-linux_ppc64le.whl
-rw-r--r-- 1 root root 7753353 Jul 23 11:41 ./dist/pyarrow-0.9.1.dev386+g7d2fbeb.d20180723-cp36-cp36m-linux_ppc64le.whl

[wesm]

That seems buggy. We should try to diagnose the underlying cause so it can be fixed

[wesm]

@nasica88 can you open a JIRA issue summarizing your findings on this? We need to track development work / bugs there and not here

[wesm]

Closing as stale