Missing package: pyarrow

[josuuribe]

Package name: pyarrow
Issue type: Build failed
Link to PyPI page: https://pypi.org/project/pyarrow
Link to piwheels page: https://www.piwheels.org/project/pyarrow/
Version: All
Python version: 3.5+
I am the maintainer: No
More information:

Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. This library is used by vaex-core that also fails

Detailed instructions about the installation can be found here:
https://arrow.apache.org/install/

Additional help
https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521

I suppose the main reason is the need for Apache arrow libraries

[bennuttall]

This has been raised before. We closed it as it didn't seem feasible to add to our automated build.

Can you follow the instructions and build it successfully on a Pi?

[josuuribe]

Not yet, I expected it would be more easy in a specialized builder machine like yours, but I have read several people has got it. The problem is this library is used by several other ones, especially those related to deal with big data.
I have the idea to create a specialized Docker container if i get how to build it, as other open source projects like PyTorch o Tensorflow does.

[josuuribe]

FROM debian:latest

ARG DEBIAN_FRONTEND=noninteractive
ARG REPO_HOME=/repos
ARG ARROW_HOME=$REPO_HOME/dist
ARG LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
ARG PYARROW_WITH_PARQUET=1
ARG PARQUET_TEST_DATA=$REPO_HOME/arrow/cpp/submodules/parquet-testing/data
ARG ARROW_TEST_DATA=$REPO_HOME/arrow/testing/data
ARG ARROW_BUILD_TYPE=release
ARG ARROW_TAG=apache-arrow-3.0.0

RUN apt-get update -y && apt-get install -y libjemalloc-dev libboost-dev
libboost-filesystem-dev
libboost-system-dev
libboost-regex-dev
make
build-essential
g++
libgflags-dev
rapidjson-dev
libre2-dev
python3-dev
libatlas-base-dev
python3-dev
autoconf
flex
bison
libgrpc-dev
git &&
rm -rf /var/lib/apt/lists/* &&
rm -rf /tmp/*

ADD https://bootstrap.pypa.io/get-pip.py get-pip.py
RUN python3 get-pip.py
RUN python3 -m pip config --global set global.extra-index-url https://www.piwheels.org/simple
RUN python3 -m pip install --upgrade
cmake
wheel
numpy

WORKDIR $REPO_HOME
RUN git clone https://github.com/apache/arrow.git
WORKDIR $REPO_HOME/arrow
RUN git checkout tags/$ARROW_TAG -b build
RUN git submodule init
RUN git submodule update

WORKDIR $REPO_HOME
RUN python3 -m pip install -r arrow/python/requirements-build.txt -r arrow/python/requirements-test.txt

WORKDIR $REPO_HOME/arrow/cpp/build
RUN cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
-DPYTHON3_EXECUTABLE=$(which python3) \
-DPYTHON_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc;print(get_python_inc())") \
-DCMAKE_INSTALL_LIBDIR=lib \
-DPYTHON_INCLUDE_DIR2=$(python3 -c "from os.path import dirname; from distutils.sysconfig import get_config_h_filename; print(dirname(get_config_h_filename()))") \
-DARROW_WITH_BZ2=ON \
-DPYTHON_LIBRARY=$(python3 -c "from distutils.sysconfig import get_config_var;from os.path import dirname,join ; print(join(dirname(get_config_var('LIBPC')),get_config_var('LDLIBRARY')))") \
-DARROW_WITH_ZLIB=ON \
-DPYTHON3_NUMPY_INCLUDE_DIRS=$(python3 -c "import numpy; print(numpy.get_include())") \
-DARROW_WITH_ZSTD=ON
-DPYTHON3_PACKAGES_PATH=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") \
-DARROW_WITH_LZ4=ON
-DARROW_WITH_SNAPPY=ON
-DARROW_WITH_BROTLI=ON
-DARROW_PARQUET=ON
-DARROW_PYTHON=ON
-DARROW_BUILD_TESTS=ON
..
RUN make -j$(nproc)
RUN make install

WORKDIR $REPO_HOME/arrow/python
RUN python3 setup.py build_ext --inplace
RUN python3 -m pytest pyarrow 2>&1 || echo "Some unit tests have failed"
RUN python3 setup.py build_ext --build-type=$ARROW_BUILD_TYPE --bundle-arrow-cpp bdist_wheel

WORKDIR /drop
RUN cp $REPO_HOME/arrow/python/dist/*.whl .

CMD ["/bin/bash"]

[josuuribe]

Execute with:
docker run -dit image_id

Copy wheel from docker image
docker cp container_id:/drop .

Now, you can stop container
docker container stop container_id

It works for Apache 4.0.0 (master) and also for latest stable version (3.0.0) anyway you can switch versions using ARROW_TAG while build (set as value the same label as exists in Arrow GitHub repository)

Original here: https://github.com/josuuribe/RaraAvis/blob/blog/Docker/build/Dockerfile.arrow

I hope this helps!!

Thanks for your effort with pywheels!