Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Initial split for libarrow components #1175

Closed
wants to merge 28 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c574b01
Initial split for libarrow, libarrow-acero, libarrow-dataset, libarro…
raulcd Sep 15, 2023
a6f40bd
Remove unnecessary build requirements
raulcd Sep 15, 2023
0b5043f
Arrow compute is required for default compute kernels to be available…
raulcd Sep 18, 2023
544d2fb
Add protobuf and grpc to flight and refactor package flags on build
raulcd Sep 18, 2023
13436e2
Add protobuf to substrait
raulcd Sep 18, 2023
fc6eb6a
Set Enable Delayed Expansion on Windows build arrow bat
raulcd Sep 19, 2023
17328d4
Only check presence of libarrow_flight_transport_ucx.so on Linux
raulcd Sep 19, 2023
a1e9138
Unconditionally fix protoc-gen-grpc path instead of only for flight
raulcd Sep 19, 2023
a031627
Modify ArrowOptions to be turned ON when building pyarrow
raulcd Sep 19, 2023
30b0915
Remove grpc and protobuf on some packages
raulcd Sep 19, 2023
80caa5b
Use Windows syntax
raulcd Sep 19, 2023
a0b1e14
Initial move to build libarrow-all and then split into components
raulcd Sep 20, 2023
2967eed
Add missing protobuf and libgrpc
raulcd Sep 20, 2023
ae94de2
Windows fixes
raulcd Sep 21, 2023
3883471
Attempt to move subfolders too
raulcd Sep 25, 2023
1c5f86c
Use xcopy instead of move to move headers and folder structure
raulcd Sep 26, 2023
3d64cd1
Add missing parquet headers on Windows
raulcd Sep 26, 2023
dafe6b6
Add Gandiva headers to libarrow-gandiva
raulcd Sep 26, 2023
688128f
Remove not required host dependencies
raulcd Sep 28, 2023
14eda57
Add missing host dependencies, remove unnecessary and host some subpa…
raulcd Sep 28, 2023
d2f91ef
Build globally and move libarrow-all to be a metapackage
raulcd Sep 29, 2023
0ce4125
Update missing dependency from rebasing main
raulcd Oct 2, 2023
6136761
Move parquet out of libarrow
raulcd Oct 2, 2023
5d15e23
Remove some unnecessary dependencies and add parquet where required
raulcd Oct 2, 2023
28edaea
Arrow cuda should not be present on gandiva now
raulcd Oct 3, 2023
285fb46
Add libarrow requirement to parquet and gandiva
raulcd Oct 3, 2023
24e2404
Remove unnecessary zstd and cuda from gandiva
raulcd Oct 3, 2023
aeb8ceb
Add run_exports to libarrow-all
raulcd Oct 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions recipe/build-arrow.bat → recipe/bld.bat
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,7 @@ cmake -G "Ninja" ^
..
if %ERRORLEVEL% neq 0 exit 1

cmake --build . --target install --config Release
cmake --build . --config Release
if %ERRORLEVEL% neq 0 exit 1

popd

:: clean up between builds (and to save space)
rmdir /s /q cpp\build
8 changes: 3 additions & 5 deletions recipe/build-arrow.sh → recipe/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ do
cp "${RECIPE_DIR}/${CHANGE}.sh" "${PREFIX}/etc/conda/${CHANGE}.d/${PKG_NAME}_${CHANGE}.sh"
done

mkdir cpp/build
mkdir -p cpp/build
pushd cpp/build

EXTRA_CMAKE_ARGS=""
Expand Down Expand Up @@ -116,9 +116,7 @@ cmake -GNinja \
${EXTRA_CMAKE_ARGS} \
..

cmake --build . --target install --config Release
# Do not install arrow, only build.
cmake --build . --config Release

popd

# clean up between builds (and to save space)
rm -rf cpp/build
66 changes: 66 additions & 0 deletions recipe/install-libarrow.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
@echo on

:: Create temporary prefix to be able to install files more granularly
mkdir temp_prefix

cmake --install .\cpp\build --prefix=.\temp_prefix

if [%PKG_NAME%] == [libarrow] (
move .\temp_prefix\lib\arrow.lib %LIBRARY_LIB%
move .\temp_prefix\bin\arrow.dll %LIBRARY_BIN%
move .\temp_prefix\lib\arrow_cuda.lib %LIBRARY_LIB% || true
move .\temp_prefix\bin\arrow_cuda.dll %LIBRARY_BIN% || true
move .\temp_prefix\lib\pkgconfig\* %LIBRARY_LIB%\pkgconfig
mkdir %LIBRARY_LIB%\cmake\Arrow
move .\temp_prefix\lib\cmake\Arrow\* %LIBRARY_LIB%\cmake\Arrow
mkdir %LIBRARY_LIB%\cmake\ArrowCUDA
move .\temp_prefix\lib\cmake\ArrowCUDA\* %LIBRARY_LIB%\cmake\ArrowCUDA || true
mkdir %LIBRARY_PREFIX%\share\doc\arrow
move .\temp_prefix\share\doc\arrow\* %LIBRARY_PREFIX%\share\doc\arrow
mkdir %LIBRARY_PREFIX%\share\arrow
xcopy /s /y .\temp_prefix\share\arrow %LIBRARY_PREFIX%\share\arrow
mkdir %LIBRARY_PREFIX%\include\arrow
xcopy /s /y .\temp_prefix\include\arrow %LIBRARY_PREFIX%\include\arrow
) else if [%PKG_NAME%] == [libparquet] (
move .\temp_prefix\lib\parquet.lib %LIBRARY_LIB%
move .\temp_prefix\bin\parquet.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\Parquet
move .\temp_prefix\lib\cmake\Parquet\* %LIBRARY_LIB%\cmake\Parquet
mkdir %LIBRARY_PREFIX%\include\parquet
xcopy /s /y .\temp_prefix\include\parquet %LIBRARY_PREFIX%\include\parquet
) else if [%PKG_NAME%] == [libarrow-acero] (
move .\temp_prefix\lib\arrow_acero.lib %LIBRARY_LIB%
move .\temp_prefix\bin\arrow_acero.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\ArrowAcero
move .\temp_prefix\lib\cmake\ArrowAcero\* %LIBRARY_LIB%\cmake\ArrowAcero
) else if [%PKG_NAME%] == [libarrow-dataset] (
move .\temp_prefix\lib\arrow_dataset.lib %LIBRARY_LIB%
move .\temp_prefix\bin\arrow_dataset.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\ArrowDataset
move .\temp_prefix\lib\cmake\ArrowDataset\* %LIBRARY_LIB%\cmake\ArrowDataset
) else if [%PKG_NAME%] == [libarrow-gandiva] (
move .\temp_prefix\lib\gandiva.lib %LIBRARY_LIB%
move .\temp_prefix\bin\gandiva.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\Gandiva
move .\temp_prefix\lib\cmake\Gandiva\* %LIBRARY_LIB%\cmake\Gandiva
mkdir %LIBRARY_PREFIX%\include\gandiva
xcopy /s /y .\temp_prefix\include\gandiva %LIBRARY_PREFIX%\include\gandiva
) else if [%PKG_NAME%] == [libarrow-substrait] (
move .\temp_prefix\lib\arrow_substrait.lib %LIBRARY_LIB%
move .\temp_prefix\bin\arrow_substrait.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\ArrowSubstrait
move .\temp_prefix\lib\cmake\ArrowSubstrait\* %LIBRARY_LIB%\cmake\ArrowSubstrait
) else if [%PKG_NAME%] == [libarrow-flight] (
move .\temp_prefix\lib\arrow_flight.lib %LIBRARY_LIB%
move .\temp_prefix\bin\arrow_flight.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\ArrowFlight
move .\temp_prefix\lib\cmake\ArrowFlight\* %LIBRARY_LIB%\cmake\ArrowFlight
) else if [%PKG_NAME%] == [libarrow-flight-sql] (
move .\temp_prefix\lib\arrow_flight_sql.lib %LIBRARY_LIB%
move .\temp_prefix\bin\arrow_flight_sql.dll %LIBRARY_BIN%
mkdir %LIBRARY_LIB%\cmake\ArrowFlightSql
move .\temp_prefix\lib\cmake\ArrowFlightSql\* %LIBRARY_LIB%\cmake\ArrowFlightSql
)

:: clean up temp_prefix between builds
rmdir /s /q temp_prefix
48 changes: 48 additions & 0 deletions recipe/install-libarrow.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/bash
set -ex

# temporary prefix to be able to install files more granularly
mkdir temp_prefix

cmake --install ./cpp/build --prefix=./temp_prefix

if [[ "${PKG_NAME}" == libarrow ]]; then
# only libarrow
mv ./temp_prefix/lib/libarrow.* $PREFIX/lib
mv ./temp_prefix/lib/libarrow_cuda.* $PREFIX/lib || true
mv ./temp_prefix/lib/cmake/* $PREFIX/lib/cmake
mv ./temp_prefix/share/arrow/* $PREFIX/share/arrow
mv ./temp_prefix/share/gdb/* $PREFIX/share/gdb
mv ./temp_prefix/share/doc/* $PREFIX/share/doc
mv ./temp_prefix/include/arrow $PREFIX/include/
elif [[ "${PKG_NAME}" == libparquet ]]; then
# only parquet
mv ./temp_prefix/lib/libparquet.* $PREFIX/lib
mv ./temp_prefix/include/parquet $PREFIX/include/
elif [[ "${PKG_NAME}" == libarrow-acero ]]; then
# only libarrow-acero
mv ./temp_prefix/lib/libarrow_acero.* $PREFIX/lib
elif [[ "${PKG_NAME}" == libarrow-dataset ]]; then
# only libarrow-dataset
mv ./temp_prefix/lib/libarrow_dataset.* $PREFIX/lib
elif [[ "${PKG_NAME}" == libarrow-gandiva ]]; then
# only libarrow-gandiva
mv ./temp_prefix/lib/libgandiva.* $PREFIX/lib
mv ./temp_prefix/include/gandiva $PREFIX/include/
elif [[ "${PKG_NAME}" == libarrow-substrait ]]; then
# only libarrow-substrait
mv ./temp_prefix/lib/libarrow_substrait.* $PREFIX/lib
elif [[ "${PKG_NAME}" == libarrow-flight ]]; then
# only libarrow-flight
mv ./temp_prefix/lib/libarrow_flight.* $PREFIX/lib
mv ./temp_prefix/lib/libarrow_flight_transport_ucx.* $PREFIX/lib || true
elif [[ "${PKG_NAME}" == libarrow-flight-sql ]]; then
# only libarrow-flight-sql
mv ./temp_prefix/lib/libarrow_flight_sql.* $PREFIX/lib
else
# libarrow-all: install everything
cmake --install ./cpp/build --prefix=$PREFIX
fi

# Clean up temp_prefix
rm -rf temp_prefix
Loading