ARROW-3567: [Gandiva][GLib] Add GLib bindings of Gandiva by shiro615 · Pull Request #2800 · apache/arrow

shiro615 · 2018-10-19T16:49:10Z

It's GLib bindings of Gandiva.

shiro615 · 2018-10-19T16:53:38Z

cpp/src/gandiva/CMakeLists.txt

Because it is possible to include <gandiva/*.h> in gandiva-glib.
But I’m not sure whether this change is correct.

Looks right to me

It's correct.

Thank you for checking.

kou

Thanks!
It's almost good! Can you check my comments?

kou · 2018-10-20T23:23:19Z

c_glib/doc/gandiva-glib/gandiva-glib-docs.xml

evaluate-expression may be better. Because this part covers not only building expression but also evaluating the built expression.

I see. I've changed it.

kou · 2018-10-20T23:23:54Z

c_glib/doc/gandiva-glib/gandiva-glib-docs.xml

Evaluate expression may be better.

I've changed it.

kou · 2018-10-20T23:26:47Z

c_glib/doc/gandiva-glib/gandiva-glib-docs.xml

expression may be better.

I've changed it.

kou · 2018-10-20T23:27:03Z

c_glib/doc/gandiva-glib/gandiva-glib-docs.xml

We can start from 0.12.0.

I've fixed it.

kou · 2018-10-20T23:45:26Z

c_glib/gandiva-glib/gandiva-glib.pc.in

gandiva is also required.

I've added gandiva.

kou · 2018-10-21T00:06:45Z

c_glib/test/gandiva/test-projector.rb

expected_output_arrays or expected_outputs is better.

kou · 2018-10-21T00:07:29Z

c_glib/test/gandiva/test-projector.rb

We can put output_columns here as literal to clear what we expect for the evaluate output.

I've put the following.

assert_equal([ [12, 15, 18, 21], [-10, -11, -12, -13] ], outputs.collect(&:values))

kou · 2018-10-21T00:08:07Z

c_glib/test/run-test.rb

It's better that we put this before Parquet = GI.load("Parquet") because we use the order in other places.

I've fixed it.

kou · 2018-10-21T00:08:40Z

c_glib/test/run-test.sh

arrow-glib arrow-gpu-glib gandiva-glib parquet-glib order is better.

I've fixed it.

kou · 2018-10-21T00:08:55Z

cpp/src/gandiva/CMakeLists.txt

It's correct.

shiro615 · 2018-10-21T06:16:03Z

Thank you for review.
Please review again.

kou

+1
I'll merge this when we fix CI failure:
https://travis-ci.org/apache/arrow/jobs/444283175#L896-L897

CMake Error at src/gandiva/CMakeLists.txt:19 (cmake_minimum_required):
  CMake 3.11 or higher is required.  You are running version 3.9.2

I don't know why C++ job isn't failed but C GLib job is failed...

wesm · 2018-10-22T01:59:43Z

The C++ build uses a new CMake from conda-forge

kou · 2018-10-22T13:58:47Z

@wesm Thanks for the information.

@shiro615 I don't want to use conda for GLib CI to test without conda. We don't use conda when we build Linux packages,

shiro615 · 2018-10-22T16:06:20Z

I see. Thank you for the update.

kou · 2018-10-23T05:10:10Z

It seems that we need Boost newer than 1.54.0 for Gandiva.

praveenbingo · 2018-10-23T07:19:50Z

It seems that we need Boost newer than 1.54.0 for Gandiva.

Yup, we have tested with 1.66 or higher version of boost.

pravindra · 2018-10-23T07:21:39Z

It seems that we need Boost newer than 1.54.0 for Gandiva.

yes, for the cpp build, we get a newer boost version from conda.

kou · 2018-10-23T08:56:26Z

Thanks for the information.
I'm trying to use vendored Boost.

kou · 2018-10-24T01:59:32Z

It seems that we can't build Gandiva with vendrored Boost.
I've created a JIRA ticket: https://issues.apache.org/jira/browse/ARROW-3603

kou · 2018-10-24T09:24:07Z

I've created a pull request: #2827

kou · 2018-10-25T00:07:07Z

#2827 has been merged.
I'll rebase this branch on master.

kou · 2018-10-25T00:45:12Z

#2828 is also needed to build with vendored Boost.

kou · 2018-10-25T00:51:29Z

I've merged #2828.
I'll rebase on master again.

kou · 2018-10-25T01:33:02Z

Umm... We need more works to build Gandiva.

https://travis-ci.org/apache/arrow/jobs/445955279#L1732

[ 87%] Linking CXX executable ../../../debug/filter_test_gandiva_static
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86FloatingPoint.cpp.o): In function `(anonymous namespace)::FPS::runOnMachineFunction(llvm::MachineFunction&) [clone .part.165]':
(.text._ZN12_GLOBAL__N_13FPS20runOnMachineFunctionERN4llvm15MachineFunctionE.part.165+0x481): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86InstructionSelector.cpp.o): In function `llvm::PredicateBitsetImpl<107ul>::PredicateBitsetImpl(std::initializer_list<unsigned int>)':
(.text._ZN4llvm19PredicateBitsetImplILm107EEC2ESt16initializer_listIjE[_ZN4llvm19PredicateBitsetImplILm107EEC5ESt16initializer_listIjE]+0x75): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x452): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x46a): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o): In function `getOpEnabled(bool, llvm::EVT, llvm::StringRef)':
(.text._ZL12getOpEnabledbN4llvm3EVTENS_9StringRefE+0x3f3): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o):(.text._ZL20getOpRefinementStepsbN4llvm3EVTENS_9StringRefE+0x367): more undefined references to `std::__throw_out_of_range_fmt(char const*, ...)' follow
collect2: error: ld returned 1 exit status
make[2]: *** [debug/if_expr_test_gandiva_static] Error 1
make[1]: *** [src/gandiva/tests/CMakeFiles/if_expr_test_gandiva_static.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86FloatingPoint.cpp.o): In function `(anonymous namespace)::FPS::runOnMachineFunction(llvm::MachineFunction&) [clone .part.165]':
(.text._ZN12_GLOBAL__N_13FPS20runOnMachineFunctionERN4llvm15MachineFunctionE.part.165+0x481): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86InstructionSelector.cpp.o): In function `llvm::PredicateBitsetImpl<107ul>::PredicateBitsetImpl(std::initializer_list<unsigned int>)':
(.text._ZN4llvm19PredicateBitsetImplILm107EEC2ESt16initializer_listIjE[_ZN4llvm19PredicateBitsetImplILm107EEC5ESt16initializer_listIjE]+0x75): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x452): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x46a): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o): In function `getOpEnabled(bool, llvm::EVT, llvm::StringRef)':
(.text._ZL12getOpEnabledbN4llvm3EVTENS_9StringRefE+0x3f3): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o):(.text._ZL20getOpRefinementStepsbN4llvm3EVTENS_9StringRefE+0x367): more undefined references to `std::__throw_out_of_range_fmt(char const*, ...)' follow
collect2: error: ld returned 1 exit status
make[2]: *** [debug/binary_test_gandiva_static] Error 1
make[1]: *** [src/gandiva/tests/CMakeFiles/binary_test_gandiva_static.dir/all] Error 2
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86FloatingPoint.cpp.o): In function `(anonymous namespace)::FPS::runOnMachineFunction(llvm::MachineFunction&) [clone .part.165]':
(.text._ZN12_GLOBAL__N_13FPS20runOnMachineFunctionERN4llvm15MachineFunctionE.part.165+0x481): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86InstructionSelector.cpp.o): In function `llvm::PredicateBitsetImpl<107ul>::PredicateBitsetImpl(std::initializer_list<unsigned int>)':
(.text._ZN4llvm19PredicateBitsetImplILm107EEC2ESt16initializer_listIjE[_ZN4llvm19PredicateBitsetImplILm107EEC5ESt16initializer_listIjE]+0x75): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x452): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x46a): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o): In function `getOpEnabled(bool, llvm::EVT, llvm::StringRef)':
(.text._ZL12getOpEnabledbN4llvm3EVTENS_9StringRefE+0x3f3): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o):(.text._ZL20getOpRefinementStepsbN4llvm3EVTENS_9StringRefE+0x367): more undefined references to `std::__throw_out_of_range_fmt(char const*, ...)' follow
collect2: error: ld returned 1 exit status
make[2]: *** [debug/hash_test_gandiva_static] Error 1
make[1]: *** [src/gandiva/tests/CMakeFiles/hash_test_gandiva_static.dir/all] Error 2
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86FloatingPoint.cpp.o): In function `(anonymous namespace)::FPS::runOnMachineFunction(llvm::MachineFunction&) [clone .part.165]':
(.text._ZN12_GLOBAL__N_13FPS20runOnMachineFunctionERN4llvm15MachineFunctionE.part.165+0x481): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMX86CodeGen.a(X86InstructionSelector.cpp.o): In function `llvm::PredicateBitsetImpl<107ul>::PredicateBitsetImpl(std::initializer_list<unsigned int>)':
(.text._ZN4llvm19PredicateBitsetImplILm107EEC2ESt16initializer_listIjE[_ZN4llvm19PredicateBitsetImplILm107EEC5ESt16initializer_listIjE]+0x75): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x452): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMAsmPrinter.a(CodeViewDebug.cpp.o): In function `llvm::CodeViewDebug::getFullFilepath(llvm::DIFile const*)':
(.text._ZN4llvm13CodeViewDebug15getFullFilepathEPKNS_6DIFileE+0x46a): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o): In function `getOpEnabled(bool, llvm::EVT, llvm::StringRef)':
(.text._ZL12getOpEnabledbN4llvm3EVTENS_9StringRefE+0x3f3): undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/usr/lib/llvm-6.0/lib/libLLVMCodeGen.a(TargetLoweringBase.cpp.o):(.text._ZL20getOpRefinementStepsbN4llvm3EVTENS_9StringRefE+0x367): more undefined references to `std::__throw_out_of_range_fmt(char const*, ...)' follow
collect2: error: ld returned 1 exit status
make[2]: *** [debug/filter_test_gandiva_static] Error 1
make[1]: *** [src/gandiva/tests/CMakeFiles/filter_test_gandiva_static.dir/all] Error 2
make: *** [all] Error 2

pravindra · 2018-10-25T03:36:09Z

.travis.yml

+    - ARROW_TRAVIS_USE_VENDORED_BOOST=1
    - ARROW_TRAVIS_PARQUET=1
    - BUILD_TORCH_EXAMPLE=no
    - CC="gcc-4.9"


for some reason, this doesn't work. Can you instead try doing this similar to the C++ gandiva entry ?

MATRIX_EVAL="CC=gcc-4.9 && CXX=g++-4.9"

pravindra · 2018-10-25T03:37:19Z

@kou - your build still used the gcc-4.8. I suggested a workaround to use 4.9 - can you please try that ?

kou · 2018-10-25T05:02:49Z

@pravindra Thanks for the information! I've added the configuration.

kou · 2018-10-25T23:45:47Z

CI failure for R job is unrelated.
I'll merge this.

kou · 2018-10-25T23:48:02Z

Merged.
Thanks all!

shiro615 · 2018-10-26T05:17:58Z

Thanks!

wesm · 2018-10-26T10:06:35Z

Thanks for doing this! @pcmoritz I guess it's time to rehabilitate the Gandiva Cython bindings to keep up with GLib =)

shiro615 commented Oct 19, 2018

View reviewed changes

shiro615 force-pushed the glib-support-gandiva branch from 14d89ba to f5da192 Compare October 20, 2018 04:32

kou requested changes Oct 21, 2018

View reviewed changes

shiro615 force-pushed the glib-support-gandiva branch 2 times, most recently from aebdf5e to e9cd59a Compare October 21, 2018 06:33

kou approved these changes Oct 22, 2018

View reviewed changes

shiro615 force-pushed the glib-support-gandiva branch from 3bf433a to 81fea3d Compare October 22, 2018 13:36

kou force-pushed the glib-support-gandiva branch 2 times, most recently from db3449d to 3de18f1 Compare October 25, 2018 00:37

shiro615 added 8 commits October 25, 2018 09:52

[GLib] Add GGandivaTreeExpressionBuilder

02803a4

[GLib] Add build system for Gandiva

ad492e4

[GLib] Add GGandivaExpression

a851492

[GLib] Add GGandivaProjector

aa32554

[GLib] Add test setting for Gandiva

f73de09

[GLib] Use TreeExprBuilder::MakeExpression in ggandiva_expression_new()

aff5a94

[GLib] Delete GGandivaTreeExpressionBuilder

2562255

[GLib] Add ggandiva_expression_to_string()

200b7b6

kou and others added 17 commits October 25, 2018 09:52

Remove redundant dependency

d4c5bdb

Use the same order in meson.build

757eacd

Fix indent

fd70a4c

Use auto

761ad7f

Align

294925e

Update cmake version in Travis CI

5d9a728

Add travis_install_linux.sh in Travis CI

11ab625

Stop to use conda

d36fb3e

Debug

d7d04c5

Remove needless clang install

00798e0

Remove install cmake

e18d3d3

Fix a typo

d23eee3

CMake isn't installed by apt

8681e30

Clang 6.0.0 is needed for Gandiva

ccd17f6

Use vendored Boost because system Boost is old for Gandiva

5307395

Remove installed CMake again

5c629ea

Set environment variable in .travis.yml

66b2bd1

kou force-pushed the glib-support-gandiva branch from 3de18f1 to 66b2bd1 Compare October 25, 2018 00:52

pravindra reviewed Oct 25, 2018

View reviewed changes

Ensure using GCC 4.9

f4d3e1c

kou closed this in e5122e3 Oct 25, 2018

shiro615 deleted the glib-support-gandiva branch October 26, 2018 05:18

asfimport mentioned this pull request Oct 26, 2018

[Gandiva] [GLib] Add GLib bindings of Gandiva #15841

Closed

Conversation

shiro615 commented Oct 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shiro615 commented Oct 21, 2018

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

wesm commented Oct 22, 2018

Uh oh!

kou commented Oct 22, 2018

Uh oh!

shiro615 commented Oct 22, 2018

Uh oh!

kou commented Oct 23, 2018

Uh oh!

praveenbingo commented Oct 23, 2018

Uh oh!

pravindra commented Oct 23, 2018

Uh oh!

kou commented Oct 23, 2018

Uh oh!

kou commented Oct 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Oct 24, 2018

Uh oh!

kou commented Oct 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Oct 25, 2018

Uh oh!

kou commented Oct 25, 2018

kou commented Oct 24, 2018 •

edited

Loading

kou commented Oct 25, 2018 •

edited

Loading