ARROW-3182: [C++][Java] Gandiva merge into Arrow #2558

Bootstrap evaluation using llvm code generation LLVM code generation is done using a mix of : - glue IR code that loops over the vector, generates function calls - byte-code files generated from simple c++ functions using clang . The glue-code and pre-compiled byte code are merged and optimized together. Expressions are specified using a "tree builder" where each node is an arrow vector, or a binary/unary function. During code generation, the expressions are "decomposed" so that the value array and bitmap array are evaluated separately to compute the expression result. This avoids the use of too many branch/conditional instructions , and hence, can be vectorized efficiently. Support added for arithmetic and logical expressions on numeric types. Travis CI support added for build on ubuntu. Change-Id: I06db1dfd398750755c76e0a395ba52bd1a01e329

Separate out the public and private target dependencies. For arrow, export an interface target. This avoids the need to add include dirs for each dependency on arrow. Removed dependency on gtest. Instead, build it as an external project. This is the recommended practice for googletest. For pre-compiled files, generate the bitcode files for each of them independently and then, link them to generate a unified bitcode file. Removed cpplint exceptions since there is no more sourcing of .cc files. Separate out the public include files from private includes, and add them in the dependency list in cmake. pass the bytecode filepath from cmake

[Gandiva] Introduce error codes as error handling strategy. Introduced status codes and using the same as the error handling strategy. The decision was taken to accommodate existing libraries that use error codes and because Arrow also uses error codes and not exception. Changed the signatures across the board for the same.

The pre-compiled functions takes an extra arg to set the result validity. At decompose time, a local bitmap is assigned to track the result validity bits for such functions. At evaluate time, sufficient number of local bitmaps are allocated for all the local bitmaps. For the final computation of the expression validity, the input bitmaps can be either one of the value-vector bitmaps, or a local bitmap.

Replaced MakeUnaryFunction, MakeBinaryFunction with a simpler MakeFunction that takes a vector of args.

An if-else expression has three sub-expressions : - condition - then-expression - else-expression Each of these can again be a node in the expression tree. The result validity of the if-else expression is saved in a local bitmap. Also, moved all of the integ tests to a different folder so that there is no mix of include files.

- moved the expression decomposition logic from Node class to a visitor class - moved node.h out of external includes - renamed Evaluator to Projector

Added support for literals .

In case of nested if-else conditions, eg. if A else if B else if C else D The else parts of A & C will not update validity bitmaps. Only the if parts and the terminal else update bitmaps.

[Gandiva] Add proto file to serialize the expression tree Fix the build also to generate the java and cpp proto files [Gandiva] Added proto files and added code to serialize an expression and other data types * Addressed review comments * Addressed code review comments * Made some classes package private Change-Id: Id7d3059782b00ffc14c20691764019a24ad143a8

Split gandiva into two sub modules : codegen & jni. - codegen is the core having cpp APIs and LLVM - jni deals with protobufs & interfacing with java

Dremio allocates the output vectors in java and passes the pointers to gandiva. In that case, gandiva will use the passed in buffers. Made Evaluate use ArrayData internally for output buffers, since Array is expected to be immutable.

Also, added "Adapted from XX" comments in ci/travis

- Added definitions for other integer types - Added definitions for unsigned types - Added a test for arithmetic ops on all int types - The functions should be inlined in the pre-compiled library, but not in the unit tests. Added a compiler flag to control this.

[Gandiva] Fix order of includes. Fixing the order of includes to follow style guideline. The order to follow is documented here : https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes Also enabled the check in lint.

[Gandiva] Gandiva Java APIs Added the JNI Implementation of the Java APIs Added Java based unit and integration tests Use cmake to build gandiva_jni Added pom.xml to build Java files

Building the java library as part of travis CI. We build the library in the before script and run the tests in the script part. This is to kickoff the travis integration, further work is coming to automate the library location and making it os independent.

Add Java coding guideline rules Add validation checks to validate Java coding guidelines Change-Id: Iae6b16dad0cc72ec760ecb2f2232a7c713dc4b8d

Validating the input schema and expressions during the projector build.

Fixed the license to be Dremio and not Apache licenses. Failing on check stlye errors. Include exceptions in the gandiva java target.

- tree builder api for and/or - decomposer/validator for and/or - code generator for and/or - tests for and/or

- add tree-builder, codegen support for null literals - moved the code for final bitmap computation to class BitMapAccumulator

add java bindings for and/or

add java bindings for null literals

Support date/time types in Java Add cpp/Java tests for date/time types

Loading Gandiva dynamically in java bindings. Packaging the dynamic library and byte code files in Gandiva JAR. Introduced configuration object to customize Gandiva at runtime.

- Track offsets buffer for string/binary - annotator/generator support for string/binary - literal support for string/binary

Modified the build to package the gandiva jni as a stand alone library that can be packaged in the Gandiva JAR. Also producing two versions of gandiva core - a static and a shared one. Fixed LLVM dependencies to be target based.

- added target "make stylecheck" to check style - added target "make stylefix" to check style - fixed README.md - fixed ci script - used stylefix to fix all existing style violations

Deploying the Gandiva Jar to OSSRH after master merge builds.

- added java bindings for varlen types/literals - minor cleanups in llvm generator and engine

Added microbenchmarks in both cpp and Java

Support isnull, isnotnull, equal, and not_equal for date/time types Support date/time types for less_than, less_than_or_equal_to, greater_than, greater_than_or_equal_to Implement all extractXxx functions

- Switched to gcc-4.9, since the stdc++ linked with 4.8 doesn't work with llvm libs. - Build arrow in travis instead of the conda build - fixed an error in node.h that showed up when I toyed with clang compiler

Exporting supported data types and functions from Gandiva. Added a JNI bridge to access this from the java layer.

* Fix missing set the include directory of gtest * Fix to use same format as other dependencies Change-Id: Iad193e219fc07f777988984db325ce97ad83b545

Fixed the implementation of extract second from time.

[Gandiva] Add hash functions on all data types [Gandiva] Fix stylecheck in travis to print diff [Gandiva] pick clang-format from llvm-binary dir [Gandiva] handle case when seed is null [Gandiva] Fix a style check

Added support for literals and null for time types.

Class references are local by default and eligible for GC. We would need to convert it to global reference on library load for it to be safely used for the program lifetime.

Add support for timestampaddXxx functions Add support for is_distinct_from, is_not_distinct_from, isnull, isnotnull, date_add/add, date_sub/subtract/date_diff, date_trunc_Xxx functions

* Temporarily matching what the dremio does for mod zero. * Used the latest Arrow APIs for allocating buffers.

- similar to projection, filter is built for a specific schema and condition - the output of filter is a selection vector

* Add java bindings for filter expr * Mv selection vector impl to internal

Fixed some bugs in the filter code path.

Change the selection vector arrays as unsigned to match dremio.

1. Added lock to holder read to address potential race condition. 2. Fixed log message. 3, Addressed breaking arrow change.

1. In evaluate to lookup module, first do without lock and fallback only if module is not found. 2. Use release builds in travis.

Introducing a cache to hold the projectors and filters for re-use. The cache is a LRU that can hold 100 entries.

[Gandiva] Fixed concurrency issue in cache. Modifications were happening in get without a mutex. Wrote a test to verify and prevent regression.

Literal string coversion was ignoring types, leading to mismatch in hashing of expressions.

- add a registry for "function holders" implemented in cpp - the function holder is instantiated at expression decomposition time - at eval time, the registered fn gets an extra param

- To get around the java load issue, create a native library and load it in the LLVM module. This module has the hooks for all the c++ function helpers. - for files that are compiled in libgandiva_helpers, add into gandiva::helpers namespace. - merged status.cc into status.h

- reduce benchmark iterations to 1M instead of 100M - add checks in benchmark test to verify that elapsed_time is atleast <= 2 * expected

…es to apache license. Fix clang-format, cpplint warnings, -Wconversion warnings and other warnings with -DBUILD_WARNING_LEVEL=CHECKIN. Fix some build toolchain issues, Arrow target dependencies. Remove some unused CMake code

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-3182: [C++][Java] Gandiva merge into Arrow #2558

ARROW-3182: [C++][Java] Gandiva merge into Arrow #2558

Commits on Sep 29, 2018