[Java][Docs] ArrowSubstrait options preventing jni modules from being build

[zinking]

Describe the enhancement requested

I am following documentation doc to build up the dataset jar. everything works fine till this step To build all JNI libraries (MacOS / Linux) except the JNI C Data Interface library:

and then I encountered following error:

-- Found the ArrowDataset static library: /Users/zhnwang/zhenw/arrow/java-dist/lib/x86_64/libarrow_dataset.a
CMake Error at dataset/CMakeLists.txt:19 (find_package):
  By not providing "FindArrowSubstrait.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "ArrowSubstrait", but CMake did not find one.

  Could not find a package configuration file provided by "ArrowSubstrait"
  with any of the following names:

    ArrowSubstraitConfig.cmake
    arrowsubstrait-config.cmake

I took a look at the commit history and tried with following updated command

mvn generate-resources \
    -Pgenerate-libs-jni-macos-linux \
    -DARROW_GANDIVA=ON \
    -DARROW_SUBSTRAIT=OFF \  <- added
    -DARROW_JAVA_JNI_ENABLE_GANDIVA=ON \
    -N

still the same error?

I suppose the documentation should be added after ArrowSubstrait change?

Component(s)

Java

[raulcd]

CC ~ @danepitkin @davisusanibar

[davisusanibar]

Hi @zinking thank for reporting this.

I will update the documentation to include Substrait changes as well.

[danepitkin]

You need to build with ARROW_SUBSTRAIT=ON. It's currently required when building the dataset module.

[danepitkin]

Hey @davisusanibar , I went ahead and created a PR to fix it.

[zinking]

@danepitkin I'm actually not sure why dataset isn't built by default in the java folder, I suppose there is currently nothing stopping us from doing so?

[danepitkin]

The dataset module relies on C++ shared libraries, which need to be built first. The default java modules (e.g. built from mvn clean install) don't rely on any shared libs.

[zinking]

@danepitkin it seems I still have to figure out how to build the complete jni dataset jar myself (mvn clean install usually misses the jni libraries), instead of one maven command.
but it's there on maven repo, shouldn't there be 1 for the development phase ?

[danepitkin]

Are you suggesting to have one maven build profile that builds all of java modules/C++ libs/JNI modules? You could script this pretty easily. I believe maven is meant to be used to build one component at a time which is why there is one maven command for each of java modules/C++ libs/JNI modules. @davisusanibar any thoughts on this?

[danepitkin]

Actually, I guess this is trivial to do. This CI build script supposedly builds all of Java in one command: https://github.com/apache/arrow/blob/main/ci/scripts/java_full_build.sh#L49

[zinking]

Actually, I guess this is trivial to do. This CI build script supposedly builds all of Java in one command: https://github.com/apache/arrow/blob/main/ci/scripts/java_full_build.sh#L49

exactly what I am looking for, thanks a lot.

[zinking]

@danepitkin one more question: I can now successfully build the arm64 arch artifact, but in the jar there is no x8664 arch artifact. I am not sure if this is possible on mac build environment. or how should I do the cross compilation to get a usable release artifact jar? thanks.

[kou]

You can use Docker to build x86_64 artifacts on arm64 machine. Our CI job also uses Docker:

arrow/dev/tasks/java-jars/github.yml

Lines 54 to 57 in 37cb592

	archery docker run \
	-e ARROW_JAVA_BUILD=OFF \
	-e ARROW_JAVA_TEST=OFF \
	java-jni-manylinux-2014

BTW, for a development use case, you may want to use nightly packages: https://arrow.apache.org/docs/dev/developers/java/building.html#installing-nightly-packages