Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Java] Distribute Apple M1 compatible JNI libraries via mavencentral #33245

Closed
asfimport opened this issue Oct 13, 2022 · 10 comments
Closed

[Java] Distribute Apple M1 compatible JNI libraries via mavencentral #33245

asfimport opened this issue Oct 13, 2022 · 10 comments
Assignees
@rok
Labels
Component: Java Type: enhancement
Milestone
11.0.0

Comments

@asfimport
Copy link

Currently JNI libraries need to be built locally to be usable on Apple silicon. We should build and distribute compatible libraries via mavencentral.

@dsusanibara @lidavidm

Also see ARROW-17267 and ARROW-16608

Reporter: Rok Mihevc / @rok
Assignee: Rok Mihevc / @rok
Watchers: Rok Mihevc / @rok

PRs and other links:

Note: This issue was originally created as ARROW-18042. Please see the migration documentation for further details.
@asfimport
Copy link
Author

David Li / @lidavidm:
@davisusanibar I thought you already had a ticket for this

@asfimport
Copy link
Author

David Dali Susanibar Arce / @davisusanibar:
Thank you @rok  for your testing on M1.

First of all I need to read is there are some SDK to be able to build jar packages and cover different OS Unix / MacOS M1 / Windows. Base on your testing using Zuul SDK you were able to use Dataset without problems.

@asfimport
Copy link
Author

David Dali Susanibar Arce / @davisusanibar:
@kou Do you have some advice about if there are some option to also cover M1 for the Dataset jar packaging?

@asfimport
Copy link
Author

Rok Mihevc / @rok:
Zulu worked fine for me. I see Oracle might support M1 too. I'm not sure how licenses are for these.

@asfimport
Copy link
Author

Kouhei Sutou / @kou:
We need to choose one of the following approaches:

  • Bundle lib*.dylib that contains binaries for both x86_64 and arm64

  • Bundle lib*.dylib for x86_64 and lib*.dylib for arm64 separately

    See also: https://issues.apache.org/jira/browse/ARROW-16608?focusedCommentId=17539166#comment-17539166

    If we choose the former, we need to add support for -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" in our Apache Arrow C++ CMake configuration.

    If we choose the latter, we need a change like the following + a Java side change that choose suitable .dylib that depends on host architecture:

    {noformat}
    diff --git a/dev/tasks/java-jars/github.yml b/dev/tasks/java-jars/github.yml
    index 6f7fdc82d5..f3f763c31a 100644
    — a/dev/tasks/java-jars/github.yml
    +++ b/dev/tasks/java-jars/github.yml
    @@ -47,8 +47,8 @@ jobs:
    run: archery docker push java-jni-manylinux-2014
    {% endif %}

  • build-cpp-macos:

  • name: Build C++ libraries macOS

  • build-cpp-macos-x86_64:
  • name: Build C++ libraries macOS x86_64
    runs-on: macos-latest
    env:
    MACOSX_DEPLOYMENT_TARGET: "10.13"
    @@ -76,12 +76,54 @@ jobs:
    $GITHUB_WORKSPACE/arrow/cpp-build
    $GITHUB_WORKSPACE/arrow/java-dist
  • name: Compress into single artifact to keep directory structure
  •    run: tar -cvzf arrow-shared-libs-macos.tar.gz arrow/java-dist/

    •    run: |

    •      mkdir -p arrow/java-dist/x86_64

    •      mv arrow/java-dist/**.** arrow/java-dist/x86_64/

    •      tar -cvzf arrow-shared-libs-macos-x86_64.tar.gz arrow/java-dist/
  • name: Upload artifacts
    uses: actions/upload-artifact@v2
    with:
  •      name: macos-shared-lib

  •      path: arrow-shared-libs-macos.tar.gz

    •      name: macos-shared-lib-x86_64

    •      path: arrow-shared-libs-macos-x86_64.tar.gz

    • build-cpp-macos-arm64:

    • name: Build C++ libraries macOS arm64

    • runs-on: ["self-hosted", "macOS", "arm64"]

    • env:

    •  MACOSX_DEPLOYMENT_TARGET: "10.13"

    • steps:

    •  ` macros.github_checkout_arrow()|indent `

    •  ` macros.github_install_archery()|indent `

    •  - name: Install dependencies

    •    run: |

    •      brew install --overwrite git

    •      brew bundle --file=arrow/cpp/Brewfile

    •      # We want to link aws-sdk-cpp statically but Homebrew's

    •      # aws-sdk-cpp provides only shared library. If we have

    •      # Homebrew's aws-sdk-cpp, our build mix Homebrew's

    •      # aws-sdk-cpp and bundled aws-sdk-cpp. We uninstall Homebrew's

    •      # aws-sdk-cpp to ensure using only bundled aws-sdk-cpp.

    •      brew uninstall aws-sdk-cpp

    •  - name: Setup ccache

    •    run: |

    •      arrow/ci/scripts/ccache_setup.sh

    •  - name: Build C++ libraries

    •    run: |

    •      set -e

    •      arrow/ci/scripts/java_jni_macos_build.sh \

    •        $GITHUB_WORKSPACE/arrow \

    •        $GITHUB_WORKSPACE/arrow/cpp-build \

    •        $GITHUB_WORKSPACE/arrow/java-dist

    •  - name: Compress into single artifact to keep directory structure

    •    run: |

    •      mkdir -p arrow/java-dist/arm64

    •      mv arrow/java-dist/**.** arrow/java-dist/arm64/

    •      tar -cvzf arrow-shared-libs-macos-arm64.tar.gz arrow/java-dist/

    •  - name: Upload artifacts

    •    uses: actions/upload-artifact@v2

    •    with:

    •      name: macos-shared-lib-arm64

    •      path: arrow-shared-libs-macos-arm64.tar.gz

      build-cpp-windows:
      name: Build C++ libraries Windows
      @@ -116,7 +158,8 @@ jobs:
      runs-on: macos-latest
      needs:

  • build-cpp-ubuntu
    • build-cpp-macos 
      •  - build-cpp-macos-x86_64

      •  - build-cpp-macos-arm64
  • build-cpp-windows
    steps:
    macros.github_checkout_arrow(fetch_depth=0)|indent
    @@ -124,10 +167,14 @@ jobs:
    uses: actions/download-artifact@v2
    with:
    name: ubuntu-shared-lib
    • name: Download macOS C++ libraries 
      •  - name: Download macOS x86_64 C++ libraries
   uses: actions/download-artifact@v2
   with:

  •      name: macos-shared-lib

    •      name: macos-shared-lib-x86_64

    •  - name: Download macOS arm64 C++ libraries

    •    uses: actions/download-artifact@v2

    •    with:

    •      name: macos-shared-lib-arm64
  • name: Download Windows C++ libraries
    uses: actions/download-artifact@v2
    with:
    @@ -135,24 +182,30 @@ jobs:
  • name: Descompress artifacts
    run: |
    tar -xvzf arrow-shared-libs-linux.tar.gz
  •      tar -xvzf arrow-shared-libs-macos.tar.gz

    •      tar -xvzf arrow-shared-libs-macos-x86_64.tar.gz

    •      tar -xvzf arrow-shared-libs-macos-arm64.tar.gz
     tar -xvzf arrow-shared-libs-windows.tar.gz
  • name: Test that shared libraries exist
    run: |
    set -x
  •      test -f arrow/java-dist/libarrow_cdata_jni.dylib

    •      test -f arrow/java-dist/x86_64/libarrow_cdata_jni.dylib

    •      test -f arrow/java-dist/arm64/libarrow_cdata_jni.dylib
     test -f arrow/java-dist/libarrow_cdata_jni.so
     test -f arrow/java-dist/arrow_cdata_jni.dll

  •      test -f arrow/java-dist/libarrow_dataset_jni.dylib

    •      test -f arrow/java-dist/x86_64/libarrow_dataset_jni.dylib

    •      test -f arrow/java-dist/arm64/libarrow_dataset_jni.dylib
     test -f arrow/java-dist/libarrow_dataset_jni.so
     test -f arrow/java-dist/arrow_dataset_jni.dll

  •      test -f arrow/java-dist/libarrow_orc_jni.dylib

    •      test -f arrow/java-dist/x86_64/libarrow_orc_jni.dylib

    •      test -f arrow/java-dist/arm64/libarrow_orc_jni.dylib
     test -f arrow/java-dist/libarrow_orc_jni.so
  1. We can enable this after ARROW-17817 is resolved.
  2. test -f arrow/java-dist/arrow_orc_jni.dll 
    •      test -f arrow/java-dist/libgandiva_jni.dylib

      •      test -f arrow/java-dist/x86_64/libgandiva_jni.dylib

      •      test -f arrow/java-dist/arm64/libgandiva_jni.dylib
     test -f arrow/java-dist/libgandiva_jni.so

    •      test -f arrow/java-dist/libplasma_java.dylib

      •      test -f arrow/java-dist/x86_64/libplasma_java.dylib

      •      test -f arrow/java-dist/arm64/libplasma_java.dylib
     test -f arrow/java-dist/libplasma_java.so
    • name: Build bundled jar
      run: |
      {noformat}

@asfimport
Copy link
Author

Rok Mihevc / @rok:
Latter seems better as it keeps CMake simpler.

@asfimport
Copy link
Author

Rok Mihevc / @rok:
@kou I can make a PR with your proposed change. Is that ok?

@asfimport
Copy link
Author

Kouhei Sutou / @kou:
Sure!

@asfimport
Copy link
Author

Rok Mihevc / @rok:
@kou I've opened a PR with this change. One thing I changed was arm64 -> aarch64 as that seems to be the canonical name (see: https://github.com/trustin/os-maven-plugin#property-osdetectedarch).

@asfimport
Copy link
Author

Kouhei Sutou / @kou:
Issue resolved by pull request 14472
#14472

@asfimport asfimport added this to the 11.0.0 milestone Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rok rok

Labels
Component: Java Type: enhancement
Projects
None yet
Milestone
11.0.0
Development

No branches or pull requests
2 participants
@rok @asfimport