Skip to content

[VL] Align standalone Arrow native and Java versions #12673

Description

@baibaichen

Description

PR #12625 moves the vcpkg build path to native Arrow 18.1.0, but the standalone build paths are not consistently aligned:

  • dev/build-arrow.sh defaults to Arrow 15.0.0.
  • dev/build-arrow-18.sh provides Arrow 18.1.0 only for the current ppc64le Spark 4.0 special case.
  • Spark 3.x Maven profiles use Arrow Java 15.0.0, while Spark 4.x profiles use Arrow Java 18.1.0.
  • ppc64le builds Arrow Java/JNI artifacts locally, so the native and Java versions must match.

A direct replacement of 15.0.0 with 18.1.0 is unsafe because the standalone Arrow patches need to be reviewed or updated for Arrow 18, and upgrading ppc64le native Arrow without coordinating the Maven Arrow version can produce incompatible JNI artifacts.

The follow-up should:

  • Define the intended native/Java Arrow version mapping for Spark 3.x and Spark 4.x standalone builds.
  • Update or remove the standalone Arrow patches as needed for Arrow 18.
  • Consolidate build-arrow.sh and build-arrow-18.sh if practical.
  • Ensure locally built ppc64le Arrow C Data and Dataset JNI artifacts match the Maven arrow.version.
  • Validate the affected standalone builds and PPC CI paths.

Gluten version: main branch

Generated-by: GitHub Copilot CLI 1.0.78-2

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions