Description
PR #12625 moves the vcpkg build path to native Arrow 18.1.0, but the standalone build paths are not consistently aligned:
dev/build-arrow.sh defaults to Arrow 15.0.0.
dev/build-arrow-18.sh provides Arrow 18.1.0 only for the current ppc64le Spark 4.0 special case.
- Spark 3.x Maven profiles use Arrow Java 15.0.0, while Spark 4.x profiles use Arrow Java 18.1.0.
- ppc64le builds Arrow Java/JNI artifacts locally, so the native and Java versions must match.
A direct replacement of 15.0.0 with 18.1.0 is unsafe because the standalone Arrow patches need to be reviewed or updated for Arrow 18, and upgrading ppc64le native Arrow without coordinating the Maven Arrow version can produce incompatible JNI artifacts.
The follow-up should:
- Define the intended native/Java Arrow version mapping for Spark 3.x and Spark 4.x standalone builds.
- Update or remove the standalone Arrow patches as needed for Arrow 18.
- Consolidate
build-arrow.sh and build-arrow-18.sh if practical.
- Ensure locally built ppc64le Arrow C Data and Dataset JNI artifacts match the Maven
arrow.version.
- Validate the affected standalone builds and PPC CI paths.
Gluten version: main branch
Generated-by: GitHub Copilot CLI 1.0.78-2
Description
PR #12625 moves the vcpkg build path to native Arrow 18.1.0, but the standalone build paths are not consistently aligned:
dev/build-arrow.shdefaults to Arrow 15.0.0.dev/build-arrow-18.shprovides Arrow 18.1.0 only for the current ppc64le Spark 4.0 special case.A direct replacement of 15.0.0 with 18.1.0 is unsafe because the standalone Arrow patches need to be reviewed or updated for Arrow 18, and upgrading ppc64le native Arrow without coordinating the Maven Arrow version can produce incompatible JNI artifacts.
The follow-up should:
build-arrow.shandbuild-arrow-18.shif practical.arrow.version.Gluten version: main branch
Generated-by: GitHub Copilot CLI 1.0.78-2