Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-47981][BUILD] Upgrade Arrow to 16.0.0 #46214

Closed
wants to merge 2 commits into from
Closed

[SPARK-47981][BUILD] Upgrade Arrow to 16.0.0 #46214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ae052f3
arrow 16.0.0
cxzl25 Apr 25, 2024
15cd30a
deps
cxzl25 Apr 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 6 additions & 6 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,11 @@ antlr4-runtime/4.13.1//antlr4-runtime-4.13.1.jar
aopalliance-repackaged/3.0.3//aopalliance-repackaged-3.0.3.jar
arpack/3.0.3//arpack-3.0.3.jar
arpack_combined_all/0.1//arpack_combined_all-0.1.jar
arrow-format/15.0.2//arrow-format-15.0.2.jar
arrow-memory-core/15.0.2//arrow-memory-core-15.0.2.jar
arrow-memory-netty/15.0.2//arrow-memory-netty-15.0.2.jar
arrow-vector/15.0.2//arrow-vector-15.0.2.jar
arrow-format/16.0.0//arrow-format-16.0.0.jar
arrow-memory-core/16.0.0//arrow-memory-core-16.0.0.jar
arrow-memory-netty-buffer-patch/16.0.0//arrow-memory-netty-buffer-patch-16.0.0.jar
arrow-memory-netty/16.0.0//arrow-memory-netty-16.0.0.jar
arrow-vector/16.0.0//arrow-vector-16.0.0.jar
audience-annotations/0.12.0//audience-annotations-0.12.0.jar
avro-ipc/1.11.3//avro-ipc-1.11.3.jar
avro-mapred/1.11.3//avro-mapred-1.11.3.jar
Expand All @@ -33,6 +34,7 @@ breeze-macros_2.13/2.1.0//breeze-macros_2.13-2.1.0.jar
breeze_2.13/2.1.0//breeze_2.13-2.1.0.jar
bundle/2.24.6//bundle-2.24.6.jar
cats-kernel_2.13/2.8.0//cats-kernel_2.13-2.8.0.jar
checker-qual/3.42.0//checker-qual-3.42.0.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was checker-qual added?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be added here to add annotation.

chill-java/0.10.0//chill-java-0.10.0.jar
chill_2.13/0.10.0//chill_2.13-0.10.0.jar
commons-cli/1.6.0//commons-cli-1.6.0.jar
Expand Down Expand Up @@ -62,8 +64,6 @@ derby/10.16.1.1//derby-10.16.1.1.jar
derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
eclipse-collections-api/11.1.0//eclipse-collections-api-11.1.0.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were these two dependencies removed in the new version?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's removed via

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got

eclipse-collections/11.1.0//eclipse-collections-11.1.0.jar
esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar
flatbuffers-java/23.5.26//flatbuffers-java-23.5.26.jar
gcs-connector/hadoop3-2.2.21/shaded/gcs-connector-hadoop3-2.2.21-shaded.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@
./python/pyspark/sql/pandas/utils.py, ./python/packaging/classic/setup.py
and ./python/packaging/connect/setup.py too.
-->
<arrow.version>15.0.2</arrow.version>
<arrow.version>16.0.0</arrow.version>
<ammonite.version>3.0.0-M1</ammonite.version>

<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
Expand Down