Skip to content

[SPARK-55803][BUILD] Bump lz4-java 1.10.4 to bring back performance #54585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pan3793 wants to merge 4 commits into from
+8 −8
Closed

[SPARK-55803][BUILD] Bump lz4-java 1.10.4 to bring back performance #54585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2161965
[SPARK-55803] Bump lz4-java 1.10.4
pan3793 Mar 3, 2026
e0de349
Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …
pan3793 Mar 3, 2026
f9fa56e
Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …
pan3793 Mar 3, 2026
7435c19
Benchmark results for org.apache.spark.io.LZ4TPCDSDataBenchmark (JDK …
pan3793 Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions core/benchmarks/LZ4TPCDSDataBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression 4 times 2611 2619 11 0.0 652760335.3 1.0X
Compression 4 times 2376 2382 9 0.0 593993111.8 1.0X

OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Decompression 4 times 896 912 20 0.0 224050201.0 1.0X
Decompression 4 times 731 757 29 0.0 182787977.2 1.0X


4 changes: 2 additions & 2 deletions core/benchmarks/LZ4TPCDSDataBenchmark-jdk25-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression 4 times 2611 2620 12 0.0 652865216.0 1.0X
Compression 4 times 2373 2380 10 0.0 593284150.8 1.0X

OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Decompression 4 times 905 916 14 0.0 226166738.0 1.0X
Decompression 4 times 694 710 23 0.0 173453930.5 1.0X


4 changes: 2 additions & 2 deletions core/benchmarks/LZ4TPCDSDataBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Compression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Compression 4 times 2602 2609 10 0.0 650438397.0 1.0X
Compression 4 times 2376 2383 10 0.0 593917596.8 1.0X

OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
AMD EPYC 7763 64-Core Processor
Decompression: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Decompression 4 times 938 966 33 0.0 234424499.5 1.0X
Decompression 4 times 759 761 2 0.0 189753707.8 1.0X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ log4j-api/2.25.3//log4j-api-2.25.3.jar
log4j-core/2.25.3//log4j-core-2.25.3.jar
log4j-layout-template-json/2.25.3//log4j-layout-template-json-2.25.3.jar
log4j-slf4j2-impl/2.25.3//log4j-slf4j2-impl-2.25.3.jar
lz4-java/1.10.3//lz4-java-1.10.3.jar
lz4-java/1.10.4//lz4-java-1.10.4.jar
metrics-core/4.2.37//metrics-core-4.2.37.jar
metrics-graphite/4.2.37//metrics-graphite-4.2.37.jar
metrics-jmx/4.2.37//metrics-jmx-4.2.37.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -888,7 +888,7 @@
<dependency>
<groupId>at.yawk.lz4</groupId>
<artifactId>lz4-java</artifactId>
<version>1.10.3</version>
<version>1.10.4</version>
</dependency>
<dependency>
<groupId>com.github.luben</groupId>
Expand Down