Skip to content

Commit

Permalink
ORC-1704: Migration to Scala 2.13 of Apache Spark 3.5.1 at SparkBench…
Browse files Browse the repository at this point in the history
…mark

### What changes were proposed in this pull request?
This PR aims to migrate to Scala 2.13 of Apache Spark 3.5.1 at SparkBenchmark.

### Why are the changes needed?
#1909 (review)

### How was this patch tested?
local test

```bash
java -jar spark/target/orc-benchmarks-spark-2.1.0-SNAPSHOT.jar spark data -format=parquet  -compress zstd -data taxi
```

```
Benchmark                                  (compression)  (dataset)  (format)  Mode  Cnt          Score       Error  Units
SparkBenchmark.partialRead                          zstd       taxi   parquet  avgt    5      17211.731 ± 11836.315  us/op
SparkBenchmark.partialRead:bytesPerRecord           zstd       taxi   parquet  avgt    5          0.002                  #
SparkBenchmark.partialRead:ops                      zstd       taxi   parquet  avgt    5         10.000                  #
SparkBenchmark.partialRead:perRecord                zstd       taxi   parquet  avgt    5          0.001 ±     0.001  us/op
SparkBenchmark.partialRead:records                  zstd       taxi   parquet  avgt    5  113791180.000                  #
```

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #1912 from cxzl25/ORC-1704.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
cxzl25 authored and dongjoon-hyun committed Apr 25, 2024
1 parent 952b479 commit dc634cb
Show file tree
Hide file tree
Showing 3 changed files with 12 additions and 10 deletions.
12 changes: 7 additions & 5 deletions java/bench/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
<junit.version>5.10.2</junit.version>
<orc.version>${project.version}</orc.version>
<parquet.version>1.13.1</parquet.version>
<scala.binary.version>2.13</scala.binary.version>
<scala.version>2.13.8</scala.version>
<spark.version>3.5.1</spark.version>
</properties>

Expand Down Expand Up @@ -284,12 +286,12 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.12</artifactId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
Expand All @@ -316,7 +318,7 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
Expand All @@ -335,7 +337,7 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<artifactId>spark-avro_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
Expand All @@ -357,7 +359,7 @@
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.12.18</version>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
Expand Down
8 changes: 4 additions & 4 deletions java/bench/spark/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,15 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.12</artifactId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
Expand All @@ -88,7 +88,7 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<artifactId>spark-avro_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.jodd</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,9 @@
import scala.Tuple2;
import scala.collection.Iterator;
import scala.collection.JavaConverters;
import scala.collection.Seq;
import scala.collection.immutable.Map;
import scala.collection.immutable.Map$;
import scala.collection.immutable.Seq;

import java.io.IOException;
import java.sql.Timestamp;
Expand Down

0 comments on commit dc634cb

Please sign in to comment.