Skip to content

Commit

Permalink
[SPARK-47400][BUILD] Upgrade gcs-connector to 2.2.20
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to upgrade `gas-connector` to 2.2.20.

### Why are the changes needed?

To bring the latest updates.
- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.20
    - Add support for renaming folders using rename backend API for Hierarchical namespace buckets
    - Upgrade java-storage to 2.32.1 and upgrade the version of related dependencies
- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.10

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

```
$ dev/make-distribution.sh -Phadoop-cloud
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
        -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
        -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
        -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.13.12 (OpenJDK 64-Bit Server VM, Java 21.0.2)
Type in expressions to have them evaluated.
Type :help for more information.
24/03/14 09:33:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1710434021996).
Spark session available as 'spark'.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
val res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+

scala>
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45521 from dongjoon-hyun/SPARK-47400.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
dongjoon-hyun authored and sweisdb committed Apr 1, 2024
1 parent b8f8347 commit dfa478a
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-met
eclipse-collections-api/11.1.0//eclipse-collections-api-11.1.0.jar
eclipse-collections/11.1.0//eclipse-collections-11.1.0.jar
flatbuffers-java/23.5.26//flatbuffers-java-23.5.26.jar
gcs-connector/hadoop3-2.2.18/shaded/gcs-connector-hadoop3-2.2.18-shaded.jar
gcs-connector/hadoop3-2.2.20/shaded/gcs-connector-hadoop3-2.2.20-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@
<aws.java.sdk.v2.version>2.20.160</aws.java.sdk.v2.version>
<!-- the producer is used in tests -->
<aws.kinesis.producer.version>0.12.8</aws.kinesis.producer.version>
<gcs-connector.version>hadoop3-2.2.18</gcs-connector.version>
<gcs-connector.version>hadoop3-2.2.20</gcs-connector.version>
<!-- org.apache.httpcomponents/httpclient-->
<commons.httpclient.version>4.5.14</commons.httpclient.version>
<commons.httpcore.version>4.4.16</commons.httpcore.version>
Expand Down

0 comments on commit dfa478a

Please sign in to comment.