Skip to content

[SPARK-45145][EXAMPLE] Add JavaSparkSQLCli example#42900

Closed
dongjoon-hyun wants to merge 2 commits intoapache:masterfrom
dongjoon-hyun:SPARK-45145
Closed

[SPARK-45145][EXAMPLE] Add JavaSparkSQLCli example#42900
dongjoon-hyun wants to merge 2 commits intoapache:masterfrom
dongjoon-hyun:SPARK-45145

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Sep 13, 2023

What changes were proposed in this pull request?

This PR aims to add a simple Java example, JavaSparkSQLCli.java. Like SparkPi example, we can take advantage of this minimal example with bin/run-example and we can run a simple SQL query as a canary test during RC testing and voting phase.

./bin/run-example SparkPi 10

After this PR,

$ bin/run-example sql.JavaSparkSQLCli "SELECT 'Spark SQL' col"
+---------+
|col      |
+---------+
|Spark SQL|
+---------+

Why are the changes needed?

Although there exists org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver and bin/spark-sql shell environment, it requires -Phive -Phive-thriftserver explicitly.

$ bin/spark-shell --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.0
      /_/
...

$ bin/run-example org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
...
Error: Failed to load class org.apache.spark.examples.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
Failed to load main class org.apache.spark.examples.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.
23/09/12 23:06:38 INFO ShutdownHookManager: Shutdown hook called
23/09/12 23:06:38 INFO ShutdownHookManager: Deleting directory /private/var/folders/d4/dr6zxyvd4cl38877bj3fxs_m0000gn/T/spark-2a9ee4a3-64c3-492c-a268-a71b9aa44a2a

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually.

$ build/sbt test:package
$ bin/run-example sql.JavaSparkSQLCli "SELECT 'Spark SQL' col"

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

Hi, @HyukjinKwon . Do you think we can have this additional example?

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon and @yaooqinn .

@dongjoon-hyun
Copy link
Member Author

Merged to master for Apache Spark 4.0.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-45145 branch September 13, 2023 07:20
viirya pushed a commit to viirya/spark-1 that referenced this pull request Oct 19, 2023
### What changes were proposed in this pull request?

This PR aims to add a simple Java example, `JavaSparkSQLCli.java`. Like SparkPi example, we can take advantage of this minimal example with `bin/run-example` and we can run a simple SQL query as a canary test during RC testing and voting phase.

- https://spark.apache.org/docs/3.5.0/

```
./bin/run-example SparkPi 10
```

 After this PR,
```
$ bin/run-example sql.JavaSparkSQLCli "SELECT 'Spark SQL' col"
+---------+
|col      |
+---------+
|Spark SQL|
+---------+
```

### Why are the changes needed?

Although there exists `org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver` and `bin/spark-sql` shell environment, it requires `-Phive -Phive-thriftserver` explicitly.
```
$ bin/spark-shell --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.0
      /_/
...

$ bin/run-example org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
...
Error: Failed to load class org.apache.spark.examples.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
Failed to load main class org.apache.spark.examples.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.
23/09/12 23:06:38 INFO ShutdownHookManager: Shutdown hook called
23/09/12 23:06:38 INFO ShutdownHookManager: Deleting directory /private/var/folders/d4/dr6zxyvd4cl38877bj3fxs_m0000gn/T/spark-2a9ee4a3-64c3-492c-a268-a71b9aa44a2a
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually.
```
$ build/sbt test:package
$ bin/run-example sql.JavaSparkSQLCli "SELECT 'Spark SQL' col"
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42900 from dongjoon-hyun/SPARK-45145.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 66f42b5)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants