Add spark connector #1780

rupal-bq · 2023-06-26T16:52:09Z

Description

Setup spark connector module

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

codecov · 2023-06-26T17:17:00Z

Codecov Report

Merging #1780 (be15dae) into main (fa840e0) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1780      +/-   ##
============================================
+ Coverage     97.32%   97.33%   +0.01%     
- Complexity     4458     4490      +32     
============================================
  Files           388      394       +6     
  Lines         11050    11118      +68     
  Branches        790      795       +5     
============================================
+ Hits          10754    10822      +68     
  Misses          289      289              
  Partials          7        7

Flag	Coverage Δ
sql-engine	`97.33% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...implementation/SparkSqlFunctionImplementation.java	`100.00% <100.00%> (ø)`
...ctions/resolver/SparkSqlTableFunctionResolver.java	`100.00% <100.00%> (ø)`
...nctions/scan/SparkSqlFunctionTableScanBuilder.java	`100.00% <100.00%> (ø)`
...ensearch/sql/spark/storage/SparkStorageEngine.java	`100.00% <100.00%> (ø)`
...nsearch/sql/spark/storage/SparkStorageFactory.java	`100.00% <100.00%> (ø)`
...a/org/opensearch/sql/spark/storage/SparkTable.java	`100.00% <100.00%> (ø)`

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

spark/lombok.config

spark/src/main/java/org/opensearch/sql/spark/client/SparkClient.java

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

Yury-Fridlyand · 2023-06-27T19:32:19Z

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

+
+    SparkQueryRequest sparkQueryRequest = new SparkQueryRequest();
+    arguments.forEach(arg -> {
+      String argName = ((NamedArgumentExpression) arg).getArgName();


Is arg always NamedArgumentExpression? How do you guarantee that?
You can do cast in constructor though.

Yury-Fridlyand · 2023-06-27T19:35:05Z

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageEngine.java

+
+  @Override
+  public Table getTable(DataSourceSchemaName dataSourceSchemaName, String tableName) {
+    return null;


Why? Maybe throw?

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageEngine.java

Yury-Fridlyand · 2023-06-27T19:35:20Z

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageFactory.java

+   */
+  StorageEngine getStorageEngine(Map<String, String> requiredConfig) {
+    SparkClient sparkClient = null;
+    //TODO: Initialize spark client


All TODOs in this PR will be resolved in follow up #1790. Trying to split spark connector in 3 parts to avoid large PR.

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageFactory.java

Yury-Fridlyand · 2023-06-27T19:35:37Z

spark/src/test/java/org/opensearch/sql/spark/functions/SqlFunctionImplementationTest.java

+    assertEquals("Invalid Function Argument:tmp", exception.getMessage());
+  }
+
+}


Yury-Fridlyand · 2023-06-27T19:36:12Z

Can you add ITs, docs and doctests?

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

Yury-Fridlyand · 2023-06-27T20:06:15Z

Is explain supported by spark queries? If not, please, add an IT. If yes, please, add an IT too (and maybe doctest/docs too).

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

rupal-bq · 2023-06-28T17:47:54Z

Can you add ITs, docs and doctests?

Yes, I need at least one client implementation for adding proper ITs. So I was planning to add ITs, docs and doctests in follow up #1790

spark/src/main/java/org/opensearch/sql/spark/request/SparkQueryRequest.java

spark/src/main/java/org/opensearch/sql/spark/storage/SparkMetricTable.java

spark/.gitignore

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java

core/src/main/java/org/opensearch/sql/datasource/model/DataSourceType.java

spark/lombok.config

spark/build.gradle

spark/src/main/java/org/opensearch/sql/spark/storage/SparkStorageFactory.java

spark/src/main/java/org/opensearch/sql/spark/functions/resolver/SqlTableFunctionResolver.java

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

penghuo

Thanks for the change. Please add IT in the following PR.

dai-chen

Thanks for the changes!

* Create Spark Connector Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Add spark client and engine Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Remove vars Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Spark connector draft Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix license header Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Add spark storage test Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update tests Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Refactor class name Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comment Signed-off-by: Rupal Mahajan <maharup@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Rupal Mahajan <maharup@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> (cherry picked from commit a816a58)

* Create Spark Connector Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Add spark client and engine Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Remove vars Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Spark connector draft Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix license header Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Add spark storage test Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update tests Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Refactor class name Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comment Signed-off-by: Rupal Mahajan <maharup@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Rupal Mahajan <maharup@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>

* Create Spark Connector Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Add spark client and engine Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Remove vars Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Spark connector draft Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix license header Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Add spark storage test Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update tests Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Refactor class name Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comment Signed-off-by: Rupal Mahajan <maharup@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Rupal Mahajan <maharup@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> (cherry picked from commit a816a58)

* Add spark connector (#1780) * Create Spark Connector Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Add spark client and engine Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Remove vars Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Spark connector draft Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix license header Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Add spark storage test Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update tests Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Refactor class name Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comment Signed-off-by: Rupal Mahajan <maharup@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Rupal Mahajan <maharup@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> (cherry picked from commit a816a58) * Upgrade httpclient to 4.5.14 to fix build Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Revert Upgrade httpclient to 4.5.14 Signed-off-by: Rupal Mahajan <maharup@amazon.com> --------- Signed-off-by: Rupal Mahajan <maharup@amazon.com> Co-authored-by: Rupal Mahajan <maharup@amazon.com>

* Create Spark Connector Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Add spark client and engine Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Remove vars Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Spark connector draft Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle errors Signed-off-by: Rupal Mahajan <maharup@amazon.com> * nit Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix license header Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Add spark storage test Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Fix checkstyle in comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Update tests Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comments Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Refactor class name Signed-off-by: Rupal Mahajan <maharup@amazon.com> * Address PR comment Signed-off-by: Rupal Mahajan <maharup@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Signed-off-by: Rupal Mahajan <maharup@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com> (cherry picked from commit a816a58) Co-authored-by: Rupal Mahajan <maharup@amazon.com>

vamsi-amazon and others added 7 commits June 7, 2023 18:00

Create Spark Connector

70588d6

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Add spark client and engine

2b8ceb7

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Remove vars

f595ff4

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Merge branch 'main' into create-spark-connector

95a86d5

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Merge branch 'main' into create-spark-connector

51700bd

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Spark connector draft

d0d5042

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

nit

5ac01b2

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

rupal-bq added 8 commits June 26, 2023 11:13

Fix checkstyle errors

4fcf4ba

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

nit

52a044d

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Fix license header

69ca116

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Add spark storage test

901e762

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Update comments

1b81453

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Fix checkstyle in comments

106218e

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Merge branch 'main' into create-spark-connector

6da93f3

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

Update tests

b953aa6

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

rupal-bq marked this pull request as ready for review June 26, 2023 23:15

rupal-bq requested review from pjfitzgibbons, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, vamsi-amazon, Swiddis, penghuo, seankao-az and MaxKsyunz as code owners June 26, 2023 23:15

Yury-Fridlyand reviewed Jun 27, 2023

View reviewed changes

matthewryanwells reviewed Jun 27, 2023

View reviewed changes

...c/main/java/org/opensearch/sql/spark/functions/implementation/SqlFunctionImplementation.java Outdated Show resolved Hide resolved

Address PR comments

0df37a8

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

dai-chen reviewed Jun 29, 2023

View reviewed changes