[SPARK-36895][SQL][FOLLOWUP] Use property to specify index type #34486

huaxingao · 2021-11-04T19:43:51Z

What changes were proposed in this pull request?

use property to specify index type

Why are the changes needed?

to address this comment #34148 (comment)

Does this PR introduce any user-facing change?

Yes

  void createIndex(String indexName,
      String indexType,
      NamedReference[] columns,
      Map<NamedReference, Map<String, String>> columnsProperties,
      Map<String, String> properties)

changed to

createIndex(String indexName,
      NamedReference[] columns,
      Map<NamedReference, Map<String, String>> columnsProperties,
      Map<String, String> properties

How was this patch tested?

new test

SparkQA · 2021-11-04T19:50:24Z

Test build #144909 has finished for PR 34486 at commit e282022.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-04T20:29:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49379/

SparkQA · 2021-11-04T21:13:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49379/

SparkQA · 2021-11-04T21:25:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49380/

SparkQA · 2021-11-04T22:23:50Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49380/

SparkQA · 2021-11-04T23:32:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49381/

SparkQA · 2021-11-05T00:14:38Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49381/

SparkQA · 2021-11-05T00:29:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49382/

SparkQA · 2021-11-05T01:23:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49382/

SparkQA · 2021-11-05T01:37:19Z

Test build #144910 has finished for PR 34486 at commit f6decfb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-05T03:46:05Z

Test build #144911 has finished for PR 34486 at commit b9ce735.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-05T04:39:20Z

Test build #144912 has finished for PR 34486 at commit 9ba3f8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-11-05T06:32:06Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala


    CreateIndex(
      createUnresolvedTable(ctx.multipartIdentifier(), "CREATE INDEX"),
      indexName,
-      indexType,


Let's keep the plan unchanged. We should put indexType in the properties in CreateIndexExec.

Also let's not hardcode the property name. We can add a static constant in SupportsIndex, similar to TableCatalog.PROP_PROVIDER

SparkQA · 2021-11-05T18:32:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49410/

SparkQA · 2021-11-05T19:25:33Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49410/

SparkQA · 2021-11-05T22:44:06Z

Test build #144938 has finished for PR 34486 at commit b2f1f90.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-11-08T03:53:31Z

...er-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala

    // MySQL doesn't allow property set on individual column, so use empty Array for
    // column properties
-    jdbcTable.createIndex("i1", "BTREE", Array(FieldReference("col1")),
+    jdbcTable.createIndex("i1", Array(FieldReference("col1")),


TBH it's a bit weird to write UT in this suite. Can we turn it into end-to-end test now as we have the SQL syntax ready?

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/index/SupportsIndex.java

cloud-fan · 2021-11-08T03:55:37Z

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala

      table match {
        case s: SupportsIndex =>
+          val propertiesWithIndexType: Map[String, String] = if (indexType.nonEmpty) {


Can we put it inside CreateIndexExec?

cloud-fan · 2021-11-08T03:55:50Z

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala

      table match {
        case s: SupportsIndex =>
+          val propertiesWithIndexType: Map[String, String] = if (indexType.nonEmpty) {
+             properties + ("indexType" -> indexType)


let's not hardcode the property name

…log/index/SupportsIndex.java Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

SparkQA · 2021-11-08T06:04:11Z

Test build #144984 has finished for PR 34486 at commit 27939f8.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-08T06:10:53Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49455/

cloud-fan · 2021-11-08T06:33:34Z

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala

@@ -432,7 +432,7 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
      AlterTableExec(table.catalog, table.identifier, a.changes) :: Nil

    case CreateIndex(ResolvedTable(_, _, table, _),
-        indexName, indexType, ifNotExists, columns, properties) =>
+      indexName, indexType, ifNotExists, columns, properties) =>


unnecessary change.

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala

…rces/v2/DataSourceV2Strategy.scala

SparkQA · 2021-11-08T07:08:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49457/

SparkQA · 2021-11-08T07:46:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49459/

SparkQA · 2021-11-08T08:10:48Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49457/

SparkQA · 2021-11-08T08:46:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49459/

SparkQA · 2021-11-08T11:46:40Z

Test build #144987 has finished for PR 34486 at commit 277e741.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-11-08T12:58:36Z

thanks, merging to master!

sarutak · 2021-11-08T14:12:34Z

@huaxingao @cloud-fan Sorry, but I'll revert this change because of the GA failure, which seems to be related this change.
https://github.com/huaxingao/spark/runs/4135539301?check_suite_focus=true#step:9:5083

huaxingao · 2021-11-08T15:06:58Z

sorry

### What changes were proposed in this pull request? use property to specify index type ### Why are the changes needed? fix scala doc in #34486 and resubmit ### Does this PR introduce _any_ user-facing change? Yes ``` void createIndex(String indexName, String indexType, NamedReference[] columns, Map<NamedReference, Map<String, String>> columnsProperties, Map<String, String> properties) ``` changed to ``` createIndex(String indexName, NamedReference[] columns, Map<NamedReference, Map<String, String>> columnsProperties, Map<String, String> properties ``` ### How was this patch tested? new test Closes #34523 from huaxingao/newDelete. Authored-by: Huaxin Gao <huaxin_gao@apple.com> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

huaxingao added 2 commits November 4, 2021 12:26

remove index type

1659343

nit

e282022

github-actions bot added the SQL label Nov 4, 2021

fix style

f6decfb

fix java doc error

b9ce735

fix java doc build failure

9ba3f8a

cloud-fan reviewed Nov 5, 2021

View reviewed changes

address comments

b2f1f90

cloud-fan reviewed Nov 8, 2021

View reviewed changes

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/index/SupportsIndex.java Outdated Show resolved Hide resolved

cloud-fan reviewed Nov 8, 2021

View reviewed changes

Update sql/catalyst/src/main/java/org/apache/spark/sql/connector/cata…

27939f8

…log/index/SupportsIndex.java Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

address comments

b972bae

cloud-fan reviewed Nov 8, 2021

View reviewed changes

cloud-fan approved these changes Nov 8, 2021

View reviewed changes

cloud-fan reviewed Nov 8, 2021

View reviewed changes

...core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala Outdated Show resolved Hide resolved

Update sql/core/src/main/scala/org/apache/spark/sql/execution/datasou…

277e741

…rces/v2/DataSourceV2Strategy.scala

cloud-fan closed this in 4011dd6 Nov 8, 2021

huaxingao mentioned this pull request Nov 8, 2021

[SPARK-36895][SQL][FOLLOWUP] Use property to specify index type #34523

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36895][SQL][FOLLOWUP] Use property to specify index type #34486

[SPARK-36895][SQL][FOLLOWUP] Use property to specify index type #34486

huaxingao commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

cloud-fan Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

cloud-fan Nov 8, 2021

cloud-fan Nov 8, 2021

cloud-fan Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

cloud-fan Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

cloud-fan commented Nov 8, 2021

sarutak commented Nov 8, 2021 •

edited

Loading

huaxingao commented Nov 8, 2021

[SPARK-36895][SQL][FOLLOWUP] Use property to specify index type #34486

[SPARK-36895][SQL][FOLLOWUP] Use property to specify index type #34486

Conversation

huaxingao commented Nov 4, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 4, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

cloud-fan Nov 5, 2021

Choose a reason for hiding this comment

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

SparkQA commented Nov 5, 2021

cloud-fan Nov 8, 2021

Choose a reason for hiding this comment

cloud-fan Nov 8, 2021

Choose a reason for hiding this comment

cloud-fan Nov 8, 2021

Choose a reason for hiding this comment

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

cloud-fan Nov 8, 2021

Choose a reason for hiding this comment

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

SparkQA commented Nov 8, 2021

cloud-fan commented Nov 8, 2021

sarutak commented Nov 8, 2021 • edited Loading

huaxingao commented Nov 8, 2021

sarutak commented Nov 8, 2021 •

edited

Loading