[SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue #32831

linhongliu-db · 2021-06-09T03:00:26Z

What changes were proposed in this pull request?

If the user creates a view in 2.4 and reads it in 3.1/3.2, there will be an incompatible schema issue.
So this PR adds a view ddl in the error message to prompt the user recreating the view to fix the
incompatible issue.
For example:

-- create view in 2.4
CREATE TABLE IF NOT EXISTS t USING parquet AS SELECT '1' as a, '20210420' as b"
CREATE OR REPLACE VIEW v AS SELECT CAST(t.a AS INT), to_date(t.b, 'yyyyMMdd') FROM t
-- select view in master
SELECT * FROM v

Then we will get below error:

cannot resolve '`to_date(spark_catalog.default.t.b, 'yyyyMMdd')`' given input columns: [a, to_date(b, yyyyMMdd)];

Why are the changes needed?

Improve the error message

Does this PR introduce any user-facing change?

Yes, the error message will change

How was this patch tested?

newly added test case

SparkQA · 2021-06-09T04:27:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44050/

SparkQA · 2021-06-09T05:02:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44050/

SparkQA · 2021-06-09T05:10:55Z

Test build #139525 has finished for PR 32831 at commit 1cda453.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

linhongliu-db · 2021-06-21T06:22:10Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+    val viewName = metadata.identifier.toString
+    val viewText = metadata.viewText
+    val viewColumns = metadata.schema.fieldNames.mkString(", ")
+    s"CREATE OR REPLACE $temp VIEW $viewName ($viewColumns) AS $viewText"


change to alter view is better

cloud-fan · 2021-06-21T06:26:50Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+    val viewName = metadata.identifier.toString
+    val viewText = metadata.viewText
+    val viewColumns = metadata.schema.fieldNames.mkString(", ")
+    s"CREATE OR REPLACE $temp VIEW $viewName ($viewColumns) AS $viewText"


we can generate ALTER VIEW

SparkQA · 2021-06-24T03:25:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44762/

SparkQA · 2021-06-24T03:34:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44762/

SparkQA · 2021-06-24T03:49:01Z

Test build #140234 has finished for PR 32831 at commit f9c4e9d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-06-24T14:03:55Z

sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala

+        sql("SELECT * FROM v").show()
+        sql("DROP TABLE t")
+        sql("CREATE TABLE t(a INT, b INT) USING json")
+        sql("ALTER VIEW `v`  AS SELECT * FROM t")


the test should check the error message, before altering the view...

incompatible schema issue address comments code clean test example code clean

SparkQA · 2021-06-30T18:23:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44970/

SparkQA · 2021-06-30T18:26:22Z

Test build #140455 has finished for PR 32831 at commit cae2be9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-30T18:59:40Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44970/

cloud-fan · 2021-07-01T03:19:57Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

@@ -846,6 +846,30 @@ class SessionCatalog(
    case None => fromCatalogTable(viewInfo.tableMeta, isTempView = true)
  }

+  private def buildViewDDL(metadata: CatalogTable, isTempView: Boolean): String = {
+    val isGlobalTemp = metadata.identifier.database.exists(_ == globalTempViewManager.database)
+    val viewType = if (isTempView && isGlobalTemp) {


I think this only applies to the permanent view? We don't store temp view in the cataog.

Ah, sounds good. this will make the code simpler.

SparkQA · 2021-07-01T09:14:48Z

Test build #140500 has finished for PR 32831 at commit 2fede50.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-01T09:32:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45010/

cloud-fan · 2021-07-01T09:45:10Z

thanks, merging to master!

cloud-fan · 2021-07-01T09:48:11Z

sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala

@@ -910,4 +911,27 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
      }
    }
  }
+
+  test("SPARK-35685: Prompt recreating view message for schema mismatch") {


oh I missed it . It's better to move this test to PersistedViewTestSuite

You can fix it in #32832

SparkQA · 2021-07-01T10:06:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45010/

… view ### What changes were proposed in this pull request? As described in #32831, Spark has compatible issues when querying a view created by an older version. The root cause is that Spark changed the auto-generated alias name. To avoid this in the future, we could ask the user to specify explicit column names when creating a view. ### Why are the changes needed? Avoid compatible issue when querying a view ### Does this PR introduce _any_ user-facing change? Yes. User will get error when running query below after this change ``` CREATE OR REPLACE VIEW v AS SELECT CAST(t.a AS INT), to_date(t.b, 'yyyyMMdd') FROM t ``` ### How was this patch tested? not yet Closes #32832 from linhongliu-db/SPARK-35686-no-auto-alias. Authored-by: Linhong Liu <linhong.liu@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

github-actions bot added the SQL label Jun 9, 2021

linhongliu-db mentioned this pull request Jun 9, 2021

[SPARK-35686][SQL] Not allow using auto-generated alias when creating view #32832

Closed

linhongliu-db marked this pull request as draft June 9, 2021 03:55

linhongliu-db commented Jun 21, 2021

View reviewed changes

cloud-fan reviewed Jun 21, 2021

View reviewed changes

cloud-fan reviewed Jun 24, 2021

View reviewed changes

[SPARK-35685][SQL] Prompt recreating the view when there is an

de7e88d

incompatible schema issue address comments code clean test example code clean

linhongliu-db changed the title ~~[WIP][SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue~~ [SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue Jun 30, 2021

linhongliu-db force-pushed the SPARK-35685-view-compatible branch from f9c4e9d to cae2be9 Compare June 30, 2021 16:53

linhongliu-db marked this pull request as ready for review June 30, 2021 16:54

cloud-fan reviewed Jul 1, 2021

View reviewed changes

linhongliu-db added 2 commits July 1, 2021 13:44

test case

dd15200

fix test

2fede50

linhongliu-db force-pushed the SPARK-35685-view-compatible branch from cae2be9 to 2fede50 Compare July 1, 2021 07:24

cloud-fan approved these changes Jul 1, 2021

View reviewed changes

cloud-fan closed this in 0c34b96 Jul 1, 2021

cloud-fan reviewed Jul 1, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue #32831

[SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue #32831

linhongliu-db commented Jun 9, 2021 •

edited

SparkQA commented Jun 9, 2021

SparkQA commented Jun 9, 2021

SparkQA commented Jun 9, 2021

linhongliu-db Jun 21, 2021

cloud-fan Jun 21, 2021

SparkQA commented Jun 24, 2021

SparkQA commented Jun 24, 2021

SparkQA commented Jun 24, 2021

cloud-fan Jun 24, 2021 •

edited

SparkQA commented Jun 30, 2021

SparkQA commented Jun 30, 2021

SparkQA commented Jun 30, 2021

cloud-fan Jul 1, 2021

linhongliu-db Jul 1, 2021

SparkQA commented Jul 1, 2021

SparkQA commented Jul 1, 2021

cloud-fan commented Jul 1, 2021

cloud-fan Jul 1, 2021

SparkQA commented Jul 1, 2021

[SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue #32831

[SPARK-35685][SQL] Prompt recreating the view when there is an incompatible schema issue #32831

Conversation

linhongliu-db commented Jun 9, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jun 9, 2021

SparkQA commented Jun 9, 2021

SparkQA commented Jun 9, 2021

linhongliu-db Jun 21, 2021

Choose a reason for hiding this comment

cloud-fan Jun 21, 2021

Choose a reason for hiding this comment

SparkQA commented Jun 24, 2021

SparkQA commented Jun 24, 2021

SparkQA commented Jun 24, 2021

cloud-fan Jun 24, 2021 • edited

Choose a reason for hiding this comment

SparkQA commented Jun 30, 2021

SparkQA commented Jun 30, 2021

SparkQA commented Jun 30, 2021

cloud-fan Jul 1, 2021

Choose a reason for hiding this comment

linhongliu-db Jul 1, 2021

Choose a reason for hiding this comment

SparkQA commented Jul 1, 2021

SparkQA commented Jul 1, 2021

cloud-fan commented Jul 1, 2021

cloud-fan Jul 1, 2021

Choose a reason for hiding this comment

SparkQA commented Jul 1, 2021

linhongliu-db commented Jun 9, 2021 •

edited

cloud-fan Jun 24, 2021 •

edited