[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

yaooqinn · 2021-07-19T17:18:42Z

What changes were proposed in this pull request?

This fixes a case sensitivity issue for desc table commands with partition specified.

Why are the changes needed?

bugfix

Does this PR introduce any user-facing change?

yes, but it's a bugfix

How was this patch tested?

new tests

before

+-- !query
+DESC EXTENDED t PARTITION (C='Us', D=1)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+Partition spec is invalid. The spec (C, D) must match the partition spec (c, d) defined in table '`default`.`t`'
+

after

https://github.com/apache/spark/pull/33424/files#diff-554189c49950974a948f99fa9b7436f615052511660c6a0ae3062fa8ca0a327cR328

…onSpec

yaooqinn · 2021-07-19T17:19:16Z

cc @cloud-fan @maropu thanks very much

SparkQA · 2021-07-19T17:58:24Z

Test build #141268 has finished for PR 33424 at commit afa5539.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-19T18:47:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45783/

SparkQA · 2021-07-19T19:21:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45783/

yaooqinn · 2021-07-20T01:07:07Z

retest this please

SparkQA · 2021-07-20T01:53:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45802/

SparkQA · 2021-07-20T02:26:57Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45802/

cloud-fan · 2021-07-20T03:24:33Z

sql/core/src/test/resources/sql-tests/results/describe.sql.out

@@ -324,6 +324,37 @@ Location [not included in comparison]/{warehouse_dir}/t
 Storage Properties  	[a=1, b=2]


+-- !query
+DESC EXTENDED t PARTITION (C='Us', D=1)


what was the result before this PR?

+-- !query +DESC EXTENDED t PARTITION (C='Us', D=1) +-- !query schema +struct<> +-- !query output +org.apache.spark.sql.AnalysisException +Partition spec is invalid. The spec (C, D) must match the partition spec (c, d) defined in table '`default`.`t`' +

SparkQA · 2021-07-20T05:45:45Z

Test build #141288 has finished for PR 33424 at commit afa5539.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-07-20T08:22:33Z

@yaooqinn, the image in the JIRA is broken, and the PR doesn't explain what bug it fixes. Can you describe it before/after the fix?

yaooqinn · 2021-07-20T08:28:10Z

thanks, @HyukjinKwon, I updated the PR desc and also fixed the broken img on JIRA side

yaooqinn · 2021-07-21T02:59:15Z

retest this please

SparkQA · 2021-07-21T04:16:03Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45893/

SparkQA · 2021-07-21T10:28:46Z

Test build #141377 has finished for PR 33424 at commit afa5539.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-21T13:15:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45927/

SparkQA · 2021-07-21T13:56:24Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45927/

SparkQA · 2021-07-21T16:48:08Z

Test build #141410 has finished for PR 33424 at commit ce08ba4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public static class AppShuffleInfo
class ContinuousWriteRDD(var prev: RDD[InternalRow], writerFactory: StreamingDataWriterFactory,
case class WriteToContinuousDataSource(write: StreamingWrite, query: LogicalPlan,
case class WriteToContinuousDataSourceExec(write: StreamingWrite, query: SparkPlan,

… with PartitionSpec ### What changes were proposed in this pull request? This fixes a case sensitivity issue for desc table commands with partition specified. ### Why are the changes needed? bugfix ### Does this PR introduce _any_ user-facing change? yes, but it's a bugfix ### How was this patch tested? new tests #### before ``` +-- !query +DESC EXTENDED t PARTITION (C='Us', D=1) +-- !query schema +struct<> +-- !query output +org.apache.spark.sql.AnalysisException +Partition spec is invalid. The spec (C, D) must match the partition spec (c, d) defined in table '`default`.`t`' + ``` #### after https://github.com/apache/spark/pull/33424/files#diff-554189c49950974a948f99fa9b7436f615052511660c6a0ae3062fa8ca0a327cR328 Closes #33424 from yaooqinn/SPARK-36213. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit 4cd6cfc) Signed-off-by: Kent Yao <yao@apache.org>

yaooqinn · 2021-07-21T16:56:03Z

thanks, merged to master/3.2/3.1/3.0

… with PartitionSpec ### What changes were proposed in this pull request? This fixes a case sensitivity issue for desc table commands with partition specified. ### Why are the changes needed? bugfix ### Does this PR introduce _any_ user-facing change? yes, but it's a bugfix ### How was this patch tested? new tests #### before ``` +-- !query +DESC EXTENDED t PARTITION (C='Us', D=1) +-- !query schema +struct<> +-- !query output +org.apache.spark.sql.AnalysisException +Partition spec is invalid. The spec (C, D) must match the partition spec (c, d) defined in table '`default`.`t`' + ``` #### after https://github.com/apache/spark/pull/33424/files#diff-554189c49950974a948f99fa9b7436f615052511660c6a0ae3062fa8ca0a327cR328 Closes apache#33424 from yaooqinn/SPARK-36213. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit 4cd6cfc) Signed-off-by: Kent Yao <yao@apache.org>

[SPARK-36213][SQL] Normalize PartitionSpec for DescTable with Partiti…

afa5539

…onSpec

github-actions bot added the SQL label Jul 19, 2021

cloud-fan reviewed Jul 20, 2021

View reviewed changes

cloud-fan approved these changes Jul 20, 2021

View reviewed changes

yaooqinn added 2 commits July 21, 2021 19:43

Merge branch 'master' into SPARK-36213

92ba197

Merge branch 'master' of github.com:apache/spark into SPARK-36213

ce08ba4

yaooqinn closed this in 4cd6cfc Jul 21, 2021

yaooqinn deleted the SPARK-36213 branch July 21, 2021 16:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

yaooqinn commented Jul 19, 2021 •

edited

yaooqinn commented Jul 19, 2021

SparkQA commented Jul 19, 2021

SparkQA commented Jul 19, 2021

SparkQA commented Jul 19, 2021

yaooqinn commented Jul 20, 2021

SparkQA commented Jul 20, 2021

SparkQA commented Jul 20, 2021

cloud-fan Jul 20, 2021

yaooqinn Jul 20, 2021

SparkQA commented Jul 20, 2021

HyukjinKwon commented Jul 20, 2021

yaooqinn commented Jul 20, 2021

yaooqinn commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

yaooqinn commented Jul 21, 2021

[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

[SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec #33424

Conversation

yaooqinn commented Jul 19, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

before

after

yaooqinn commented Jul 19, 2021

SparkQA commented Jul 19, 2021

SparkQA commented Jul 19, 2021

SparkQA commented Jul 19, 2021

yaooqinn commented Jul 20, 2021

SparkQA commented Jul 20, 2021

SparkQA commented Jul 20, 2021

cloud-fan Jul 20, 2021

Choose a reason for hiding this comment

yaooqinn Jul 20, 2021

Choose a reason for hiding this comment

SparkQA commented Jul 20, 2021

HyukjinKwon commented Jul 20, 2021

yaooqinn commented Jul 20, 2021

yaooqinn commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

SparkQA commented Jul 21, 2021

yaooqinn commented Jul 21, 2021

yaooqinn commented Jul 19, 2021 •

edited