[SPARK-37577][SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning #34845

viirya · 2021-12-09T08:52:59Z

What changes were proposed in this pull request?

This patch fixes a bug in nested column pruning for Generate. The bug happens after we replace extractor's reference attribute with generator's input, then when we try to replace a top extractor (e.g. GetStructField), if the replaced child expression is array type, extractFieldName will fail by throwing ClassCastException.

Why are the changes needed?

In particular, when we transform the extractor and replace with generator's input, we cannot simply do a transform up because because if we replace the attribute, extractFieldName could cause ClassCastException error. We need to get the field name before replacing down the attribute/other extractor.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added test.

SparkQA · 2021-12-09T09:41:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50508/

SparkQA · 2021-12-09T10:42:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50508/

viirya · 2021-12-09T21:07:44Z

cc @cloud-fan @maropu @dongjoon-hyun

dongjoon-hyun · 2021-12-09T21:09:10Z

Thank you for pinging me, @viirya .

dongjoon-hyun · 2021-12-09T21:11:01Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala

+      StructField("o", ArrayType(StructType(
+        Seq(StructField("s", StringType, false),
+          StructField("b", ArrayType(StructType(
+            Seq(StructField("e", StringType, false))), true), false)


Can we use more intrusive field names? o, s, b, e looks a little confusing.

sure, i will update it.

SparkQA · 2021-12-09T21:46:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50526/

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala

SparkQA · 2021-12-09T22:45:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50526/

SparkQA · 2021-12-10T01:47:19Z

Test build #146051 has finished for PR 34845 at commit d034ea0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-10T02:44:22Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50528/

SparkQA · 2021-12-10T03:44:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50528/

SparkQA · 2021-12-10T05:56:13Z

Test build #146053 has finished for PR 34845 at commit 2f47271.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-12-10T08:36:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala

+        // `extractFieldName` could cause `ClassCastException` error. We need to get the
+        // field name before replacing down the attribute/other extractor.
+        val fieldName = g.extractFieldName
+        val newChild = replaceGenerator(generator, g.child)


Can we do a manual top-down traversal via mapChildren? It's weird to see a recursive call inside transformDown.

Okay, let me rewrite it using mapChildren.

SparkQA · 2021-12-10T21:53:46Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50555/

SparkQA · 2021-12-10T22:54:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50555/

SparkQA · 2021-12-11T01:50:02Z

Test build #146080 has finished for PR 34845 at commit a9862c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM from my side, @viirya . Thank you.

dongjoon-hyun · 2021-12-13T01:52:17Z

Retest this please

SparkQA · 2021-12-13T02:56:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50589/

SparkQA · 2021-12-13T03:41:10Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50589/

SparkQA · 2021-12-13T06:54:22Z

Test build #146114 has finished for PR 34845 at commit a9862c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-12-13T07:17:49Z

Thank you, @viirya , @HyukjinKwon , @cloud-fan . Merged to master.

Could you make a backport to branch-3.2, @viirya ?

viirya · 2021-12-13T07:46:05Z

Thanks. I will make a backport to branch-3.2.

…ast to StructType for Generate Pruning ### What changes were proposed in this pull request? This patch fixes a bug in nested column pruning for Generate. The bug happens after we replace extractor's reference attribute with generator's input, then when we try to replace a top extractor (e.g. `GetStructField`), if the replaced child expression is array type, `extractFieldName` will fail by throwing `ClassCastException`. This is the backport from #34845 to branch-3.2. ### Why are the changes needed? In particular, when we transform the extractor and replace with generator's input, we cannot simply do a transform up because because if we replace the attribute, `extractFieldName` could cause `ClassCastException` error. We need to get the field name before replacing down the attribute/other extractor. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added test. Closes #34885 from viirya/SPARK-37577-3.2. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ast to StructType for Generate Pruning ### What changes were proposed in this pull request? This patch fixes a bug in nested column pruning for Generate. The bug happens after we replace extractor's reference attribute with generator's input, then when we try to replace a top extractor (e.g. `GetStructField`), if the replaced child expression is array type, `extractFieldName` will fail by throwing `ClassCastException`. This is the backport from apache#34845 to branch-3.2. ### Why are the changes needed? In particular, when we transform the extractor and replace with generator's input, we cannot simply do a transform up because because if we replace the attribute, `extractFieldName` could cause `ClassCastException` error. We need to get the field name before replacing down the attribute/other extractor. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added test. Closes apache#34885 from viirya/SPARK-37577-3.2. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added the SQL label Dec 9, 2021

This comment has been minimized.

Sign in to view

Fix ClassCastException: ArrayType cannot be cast to StructType.

d034ea0

viirya force-pushed the SPARK-37577 branch from ab773fb to d034ea0 Compare December 9, 2021 20:27

dongjoon-hyun reviewed Dec 9, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala Show resolved Hide resolved

For review comment.

2f47271

cloud-fan reviewed Dec 10, 2021

View reviewed changes

Use mapChildren instead.

a9862c3

dongjoon-hyun approved these changes Dec 13, 2021

View reviewed changes

HyukjinKwon approved these changes Dec 13, 2021

View reviewed changes

cloud-fan approved these changes Dec 13, 2021

View reviewed changes

dongjoon-hyun closed this in dc0fa1e Dec 13, 2021

viirya mentioned this pull request Dec 13, 2021

[SPARK-37577][SQL][3.2] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning #34885

Closed

viirya deleted the SPARK-37577 branch December 27, 2023 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-37577][SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning #34845

[SPARK-37577][SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning #34845

viirya commented Dec 9, 2021 •

edited

SparkQA commented Dec 9, 2021

SparkQA commented Dec 9, 2021

This comment has been minimized.

viirya commented Dec 9, 2021

dongjoon-hyun commented Dec 9, 2021

dongjoon-hyun Dec 9, 2021

viirya Dec 9, 2021

SparkQA commented Dec 9, 2021

SparkQA commented Dec 9, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

cloud-fan Dec 10, 2021

viirya Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 11, 2021

dongjoon-hyun left a comment

dongjoon-hyun commented Dec 13, 2021

SparkQA commented Dec 13, 2021

SparkQA commented Dec 13, 2021

SparkQA commented Dec 13, 2021

dongjoon-hyun commented Dec 13, 2021

viirya commented Dec 13, 2021

[SPARK-37577][SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning #34845

[SPARK-37577][SQL] Fix ClassCastException: ArrayType cannot be cast to StructType for Generate Pruning #34845

Conversation

viirya commented Dec 9, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Dec 9, 2021

SparkQA commented Dec 9, 2021

This comment has been minimized.

viirya commented Dec 9, 2021

dongjoon-hyun commented Dec 9, 2021

dongjoon-hyun Dec 9, 2021

Choose a reason for hiding this comment

viirya Dec 9, 2021

Choose a reason for hiding this comment

SparkQA commented Dec 9, 2021

SparkQA commented Dec 9, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

cloud-fan Dec 10, 2021

Choose a reason for hiding this comment

viirya Dec 10, 2021

Choose a reason for hiding this comment

SparkQA commented Dec 10, 2021

SparkQA commented Dec 10, 2021

SparkQA commented Dec 11, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Dec 13, 2021

SparkQA commented Dec 13, 2021

SparkQA commented Dec 13, 2021

SparkQA commented Dec 13, 2021

dongjoon-hyun commented Dec 13, 2021

viirya commented Dec 13, 2021

viirya commented Dec 9, 2021 •

edited