[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

viirya · 2020-02-10T05:21:57Z

What changes were proposed in this pull request?

This patch proposes to prune unnecessary nested fields from Generate which has no Project on top of it.

Why are the changes needed?

In Optimizer, we can prune nested columns from Project(projectList, Generate). However, unnecessary columns could still possibly be read in Generate, if no Project on top of it. We should prune it too.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test.

…out Project ### What changes were proposed in this pull request? This patch proposes to prune unnecessary nested fields from Generate which has no Project on top of it. ### Why are the changes needed? In Optimizer, we can prune nested columns from Project(projectList, Generate). However, unnecessary columns could still possibly be read in Generate, if no Project on top of it. We should prune it too. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Unit test. Closes apache#26978 from viirya/SPARK-29721. Lead-authored-by: Liang-Chi Hsieh <liangchi@uber.com> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

viirya · 2020-02-10T05:23:58Z

This is re-submitssion of SPARK-29721. I think original patch did not consider a corner case at beginning. The original patch was reverted. This patch adds the corner case.

I will also go to add more tests here.

cc @dongjoon-hyun

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala

HyukjinKwon · 2020-02-10T06:55:39Z

The fix seems good to me.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala

SparkQA · 2020-02-10T08:05:01Z

Test build #118117 has finished for PR 27517 at commit f622d53.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-02-10T08:06:15Z

retest this please

SparkQA · 2020-02-10T10:10:24Z

Test build #118132 has finished for PR 27517 at commit f622d53.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-02-10T19:36:09Z

Thank you for working on this again, but this is now 3.1.0. Please don't land this to branch-3.0.

dongjoon-hyun · 2020-02-10T19:37:28Z

BTW, is this including @gatorsmile 's test case on the previous PR?

viirya · 2020-02-10T20:51:42Z

@dongjoon-hyun Yes, this is for 3.1.0 now. The new test case was added here too.

viirya · 2020-02-10T20:52:00Z

retest this please

SparkQA · 2020-02-11T00:25:18Z

Test build #118188 has finished for PR 27517 at commit 0383592.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-02-11T00:59:08Z

retest this please

SparkQA · 2020-02-11T01:33:36Z

Test build #118180 has finished for PR 27517 at commit f622d53.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-11T06:14:29Z

Test build #118192 has finished for PR 27517 at commit 0383592.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-02-12T06:03:52Z

Got it. Thank you for updating.

SparkQA · 2020-02-29T06:34:09Z

Test build #119110 has finished for PR 27517 at commit 3928eb0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-03-05T08:58:07Z

@dongjoon-hyun @HyukjinKwon @cloud-fan More tests are added now. Please help take another look when you have time. Thanks!

SparkQA · 2020-03-05T13:50:58Z

Test build #119378 has finished for PR 27517 at commit 289355f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-03-14T06:23:02Z

retest this please

SparkQA · 2020-03-14T07:05:02Z

Test build #119786 has finished for PR 27517 at commit 289355f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-03-14T07:14:05Z

retest this please

SparkQA · 2020-03-14T11:32:54Z

Test build #119789 has finished for PR 27517 at commit 289355f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-03-16T08:03:23Z

@dongjoon-hyun @HyukjinKwon @cloud-fan Please take another look. Thanks!

viirya · 2020-03-22T06:23:48Z

retest this please

SparkQA · 2020-03-22T07:05:02Z

Test build #120152 has finished for PR 27517 at commit 289355f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-03-22T07:08:49Z

retest this please

SparkQA · 2020-03-22T11:43:22Z

Test build #120159 has finished for PR 27517 at commit 289355f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-03-27T07:55:57Z

Retest this please.

SparkQA · 2020-03-27T13:14:15Z

Test build #120471 has finished for PR 27517 at commit 289355f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM. Thank you, @viirya and @HyukjinKwon .
Merged to master for Apache Spark 3.1.0.

cc @dbtsai and @holdenk

viirya · 2020-03-27T18:32:22Z

Thanks!

…out Project ### What changes were proposed in this pull request? This patch proposes to prune unnecessary nested fields from Generate which has no Project on top of it. ### Why are the changes needed? In Optimizer, we can prune nested columns from Project(projectList, Generate). However, unnecessary columns could still possibly be read in Generate, if no Project on top of it. We should prune it too. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Unit test. Closes apache#27517 from viirya/SPARK-29721-2. Lead-authored-by: Liang-Chi Hsieh <liangchi@uber.com> Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

viirya and others added 2 commits February 9, 2020 21:20

For required child outputs, we don't want to prune it from Generate.

f622d53

HyukjinKwon reviewed Feb 10, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 10, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 10, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala Outdated Show resolved Hide resolved

dongjoon-hyun added the SQL label Feb 10, 2020

Address comments and combine new test.

0383592

viirya force-pushed the SPARK-29721-2 branch from ec1ae43 to 0383592 Compare February 10, 2020 23:22

HyukjinKwon mentioned this pull request Feb 27, 2020

[SPARK-30955][SQL] Exclude Generate output when aliasing in nested column pruning #27702

Closed

Merge remote-tracking branch 'upstream/master' into SPARK-29721-2

3928eb0

Add more tests.

289355f

viirya force-pushed the SPARK-29721-2 branch from 8da952b to 289355f Compare March 5, 2020 08:56

viirya changed the title ~~[WIP][SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project~~ [SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project Mar 5, 2020

dongjoon-hyun approved these changes Mar 27, 2020

View reviewed changes

dongjoon-hyun closed this in aa8776b Mar 27, 2020

viirya deleted the SPARK-29721-2 branch December 27, 2023 18:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

viirya commented Feb 10, 2020

viirya commented Feb 10, 2020

HyukjinKwon commented Feb 10, 2020

SparkQA commented Feb 10, 2020

HyukjinKwon commented Feb 10, 2020

SparkQA commented Feb 10, 2020

dongjoon-hyun commented Feb 10, 2020 •

edited

Loading

dongjoon-hyun commented Feb 10, 2020

viirya commented Feb 10, 2020

viirya commented Feb 10, 2020

SparkQA commented Feb 11, 2020

viirya commented Feb 11, 2020

SparkQA commented Feb 11, 2020

SparkQA commented Feb 11, 2020

dongjoon-hyun commented Feb 12, 2020

SparkQA commented Feb 29, 2020

viirya commented Mar 5, 2020

SparkQA commented Mar 5, 2020

viirya commented Mar 14, 2020

SparkQA commented Mar 14, 2020

maropu commented Mar 14, 2020

SparkQA commented Mar 14, 2020

viirya commented Mar 16, 2020

viirya commented Mar 22, 2020

SparkQA commented Mar 22, 2020

viirya commented Mar 22, 2020

SparkQA commented Mar 22, 2020

dongjoon-hyun commented Mar 27, 2020

SparkQA commented Mar 27, 2020

dongjoon-hyun left a comment

viirya commented Mar 27, 2020

[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

Conversation

viirya commented Feb 10, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

viirya commented Feb 10, 2020

HyukjinKwon commented Feb 10, 2020

SparkQA commented Feb 10, 2020

HyukjinKwon commented Feb 10, 2020

SparkQA commented Feb 10, 2020

dongjoon-hyun commented Feb 10, 2020 • edited Loading

dongjoon-hyun commented Feb 10, 2020

viirya commented Feb 10, 2020

viirya commented Feb 10, 2020

SparkQA commented Feb 11, 2020

viirya commented Feb 11, 2020

SparkQA commented Feb 11, 2020

SparkQA commented Feb 11, 2020

dongjoon-hyun commented Feb 12, 2020

SparkQA commented Feb 29, 2020

viirya commented Mar 5, 2020

SparkQA commented Mar 5, 2020

viirya commented Mar 14, 2020

SparkQA commented Mar 14, 2020

maropu commented Mar 14, 2020

SparkQA commented Mar 14, 2020

viirya commented Mar 16, 2020

viirya commented Mar 22, 2020

SparkQA commented Mar 22, 2020

viirya commented Mar 22, 2020

SparkQA commented Mar 22, 2020

dongjoon-hyun commented Mar 27, 2020

SparkQA commented Mar 27, 2020

dongjoon-hyun left a comment

Choose a reason for hiding this comment

viirya commented Mar 27, 2020

dongjoon-hyun commented Feb 10, 2020 •

edited

Loading