[SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases including GetArrayStructField #24599

viirya · 2019-05-14T10:09:29Z

What changes were proposed in this pull request?

NestedColumnAliasing rule covers GetStructField only, currently. It means that some nested field extraction expressions aren't pruned. For example, if only accessing a nested field in an array of struct (GetArrayStructFields), this column isn't pruned.

This patch extends the rule to cover general nested field cases, including GetArrayStructFields.

How was this patch tested?

Added tests.

SparkQA · 2019-05-14T13:14:13Z

Test build #105381 has finished for PR 24599 at commit a351e82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2019-05-14T15:16:06Z

cc @dongjoon-hyun @dbtsai @cloud-fan

viirya · 2019-05-22T01:53:51Z

...alyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala

@@ -25,7 +25,7 @@ import org.apache.spark.sql.catalyst.dsl.plans._
 import org.apache.spark.sql.catalyst.expressions._
 import org.apache.spark.sql.catalyst.plans.logical._
 import org.apache.spark.sql.catalyst.rules.RuleExecutor
-import org.apache.spark.sql.types.{StringType, StructType}
+import org.apache.spark.sql.types.{StringType, StructField, StructType}

 class NestedColumnAliasingSuite extends SchemaPruningTest {


There still are many usage of GetStructField in this test suite. Maybe make a minor PR to rewrite them.

dongjoon-hyun · 2019-06-09T01:51:48Z

Retest this please.

SparkQA · 2019-06-09T04:54:03Z

Test build #106314 has finished for PR 24599 at commit a351e82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala

dongjoon-hyun

@viirya . Since this is important improvement, could you add a benchmark case to NestedSchemaPruningBenchmark? Also, please enumerate some newly support examples explicitly instead of more nested field cases in the PR description (at least).

viirya · 2019-06-10T08:45:57Z

@dongjoon-hyun Thanks for looking into this.

I will add the benchmark case. The PR title and description were updated.

SparkQA · 2019-06-10T12:02:34Z

Test build #106347 has finished for PR 24599 at commit 6fc422c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

...e/src/test/scala/org/apache/spark/sql/execution/benchmark/NestedSchemaPruningBenchmark.scala

SparkQA · 2019-06-10T18:26:06Z

Test build #106353 has finished for PR 24599 at commit fb821c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-06-11T03:18:30Z

Test build #106366 has finished for PR 24599 at commit 4c8d371.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

...e/src/test/scala/org/apache/spark/sql/execution/benchmark/NestedSchemaPruningBenchmark.scala

…tract-value

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

SparkQA · 2019-06-11T10:52:58Z

Test build #106383 has finished for PR 24599 at commit 3075f8c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-06-11T11:32:43Z

Test build #106384 has finished for PR 24599 at commit 64a98fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

~~On the master branch, it seems that there is a regression only in Orc (v1). I verified that Parquet/OrcV2 are consistent in master branch.~~ This PR (@viirya ) is irrelevant to that.
cc @gatorsmile

dongjoon-hyun · 2019-06-11T18:00:55Z

Hi, @viirya . I made a benchmark result PR to you. Could you review and merge that?

EC2 result viirya/spark-1#6

dongjoon-hyun · 2019-06-11T18:01:44Z

...e/src/test/scala/org/apache/spark/sql/execution/benchmark/NestedSchemaPruningBenchmark.scala

@@ -51,7 +55,7 @@ abstract class NestedSchemaPruningBenchmark extends SqlBasedBenchmark {
    withTempPath { dir =>
      val path = dir.getCanonicalPath

-      Seq(1, 2).foreach { i =>
+      Seq(1, 2, 3).foreach { i =>


EC2 result

viirya · 2019-06-12T00:56:31Z

Thanks @dongjoon-hyun! Merged the benchmark results now.

dongjoon-hyun

+1, LGTM. Thank you so much, @viirya .
The last commit is .txt-file only updates about benchmark result.
Merged to master.

SparkQA · 2019-06-12T04:04:49Z

Test build #106398 has finished for PR 24599 at commit 3aab8bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…d cases including GetArrayStructField ## What changes were proposed in this pull request? `NestedColumnAliasing` rule covers `GetStructField` only, currently. It means that some nested field extraction expressions aren't pruned. For example, if only accessing a nested field in an array of struct (`GetArrayStructFields`), this column isn't pruned. This patch extends the rule to cover general nested field cases, including `GetArrayStructFields`. ## How was this patch tested? Added tests. Closes apache#24599 from viirya/nested-pruning-extract-value. Lead-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

Extend NestedColumnAliasing.

a351e82

viirya commented May 22, 2019

View reviewed changes

dongjoon-hyun reviewed Jun 9, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala Show resolved Hide resolved

dongjoon-hyun reviewed Jun 9, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala Show resolved Hide resolved

dongjoon-hyun reviewed Jun 9, 2019

View reviewed changes

viirya changed the title ~~[SPARK-27701][SQL] Extend NestedColumnAliasing to more nested field cases~~ [SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases like GetArrayStructField and chain of ExtractValue Jun 10, 2019

viirya changed the title ~~[SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases like GetArrayStructField and chain of ExtractValue~~ [SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases including GetArrayStructField Jun 10, 2019

Address comment.

6fc422c

Add benchmark case.

fb821c2

viirya commented Jun 10, 2019

View reviewed changes

...e/src/test/scala/org/apache/spark/sql/execution/benchmark/NestedSchemaPruningBenchmark.scala Show resolved Hide resolved

Run all tests with nestedSchemaPruning as true.

4c8d371

dongjoon-hyun reviewed Jun 11, 2019

View reviewed changes

...e/src/test/scala/org/apache/spark/sql/execution/benchmark/NestedSchemaPruningBenchmark.scala Show resolved Hide resolved

viirya added 2 commits June 11, 2019 14:44

Update benchmark.

c8b6f6b

Merge remote-tracking branch 'upstream/master' into nested-pruning-ex…

3075f8c

…tract-value

dongjoon-hyun reviewed Jun 11, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated Show resolved Hide resolved

Revert nestedSchemaPruning default value back.

64a98fb

dongjoon-hyun reviewed Jun 11, 2019

View reviewed changes

EC2 result

b4a4754

dongjoon-hyun reviewed Jun 11, 2019

View reviewed changes

Merge pull request #6 from dongjoon-hyun/PR-24599

3aab8bf

EC2 result

dongjoon-hyun approved these changes Jun 12, 2019

View reviewed changes

dongjoon-hyun closed this in 2c9597f Jun 12, 2019

viirya deleted the nested-pruning-extract-value branch December 27, 2023 18:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases including GetArrayStructField #24599

[SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases including GetArrayStructField #24599

viirya commented May 14, 2019 •

edited

Loading

SparkQA commented May 14, 2019

viirya commented May 14, 2019

viirya May 22, 2019

dongjoon-hyun commented Jun 9, 2019

SparkQA commented Jun 9, 2019

dongjoon-hyun left a comment •

edited

Loading

viirya commented Jun 10, 2019

SparkQA commented Jun 10, 2019

SparkQA commented Jun 10, 2019

SparkQA commented Jun 11, 2019

SparkQA commented Jun 11, 2019

SparkQA commented Jun 11, 2019

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Jun 11, 2019

dongjoon-hyun Jun 11, 2019

viirya commented Jun 12, 2019

dongjoon-hyun left a comment

SparkQA commented Jun 12, 2019

[SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases including GetArrayStructField #24599

[SPARK-27701][SQL] Extend NestedColumnAliasing to general nested field cases including GetArrayStructField #24599

Conversation

viirya commented May 14, 2019 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented May 14, 2019

viirya commented May 14, 2019

viirya May 22, 2019

Choose a reason for hiding this comment

dongjoon-hyun commented Jun 9, 2019

SparkQA commented Jun 9, 2019

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

viirya commented Jun 10, 2019

SparkQA commented Jun 10, 2019

SparkQA commented Jun 10, 2019

SparkQA commented Jun 11, 2019

SparkQA commented Jun 11, 2019

SparkQA commented Jun 11, 2019

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Jun 11, 2019

dongjoon-hyun Jun 11, 2019

Choose a reason for hiding this comment

viirya commented Jun 12, 2019

dongjoon-hyun left a comment

Choose a reason for hiding this comment

SparkQA commented Jun 12, 2019

viirya commented May 14, 2019 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading