Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

Closed
wants to merge 5 commits into from
Closed

[SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project #27517

wants to merge 5 commits into from

Commits on Feb 10, 2020

  1. [SPARK-29721][SQL] Prune unnecessary nested fields from Generate with… 

    …out Project

### What changes were proposed in this pull request?

This patch proposes to prune unnecessary nested fields from Generate which has no Project on top of it.

### Why are the changes needed?

In Optimizer, we can prune nested columns from Project(projectList, Generate). However, unnecessary columns could still possibly be read in Generate, if no Project on top of it. We should prune it too.

### Does this PR introduce any user-facing change?

No

### How was this patch tested?

Unit test.

Closes apache#26978 from viirya/SPARK-29721.

Lead-authored-by: Liang-Chi Hsieh <liangchi@uber.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
    @viirya
    viirya and viirya committed Feb 10, 2020
    Configuration menu
    Copy the full SHA
    14dba8b View commit details
    Browse the repository at this point in the history

  2. For required child outputs, we don't want to prune it from Generate.

    @viirya
    viirya committed Feb 10, 2020
    Configuration menu
    Copy the full SHA
    f622d53 View commit details
    Browse the repository at this point in the history

  3. Address comments and combine new test.

    @viirya
    viirya committed Feb 10, 2020
    Configuration menu
    Copy the full SHA
    0383592 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2020

  1. Merge remote-tracking branch 'upstream/master' into SPARK-29721-2

    @viirya
    viirya committed Feb 29, 2020
    Configuration menu
    Copy the full SHA
    3928eb0 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2020

  1. Add more tests.

    @viirya
    viirya committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    289355f View commit details
    Browse the repository at this point in the history