Skip to content

Conversation

@viirya
Copy link
Member

@viirya viirya commented Jul 5, 2015

JIRA: https://issues.apache.org/jira/browse/SPARK-8794

Currently PrunedScan works only when followed by project or filter operations. However, even if there is a Sample between these operations and PrunedScan, PrunedScan should work too.

@SparkQA
Copy link

SparkQA commented Jul 5, 2015

Test build #36549 has finished for PR 7228 at commit df82785.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@chenghao-intel
Copy link
Contributor

I am not sure if I understand correctly, probably a better place to prune the column or push down the filter is in the Optimzier.

@cloud-fan
Copy link
Contributor

agree with @chenghao-intel , currently we push down project through Union, Aggregate, Join, Limit and Sort, and Sample should be included too.

@viirya
Copy link
Member Author

viirya commented Jul 6, 2015

@chenghao-intel @cloud-fan you are right. I will update this later. Thanks.

@SparkQA
Copy link

SparkQA commented Jul 6, 2015

Test build #36579 has finished for PR 7228 at commit 7e4ba76.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to put this test in catalyst. For example, FilterPushdownSuite.

@SparkQA
Copy link

SparkQA commented Jul 7, 2015

Test build #36640 has finished for PR 7228 at commit ede7cd8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Jul 7, 2015

Thanks! Merged to master.

@asfgit asfgit closed this in da56c4e Jul 7, 2015
@viirya viirya deleted the sample_prunedscan branch December 27, 2023 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants