-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-8794][SQL] Make PrunedScan work for Sample #7228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #36549 has finished for PR 7228 at commit
|
|
I am not sure if I understand correctly, probably a better place to prune the column or push down the filter is in the Optimzier. |
|
agree with @chenghao-intel , currently we push down project through Union, Aggregate, Join, Limit and Sort, and Sample should be included too. |
|
@chenghao-intel @cloud-fan you are right. I will update this later. Thanks. |
|
Test build #36579 has finished for PR 7228 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be better to put this test in catalyst. For example, FilterPushdownSuite.
|
Test build #36640 has finished for PR 7228 at commit
|
|
Thanks! Merged to master. |
JIRA: https://issues.apache.org/jira/browse/SPARK-8794
Currently
PrunedScanworks only when followed by project or filter operations. However, even if there is aSamplebetween these operations andPrunedScan,PrunedScanshould work too.