[SPARK-12705] [SQL] push missing attributes for Sort #11153
Conversation
|
cc @marmbrus |
|
Test build #51040 has finished for PR 11153 at commit
|
davies
changed the title
|
Test build #2533 has finished for PR 11153 at commit
|
|
cc @gatorsmile |
|
Sorry, my previous fix does not cover all the scenarios. Will read your fix tomorrow. Thanks! |
|
Test build #51085 has finished for PR 11153 at commit
|
| case other => (Seq.empty[SortOrder], Seq.empty[Attribute]) | ||
| val newAggregateExpressions = a.aggregateExpressions ++ missingAttrs | ||
| a.copy(aggregateExpressions = newAggregateExpressions) | ||
| case u: UnaryNode => |
There was a problem hiding this comment.
It sounds like we want to cover all the UnaryNode here? This is different from what we discussed. I am fine if you want to add more supports (e.g., crossing the boundary of subquery), but we might need to add more test cases to ensure it does not break anything.
There was a problem hiding this comment.
If needed, I can add more test cases after this is merged, since it could be time-consuming. Please feel free to let me know.
There was a problem hiding this comment.
Will exclude Subquery, feel free to add more tests.
|
Test build #51124 has finished for PR 11153 at commit
|
|
LGTM |
|
cc @cloud-fan for review too |
|
LGTM except one comment |
|
Merging this into master, thanks! |
The current implementation of ResolveSortReferences can only push one missing attributes into it's child, it failed to analyze TPCDS Q98, because of there are two missing attributes in that (one from Window, another from Aggregate).