Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-13254][SQL] Fix planning of TakeOrderedAndProject operator #11145

Closed

Conversation

JoshRosen
Copy link
Contributor

The patch for SPARK-8964 ("use Exchange to perform shuffle in Limit" / #7334) inadvertently broke the planning of the TakeOrderedAndProject operator: because ReturnAnswer was the new root of the query plan, the TakeOrderedAndProject rule was unable to match before BasicOperators.

This patch fixes this by moving the TakeOrderedAndCollect and CollectLimit rules into the same strategy.

In addition, I made changes to the TakeOrderedAndProject operator in order to make its doExecute() method lazy and added a new TakeOrderedAndProjectSuite which tests the new code path.

/cc @davies and @marmbrus for review.

@davies
Copy link
Contributor

davies commented Feb 9, 2016

LGTM, pending tests.

@SparkQA
Copy link

SparkQA commented Feb 10, 2016

Test build #51008 has finished for PR 11145 at commit 2ccf9ed.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 10, 2016

Test build #51013 has finished for PR 11145 at commit 5cd4aa4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@JoshRosen
Copy link
Contributor Author

Alright, merging this to master.

@asfgit asfgit closed this in 5cf2059 Feb 10, 2016
@JoshRosen JoshRosen deleted the take-ordered-and-project-fix branch August 29, 2016 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants