New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-15453] [SQL] [Follow-up] FileSourceScanExec to extract outputOrdering
information
#16994
Conversation
@@ -240,6 +240,7 @@ class BucketedReadSuite extends QueryTest with SQLTestUtils with TestHiveSinglet | |||
joinCondition: (DataFrame, DataFrame) => Column, | |||
shuffleLeft: Boolean, | |||
shuffleRight: Boolean, | |||
numPartitions: Int = 10, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since other vars are for left and right side, wondering if same could to be done for numPartitions
. ie. numPartitionsLeft
and numPartitionsRight
.
That could make the new test you added more interesting :
numPartitionsLeft = 1,
numPartitionsRight = 10,
sortLeft = false,
sortRight = true
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, let me do it.
Test build #73135 has finished for PR 16994 at commit
|
Test build #73140 has finished for PR 16994 at commit
|
Test build #73142 has finished for PR 16994 at commit
|
LGTM |
Thanks! Merging to master. |
…dering` information ### What changes were proposed in this pull request? `outputOrdering` is also dependent on whether the bucket has more than one files. The test cases fail when we try to move them to sql/core. This PR is to fix the test cases introduced in apache#14864 and add a test case to verify [the related logics](https://github.com/tejasapatil/spark/blob/070c24994747c0479fb2520774ede27ff1cf8cac/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L197-L206). ### How was this patch tested? N/A Author: Xiao Li <gatorsmile@gmail.com> Closes apache#16994 from gatorsmile/bucketingTS.
What changes were proposed in this pull request?
outputOrdering
is also dependent on whether the bucket has more than one files. The test cases fail when we try to move them to sql/core. This PR is to fix the test cases introduced in #14864 and add a test case to verify the related logics.How was this patch tested?
N/A