New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-1021] Defer the data-driven computation of partition bounds in so... #3079
Conversation
Reboot of #1689 |
Test build #22828 has started for PR 3079 at commit
|
@marmbrus, @scwf, FWIW, the Not sure why, but running |
@erikerlandson I think you also need -Phive for the tests to run. It is possible some other things changed (or even that that test case changed with the upgrade to hive 13). Perhaps you can include a dummy change to the Hive code to make sure those tests are run in this PR so we can see what Jenkins thinks? |
Test build #22828 has finished for PR 3079 at commit
|
Test FAILed. |
019ac27
to
0fc30fe
Compare
Test build #22880 has started for PR 3079 at commit
|
Test build #22880 has finished for PR 3079 at commit
|
Test FAILed. |
0fc30fe
to
2183325
Compare
Test build #22892 has started for PR 3079 at commit
|
Test build #22892 has finished for PR 3079 at commit
|
Test FAILed. |
@@ -113,8 +117,12 @@ class RangePartitioner[K : Ordering : ClassTag, V]( | |||
private var ordering = implicitly[Ordering[K]] | |||
|
|||
// An array of upper bounds for the first (partitions - 1) partitions | |||
private var rangeBounds: Array[K] = { | |||
if (partitions <= 1) { | |||
@volatile private var valRB: Array[K] = null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
valRD
is a kinda confusing name. I think the convention would be to name it _rangeBounds
. Eg.
For reference, this other issue has some overlap: |
hi @erikerlandson, thanks for working on this. It would be great to have a solution to this long running problem. Since it looks like there is still some work to be done, I propose we close this issue for now. We are on a renewed mission to keep the PR queue small and limited to active PRs so that things don't fall through the cracks. |
Do please reopen though once you having something that is passing tests :) |
...rtByKey() until evaluation.