New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-7181][CORE]fix inifite loop in Externalsorter's mergeWithAggregation #5737
Conversation
Test build #31123 has finished for PR 5737 at commit
|
LGTM. CC @mateiz for a quick look, as the original author. |
Hm, did we just never run into that else case? LGTM. |
|
||
// avoid combine before spill | ||
sorter.insertAll((0 until 50000).iterator.map(i => (i , 2 * i))) | ||
sorter.insertAll((0 until 50000).iterator.map(i => (i, 2 * i + 1))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this actually run the case that you modified? (i.e. did this modified test fail before your fix?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it failed on my computer(this testcase didn't stop). I set the memory limit to a small value on my computer to ganrantee spills. I think we can also set spark.shuffle.memoryFraction
to a small value to do this. The origin value of spark.shuffle.memoryFraction
is 0.001, I don't know if it can ganrantee spills. If not, we should change it to a smaller value.
…regation see [SPARK-7181](https://issues.apache.org/jira/browse/SPARK-7181). Author: Qiping Li <liqiping1991@gmail.com> Closes #5737 from chouqin/externalsorter and squashes the following commits: 2924b93 [Qiping Li] fix inifite loop in Externalsorter's mergeWithAggregation (cherry picked from commit 7f4b583) Signed-off-by: Sean Owen <sowen@cloudera.com>
…regation see [SPARK-7181](https://issues.apache.org/jira/browse/SPARK-7181). Author: Qiping Li <liqiping1991@gmail.com> Closes #5737 from chouqin/externalsorter and squashes the following commits: 2924b93 [Qiping Li] fix inifite loop in Externalsorter's mergeWithAggregation (cherry picked from commit 7f4b583) Signed-off-by: Sean Owen <sowen@cloudera.com>
…regation see [SPARK-7181](https://issues.apache.org/jira/browse/SPARK-7181). Author: Qiping Li <liqiping1991@gmail.com> Closes apache#5737 from chouqin/externalsorter and squashes the following commits: 2924b93 [Qiping Li] fix inifite loop in Externalsorter's mergeWithAggregation
…regation see [SPARK-7181](https://issues.apache.org/jira/browse/SPARK-7181). Author: Qiping Li <liqiping1991@gmail.com> Closes apache#5737 from chouqin/externalsorter and squashes the following commits: 2924b93 [Qiping Li] fix inifite loop in Externalsorter's mergeWithAggregation
…regation see [SPARK-7181](https://issues.apache.org/jira/browse/SPARK-7181). Author: Qiping Li <liqiping1991@gmail.com> Closes apache#5737 from chouqin/externalsorter and squashes the following commits: 2924b93 [Qiping Li] fix inifite loop in Externalsorter's mergeWithAggregation
see SPARK-7181.