[SPARK-9709] [SQL] Avoid starving unsafe operators that use sort #8011
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
August 6, 2015 12:05
andrewor14
force-pushed
the
unsafe-starve-memory
branch
from
fdd3c92
to
5d5afdf
Compare
| @@ -124,7 +124,7 @@ private[spark] class ShuffleMemoryManager(maxMemory: Long) extends Logging { | |||
| } | |||
| } | |||
|
|
|||
| private object ShuffleMemoryManager { | |||
| private[spark] object ShuffleMemoryManager { | |||
andrewor14
force-pushed
the
unsafe-starve-memory
branch
from
7c34b09
to
0b07782
Compare
|
High level approach looks good. Would be great to simplify the test case to not rely on memory management. Just use an atomicinteger somewhere. |
andrewor14
force-pushed
the
unsafe-starve-memory
branch
from
db8b6e0
to
35b69a4
Compare
|
Test build #40091 has finished for PR 8011 at commit
|
|
LGTM |
|
retest this please |
|
Test build #40093 timed out for PR 8011 at commit |
|
Test build #40103 has finished for PR 8011 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Aug 7, 2015
The issue is that a task may run multiple sorts, and the sorts run by the child operator (i.e. parent RDD) may acquire all available memory such that other sorts in the same task do not have enough to proceed. This manifests itself in an `IOException("Unable to acquire X bytes of memory")` thrown by `UnsafeExternalSorter`.
The solution is to reserve a page in each sorter in the chain before computing the child operator's (parent RDD's) partitions. This requires us to use a new special RDD that does some preparation before computing the parent's partitions.
Author: Andrew Or <andrew@databricks.com>
Closes #8011 from andrewor14/unsafe-starve-memory and squashes the following commits:
35b69a4 [Andrew Or] Simplify test
0b07782 [Andrew Or] Minor: update comments
5d5afdf [Andrew Or] Merge branch 'master' of github.com:apache/spark into unsafe-starve-memory
254032e [Andrew Or] Add tests
234acbd [Andrew Or] Reserve a page in sorter when preparing each partition
b889e08 [Andrew Or] MapPartitionsWithPreparationRDD
(cherry picked from commit 014a9f9)
Signed-off-by: Reynold Xin <rxin@databricks.com>
|
Test build #40107 has finished for PR 8011 at commit
|
|
Test build #1397 has finished for PR 8011 at commit
|
|
Test build #1396 has finished for PR 8011 at commit
|
|
Test build #40124 timed out for PR 8011 at commit |
asfgit
pushed a commit
that referenced
this pull request
Aug 12, 2015
This is the sister patch to #8011, but for aggregation. In a nutshell: create the `TungstenAggregationIterator` before computing the parent partition. Internally this creates a `BytesToBytesMap` which acquires a page in the constructor as of this patch. This ensures that the aggregation operator is not starved since we reserve at least 1 page in advance. rxin yhuai Author: Andrew Or <andrew@databricks.com> Closes #8038 from andrewor14/unsafe-starve-memory-agg. (cherry picked from commit e011079) Signed-off-by: Reynold Xin <rxin@databricks.com>
asfgit
pushed a commit
that referenced
this pull request
Aug 12, 2015
This is the sister patch to #8011, but for aggregation. In a nutshell: create the `TungstenAggregationIterator` before computing the parent partition. Internally this creates a `BytesToBytesMap` which acquires a page in the constructor as of this patch. This ensures that the aggregation operator is not starved since we reserve at least 1 page in advance. rxin yhuai Author: Andrew Or <andrew@databricks.com> Closes #8038 from andrewor14/unsafe-starve-memory-agg.
CodingCat
pushed a commit
to CodingCat/spark
that referenced
this pull request
Aug 17, 2015
This is the sister patch to apache#8011, but for aggregation. In a nutshell: create the `TungstenAggregationIterator` before computing the parent partition. Internally this creates a `BytesToBytesMap` which acquires a page in the constructor as of this patch. This ensures that the aggregation operator is not starved since we reserve at least 1 page in advance. rxin yhuai Author: Andrew Or <andrew@databricks.com> Closes apache#8038 from andrewor14/unsafe-starve-memory-agg.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The issue is that a task may run multiple sorts, and the sorts run by the child operator (i.e. parent RDD) may acquire all available memory such that other sorts in the same task do not have enough to proceed. This manifests itself in an
IOException("Unable to acquire X bytes of memory")thrown byUnsafeExternalSorter.The solution is to reserve a page in each sorter in the chain before computing the child operator's (parent RDD's) partitions. This requires us to use a new special RDD that does some preparation before computing the parent's partitions.