[SPARK-12155] [SPARK-12253] Fix executor OOM in unified memory management #10240

andrewor14 · 2015-12-10T02:49:44Z

Problem. In unified memory management, acquiring execution memory may lead to eviction of storage memory. However, the space freed from evicting cached blocks is distributed among all active tasks. Thus, an incorrect upper bound on the execution memory per task can cause the acquisition to fail, leading to OOM's and premature spills.

Example. Suppose total memory is 1000B, cached blocks occupy 900B, spark.memory.storageFraction is 0.4, and there are two active tasks. In this case, the cap on task execution memory is 100B / 2 = 50B. If task A tries to acquire 200B, it will evict 100B of storage but can only acquire 50B because of the incorrect cap. For another example, see this regression test that I stole from @JoshRosen.

Solution. Fix the cap on task execution memory. It should take into account the space that could have been freed by storage in addition to the current amount of memory available to execution. In the example above, the correct cap should have been 600B / 2 = 300B.

This patch also guards against the race condition (SPARK-12253):
(1) Existing tasks collectively occupy all execution memory
(2) New task comes in and blocks while existing tasks spill
(3) After tasks finish spilling, another task jumps in and puts in a large block, stealing the freed memory
(4) New task still cannot acquire memory and goes back to sleep

andrewor14 · 2015-12-10T02:56:40Z

@JoshRosen @davies

SparkQA · 2015-12-10T04:39:47Z

Test build #47474 has finished for PR 10240 at commit cd0c680.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * logInfo(s\"HBase class not found $e\")\n * logDebug(\"HBase class not found\", e)\n

davies · 2015-12-10T06:02:51Z

core/src/main/scala/org/apache/spark/memory/ExecutionMemoryPool.scala

        // We want to let each task get at least 1 / (2 * numActiveTasks) before blocking;
        // if we can't give it this much now, wait for other tasks to free up memory
        // (this happens if older tasks allocated lots of memory before N grew)
-        if (memoryFree >= math.min(maxToGrant, poolSize / (2 * numActiveTasks) - curMem)) {
+        if (memoryFree >= math.min(maxToGrant, poolSize / minMemoryPerTask)) {


poolSize / minMemoryPerTask should be minMemoryPerTask - curMem

oops good catch!!

SparkQA · 2015-12-10T08:42:08Z

Test build #47491 has finished for PR 10240 at commit d8be669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-10T15:03:53Z

Test build #2195 has finished for PR 10240 at commit d8be669.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-12-10T17:51:54Z

ok, retest this please

andrewor14 · 2015-12-10T17:52:19Z

Latest commit actually passed tests last night.

SparkQA · 2015-12-10T19:48:51Z

Test build #47528 has finished for PR 10240 at commit d8be669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@davies

Per suggestion from @davies. For a detailed proof of why these two are the same, see this gist: https://gist.github.com/andrewor14/aea58796dd25d2ec9f20

andrewor14 · 2015-12-10T20:19:53Z

retest this please

andrewor14 · 2015-12-10T20:20:24Z

@davies please look at the final changes.

nongli · 2015-12-10T20:32:40Z

core/src/main/scala/org/apache/spark/memory/ExecutionMemoryPool.scala

+   * @param maybeGrowPool a callback that potentially grows the size of this pool. It takes in
+   *                      one parameter (Long) that represents the desired amount of memory by
+   *                      which this pool should be expanded.
+   * @param computeMaxPoolSize a callback that returns the maximum allowable size of this pool


If you take into account the memory that can be freed then isn't this a fixed value?

The actual used memory by storage could be changed, so it's not a fixed value

Nvm, I see now.

no, because if storage memory used is below a certain mark (default 0.5 of max memory) then it cannot be evicted. In this case the max pool size depends on how much unevictable storage memory there is, which varies over time.

davies · 2015-12-10T20:37:09Z

LGTM

SparkQA · 2015-12-10T22:20:03Z

Test build #2198 has finished for PR 10240 at commit d8be669.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * logInfo(s\"HBase class not found $e\")\n * logDebug(\"HBase class not found\", e)\n

SparkQA · 2015-12-10T22:33:16Z

Test build #47539 has finished for PR 10240 at commit 2929640.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-10T22:34:33Z

Test build #2197 has finished for PR 10240 at commit d8be669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-10T22:41:32Z

Test build #2196 has finished for PR 10240 at commit d8be669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-12-10T23:29:56Z

Thanks, merging into master 1.6.

**Problem.** In unified memory management, acquiring execution memory may lead to eviction of storage memory. However, the space freed from evicting cached blocks is distributed among all active tasks. Thus, an incorrect upper bound on the execution memory per task can cause the acquisition to fail, leading to OOM's and premature spills. **Example.** Suppose total memory is 1000B, cached blocks occupy 900B, `spark.memory.storageFraction` is 0.4, and there are two active tasks. In this case, the cap on task execution memory is 100B / 2 = 50B. If task A tries to acquire 200B, it will evict 100B of storage but can only acquire 50B because of the incorrect cap. For another example, see this [regression test](https://github.com/andrewor14/spark/blob/fix-oom/core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala#L233) that I stole from JoshRosen. **Solution.** Fix the cap on task execution memory. It should take into account the space that could have been freed by storage in addition to the current amount of memory available to execution. In the example above, the correct cap should have been 600B / 2 = 300B. This patch also guards against the race condition (SPARK-12253): (1) Existing tasks collectively occupy all execution memory (2) New task comes in and blocks while existing tasks spill (3) After tasks finish spilling, another task jumps in and puts in a large block, stealing the freed memory (4) New task still cannot acquire memory and goes back to sleep Author: Andrew Or <andrew@databricks.com> Closes #10240 from andrewor14/fix-oom. (cherry picked from commit 5030923) Signed-off-by: Andrew Or <andrew@databricks.com>

Andrew Or added 3 commits December 9, 2015 15:48

Pass in callbacks (gross)

6833775

Notify all on task completion

35392f5

Rename silly method names + add detailed comments

cd0c680

JoshRosen mentioned this pull request Dec 10, 2015

[SPARK-12155][WIP] Fix bug in eviction of storage memory by execution w/ multiple active tasks #10230

Closed

andrewor14 changed the title ~~[SPARK-12155] Fix executor OOM in unified memory management~~ [SPARK-12155] [SPARK-12253] Fix executor OOM in unified memory management Dec 10, 2015

davies reviewed Dec 10, 2015
View reviewed changes

Fix unintended change

d8be669

Simplify 1/2N code but preserve behavior

2929640

Per suggestion from @davies. For a detailed proof of why these two are the same, see this gist: https://gist.github.com/andrewor14/aea58796dd25d2ec9f20

nongli reviewed Dec 10, 2015
View reviewed changes

asfgit closed this in 5030923 Dec 10, 2015

andrewor14 deleted the fix-oom branch December 10, 2015 23:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12155] [SPARK-12253] Fix executor OOM in unified memory management #10240

[SPARK-12155] [SPARK-12253] Fix executor OOM in unified memory management #10240

andrewor14 commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

SparkQA commented Dec 10, 2015

davies Dec 10, 2015

andrewor14 Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

SparkQA commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

nongli Dec 10, 2015

davies Dec 10, 2015

nongli Dec 10, 2015

andrewor14 Dec 10, 2015

davies commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

[SPARK-12155] [SPARK-12253] Fix executor OOM in unified memory management #10240

[SPARK-12155] [SPARK-12253] Fix executor OOM in unified memory management #10240

Conversation

andrewor14 commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

SparkQA commented Dec 10, 2015

davies Dec 10, 2015

Choose a reason for hiding this comment

andrewor14 Dec 10, 2015

Choose a reason for hiding this comment

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

SparkQA commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

andrewor14 commented Dec 10, 2015

nongli Dec 10, 2015

Choose a reason for hiding this comment

davies Dec 10, 2015

Choose a reason for hiding this comment

nongli Dec 10, 2015

Choose a reason for hiding this comment

andrewor14 Dec 10, 2015

Choose a reason for hiding this comment

davies commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

SparkQA commented Dec 10, 2015

andrewor14 commented Dec 10, 2015