[SPARK-26117][CORE][SQL]use SparkOutOfMemoryError instead of OutOfMemoryError when catch exception #23084
Conversation
| @@ -741,7 +742,7 @@ public boolean append(Object kbase, long koff, int klen, Object vbase, long voff | |||
| if (numKeys >= growthThreshold && longArray.size() < MAX_CAPACITY) { | |||
| try { | |||
| growAndRehash(); | |||
| } catch (OutOfMemoryError oom) { | |||
| } catch (SparkOutOfMemoryError oom) { | |||
There was a problem hiding this comment.
do you know what was the behavior before? Will we propagate the exception all the way up and kill the executor?
|
ok to test |
|
add to whitelist |
|
good catch! thanks! |
|
I think that we need to take care of |
|
Test build #99006 has finished for PR 23084 at commit
|
|
Test build #99002 has finished for PR 23084 at commit
|
heary-cao
force-pushed
the
SparkOutOfMemoryError
branch
from
7345942
to
400bcd5
Compare
|
@cloud-fan, I think not kill executor. the memory allocate fails, and the caller is notified to perform other operations. for example, HashAggregateExec will carry out spill the map and fallback to sort-based. |
|
Test build #99045 has finished for PR 23084 at commit
|
|
retest this please |
|
Test build #99046 has finished for PR 23084 at commit
|
Do you mean this patch change nothing for this case? |
|
I think |
|
so this PR is just code cleanup, not a real bug. thanks. |
|
I think it's safer to only catch the spark-thrown OOM, not the system OOM, so LGTM |
|
thanks, merging to master! |
|
@cloud-fan,thanks |
…moryError when catch exception ## What changes were proposed in this pull request? the pr apache#20014 which introduced `SparkOutOfMemoryError` to avoid killing the entire executor when an `OutOfMemoryError `is thrown. so apply for memory using `MemoryConsumer. allocatePage `when catch exception, use `SparkOutOfMemoryError `instead of `OutOfMemoryError` ## How was this patch tested? N / A Closes apache#23084 from heary-cao/SparkOutOfMemoryError. Authored-by: caoxuewen <cao.xuewen@zte.com.cn> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
the pr #20014 which introduced
SparkOutOfMemoryErrorto avoid killing the entire executor when anOutOfMemoryErroris thrown.so apply for memory using
MemoryConsumer. allocatePagewhen catch exception, useSparkOutOfMemoryErrorinstead ofOutOfMemoryErrorHow was this patch tested?
N / A