Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bugfix][Core] Fix Handover using linked blocking queue cause the oom #3469

Merged
merged 3 commits into from
Jan 19, 2023

Conversation

harveyyue
Copy link
Contributor

@harveyyue harveyyue commented Nov 18, 2022

Fix following OOM with spark engine.

22/11/18 10:04:51 INFO AsyncEventQueue: Process of event SparkListenerExecutorMetricsUpdate(driver,WrappedArray((0,0,0,Vector(AccumulableInfo(7,None,Some(35908),None,false,true,None), AccumulableInfo(24,None,Some(2000),None,false,true,None), AccumulableInfo(1,Some(number of output rows),Some(2000),None,true,true,Some(sql))))),Map((0,0) -> org.apache.spark.executor.ExecutorMetrics@20f22a4a)) by listener AppStatusListener took 1.304755574s.

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RemoteBlock-temp-file-clean-thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "mysql-cj-abandoned-connection-cleanup"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "parallel-split-enumerator-executor-0-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "netty-rpc-env-timeout"
^C^C^COpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "dispatcher-event-loop-0"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "driver-heartbeater"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SparkUI-38"

Purpose of this pull request

Check list

@TyrantLucifer
Copy link
Member

@ashulin PTAL.

hailin0
hailin0 previously approved these changes Nov 19, 2022
this.lock = new ReentrantLock();
this.isNotFull = lock.newCondition();

this.queue = new ArrayDeque<>(DEFAULT_QUEUE_SIZE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use ArrayBlockingQueue?

Copy link
Member

@ashulin ashulin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, Add a default capacity to LinkedBlockingQueue

@hailin0 hailin0 removed the approved label Nov 21, 2022
@hailin0 hailin0 self-requested a review November 21, 2022 01:33
@hailin0
Copy link
Contributor

hailin0 commented Nov 21, 2022

IMO, Add a default capacity to LinkedBlockingQueue

agree with you

@harveyyue please use new LinkedBlockingQueue<>(capacity) or new ArrayBlockingQueue<>(capacity)

@CalvinKirs
Copy link
Member

hi, how is the progress?

Copy link
Contributor

@dijiekstra dijiekstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@wuchunfu wuchunfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TyrantLucifer TyrantLucifer merged commit 73b523d into apache:dev Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants