Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[LIVY-336][SERVER] Livy should not spawn one thread per job to track the job on Yarn #242

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[LIVY-336][SERVER] Livy should not spawn one thread per job to track the job on Yarn #242

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
36a4708
[LIVY-336] Livy should not spawn one thread per job to track the job …
runzhiwang Oct 6, 2019
b0d1900
[LIVY-336] fix code review
runzhiwang Oct 13, 2019
56a56b6
[LIVY-336] fix code review
runzhiwang Nov 8, 2019
78c7898
[LIVY-336] fix code review
runzhiwang Nov 18, 2019
8b82288
[LIVY-336] fix code review
runzhiwang Nov 19, 2019
e8f83e2
[LIVY-336] fix code review
runzhiwang Nov 19, 2019
0584a8a
[LIVY-336] remove redundant code
runzhiwang Nov 20, 2019
9521863
[LIVY-336] fix code review
runzhiwang Nov 26, 2019
b21e404
[LIVY-336] add a thread to check whether the monitor was blocked
runzhiwang Nov 29, 2019
fc698c9
remove redundant config and break while when interrupt
runzhiwang Dec 6, 2019
c3c86e3
Fix error default value in config
runzhiwang Dec 6, 2019
3144ff1
fix code review
runzhiwang Dec 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 7 additions & 2 deletions conf/livy.conf.template
View file
Open in desktop
Expand Up @@ -121,11 +121,16 @@

# If Livy can't find the yarn app within this time, consider it lost.
# livy.server.yarn.app-lookup-timeout = 120s
# If Livy can't find the yarn app within this max times, consider it lost.
# livy.server.yarn.app-lookup.max-failed.times = 120
# The size of thread pool to monitor all yarn apps.
# livy.server.yarn.app-lookup.thread-pool.size = 4

# When the cluster is busy, we may fail to launch yarn app in app-lookup-timeout, then it would
# cause session leakage, so we need to check session leakage.
# How long to check livy session leakage
# How long to check livy session leakage.
# livy.server.yarn.app-leakage.check-timeout = 600s
# how often to check livy session leakage
# How often to check livy session leakage.
# livy.server.yarn.app-leakage.check-interval = 60s

# How often Livy polls YARN to refresh YARN app state.
Expand Down
11 changes: 9 additions & 2 deletions server/src/main/scala/org/apache/livy/LivyConf.scala
View file
Open in desktop
Expand Up @@ -205,6 +205,13 @@ object LivyConf {
// If Livy can't find the yarn app within this time, consider it lost.
val YARN_APP_LOOKUP_TIMEOUT = Entry("livy.server.yarn.app-lookup-timeout", "120s")

// If Livy can't find the yarn app within this max times, consider it lost.
val YARN_APP_LOOKUP_MAX_FAILED_TIMES = Entry("livy.server.yarn.app-lookup.max-failed.times", 120)
runzhiwang marked this conversation as resolved.
Show resolved Hide resolved

// The size of thread pool to monitor all yarn apps.
val YARN_APP_LOOKUP_THREAD_POOL_SIZE =
Entry("livy.server.yarn.app-lookup.thread-pool.size", 4)

// How often Livy polls YARN to refresh YARN app state.
val YARN_POLL_INTERVAL = Entry("livy.server.yarn.poll-interval", "5s")
runzhiwang marked this conversation as resolved.
Show resolved Hide resolved

Expand All @@ -216,9 +223,9 @@ object LivyConf {
// RSC related jars separated with comma.
val RSC_JARS = Entry("livy.rsc.jars", null)

// How long to check livy session leakage
// How long to check livy session leakage.
val YARN_APP_LEAKAGE_CHECK_TIMEOUT = Entry("livy.server.yarn.app-leakage.check-timeout", "600s")
// how often to check livy session leakage
// How often to check livy session leakage.
val YARN_APP_LEAKAGE_CHECK_INTERVAL = Entry("livy.server.yarn.app-leakage.check-interval", "60s")

// Whether session timeout should be checked, by default it will be checked, which means inactive
Expand Down