[SPARK-35402][WebUI] Increase the max thread pool size of jetty server in HistoryServer UI #32539

yaooqinn · 2021-05-13T17:00:48Z

What changes were proposed in this pull request?

For different UIs, e.g. History Server or Spark Live UI, maybe need different capabilities to handle HTTP requests. Usually, a History Server is for multi-users and needs more threads to increase concurrency, while Live UI is per application, which needn't that large pool size.

In this PR, we increase the max pool size of the History Server's jetty backend

Why are the changes needed?

increase the client concurrency of HistoryServer

Does this PR introduce any user-facing change?

no

How was this patch tested?

new tests

…igurable

SparkQA · 2021-05-13T18:18:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43039/

SparkQA · 2021-05-13T20:05:52Z

Test build #138518 has finished for PR 32539 at commit 19bf355.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2021-05-14T01:53:53Z

cc @cloud-fan @srowen @maropu @sarutak thanks

core/src/main/scala/org/apache/spark/internal/config/UI.scala

…igurable

SparkQA · 2021-05-14T03:37:28Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43057/

SparkQA · 2021-05-14T03:37:29Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43057/

SparkQA · 2021-05-14T05:26:54Z

Test build #138536 has finished for PR 32539 at commit 37a168a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-05-25T14:14:39Z

Can we simply use different max threads num in live UI and history server UI? e.g. 200 for live UI and 1000 for history server UI? We still hardcode it and no need to add a config.

yaooqinn · 2021-05-25T15:06:38Z

Can we simply use different max threads num in live UI and history server UI? e.g. 200 for live UI and 1000 for history server UI? We still hardcode it and no need to add a config.

yea, I will try

SparkQA · 2021-05-29T08:39:02Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43600/

SparkQA · 2021-05-29T10:07:37Z

Test build #139079 has finished for PR 32539 at commit fecbb42.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala

cloud-fan · 2021-06-01T13:03:35Z

core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala

    port: Int)
-  extends WebUI(securityManager, securityManager.getSSLOptions("historyServer"), port, conf)
+  extends WebUI(securityManager, securityManager.getSSLOptions("historyServer"),
+    port, conf, poolSize = 1000)


We can add a bit more comments to explain why history server needs a larger pool size.

BTW we can also set the name parameter properly as you did before.

SparkQA · 2021-06-01T13:08:25Z

Test build #139160 has finished for PR 32539 at commit 21a799a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

srowen

Looks OK, but let's update the title of this PR and JIRA then.

SparkQA · 2021-06-01T13:42:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43677/

SparkQA · 2021-06-01T13:49:43Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43679/

SparkQA · 2021-06-01T13:50:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43677/

SparkQA · 2021-06-01T14:42:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43684/

SparkQA · 2021-06-01T15:10:39Z

Test build #139158 has finished for PR 32539 at commit d6a27df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-06-01T15:14:08Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43684/

yaooqinn · 2021-06-01T15:58:39Z

thanks, @srowen, I've updated the content accordingly.

SparkQA · 2021-06-01T16:35:39Z

Test build #139164 has finished for PR 32539 at commit 065ddce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2021-06-01T17:03:16Z

thanks, @srowen @cloud-fan for the review, merged to master for 3.2.0

[SPARK-35402][WebUI] Make thread pool of jetty server in Web UIs conf…

19bf355

…igurable

github-actions bot added CORE WEB UI labels May 13, 2021

This comment has been minimized.

Sign in to view

srowen reviewed May 14, 2021

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/UI.scala Outdated Show resolved Hide resolved

yaooqinn added 3 commits May 14, 2021 10:00

[SPARK-35402][WebUI] Make thread pool of jetty server in Web UIs conf…

58b651a

…igurable

[SPARK-35402][WebUI] Make thread pool of jetty server in Web UIs conf…

e876efe

…igurable

[SPARK-35402][WebUI] Make thread pool of jetty server in Web UIs conf…

37a168a

…igurable

yaooqinn self-assigned this May 14, 2021

hardcode

fecbb42

cloud-fan reviewed Jun 1, 2021

View reviewed changes

core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala Outdated Show resolved Hide resolved

yaooqinn added 3 commits June 1, 2021 20:33

address comments

d6a27df

Merge branch 'master' into SPARK-35402

0c0993a

address comments

21a799a

cloud-fan reviewed Jun 1, 2021

View reviewed changes

address comments

065ddce

cloud-fan approved these changes Jun 1, 2021

View reviewed changes

srowen reviewed Jun 1, 2021

View reviewed changes

yaooqinn changed the title ~~[SPARK-35402][WebUI] Make thread pool of jetty server in Web UIs configurable~~ [SPARK-35402][WebUI] Increase the max thread pool size of jetty server in HistoryServer UI Jun 1, 2021

yaooqinn closed this in a127d91 Jun 1, 2021

yaooqinn deleted the SPARK-35402 branch June 1, 2021 17:03

[SPARK-35402][WebUI] Increase the max thread pool size of jetty server in HistoryServer UI #32539

[SPARK-35402][WebUI] Increase the max thread pool size of jetty server in HistoryServer UI #32539

Uh oh!

Conversation

yaooqinn commented May 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

This comment has been minimized.

SparkQA commented May 13, 2021

Uh oh!

SparkQA commented May 13, 2021

Uh oh!

yaooqinn commented May 14, 2021

Uh oh!

Uh oh!

SparkQA commented May 14, 2021

Uh oh!

SparkQA commented May 14, 2021

Uh oh!

SparkQA commented May 14, 2021

Uh oh!

cloud-fan commented May 25, 2021

Uh oh!

yaooqinn commented May 25, 2021

Uh oh!

SparkQA commented May 29, 2021

Uh oh!

SparkQA commented May 29, 2021

Uh oh!

Uh oh!

cloud-fan Jun 1, 2021

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 1, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

yaooqinn commented Jun 1, 2021

Uh oh!

SparkQA commented Jun 1, 2021

Uh oh!

yaooqinn commented Jun 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yaooqinn commented May 13, 2021 •

edited

Loading