[SPARK-4266] [Web-UI] Reduce stage page load time. #3328

kayousterhout · 2014-11-17T23:25:54Z

The commit changes the java script used to show/hide additional
metrics in order to reduce page load time. SPARK-4016 significantly
increased page load time for the stage page when stages had a lot
(thousands or tens of thousands) of tasks, due to the additional
Javascript to hide some metrics by default and stripe the tables.
This commit reduces page load time in two ways:

(1) Now, all of the metrics that are hidden by default are
hidden by setting "display: none;" using CSS for the page,
rather than hiding them using javascript after the page loads.
Without this change, for stages with thousands of tasks, there
was a few second delay after page load, where first the additional
metrics were shown, and then after a delay were hidden once the
relevant JS finished running.

(2) CSS is used to stripe all of the tables except for the summary
table. The summary table needs javascript to do the striping because
some rows are hidden, but the javascript striping is slower, which
again resulted in a delay when it was used for the task table (where
for a few seconds after page load, all of the rows in the task table
would be white, while the browser finished running the JS to stripe
the table).

cc @pwendell

This change is intended to be backported to 1.2 to avoid a regression in
UI performance when users run large jobs.

SparkQA · 2014-11-17T23:29:54Z

Test build #23512 has started for PR 3328 at commit 5d98669.

This patch merges cleanly.

SparkQA · 2014-11-18T00:55:07Z

Test build #23512 has finished for PR 3328 at commit 5d98669.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-18T00:55:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23512/
Test PASSed.

andrewor14 · 2014-11-19T02:51:10Z

core/src/main/scala/org/apache/spark/ui/UIUtils.scala

-      listingTableClass += " table-fixed"
+    var listingTableClass = {
+      if (stripeRowsWithCss) TABLE_CLASS_STRIPED
+      else TABLE_CLASS_NOT_STRIPED
    }


can you do

var listingTableClass = if (stripeRowsWithCss) { TABLE_CLASS_STRIPED } else { TABLE_CLASS_NOT_STRIPED }

kayousterhout · 2014-11-19T19:28:16Z

Thanks Andrew! On further inspection I realized this fits in one line which is, I think, the cleanest way to do this.

SparkQA · 2014-11-19T19:35:15Z

Test build #23618 has started for PR 3328 at commit e2d2bb1.

This patch merges cleanly.

SparkQA · 2014-11-19T21:01:42Z

Test build #23618 has finished for PR 3328 at commit e2d2bb1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-19T21:01:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23618/
Test FAILed.

SparkQA · 2014-11-19T21:20:09Z

Test build #23629 has started for PR 3328 at commit 9470b4f.

This patch merges cleanly.

SparkQA · 2014-11-19T22:46:06Z

Test build #23629 has finished for PR 3328 at commit 9470b4f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-19T22:46:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23629/
Test FAILed.

andrewor14 · 2014-11-20T00:37:50Z

retest this please

SparkQA · 2014-11-20T00:45:37Z

Test build #23647 has started for PR 3328 at commit 9470b4f.

This patch merges cleanly.

SparkQA · 2014-11-20T02:16:37Z

Test build #23647 has finished for PR 3328 at commit 9470b4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-20T02:16:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23647/
Test PASSed.

kayousterhout · 2014-11-24T21:00:35Z

@andrewor14 @pwendell any more comments here?

pwendell · 2014-11-24T21:15:47Z

LGTM

andrewor14 · 2014-11-24T21:45:34Z

Yeah, LGTMT. Feel free to merge it

pwendell · 2014-11-24T21:56:09Z

core/src/main/resources/org/apache/spark/ui/static/table.js

+ * An ID selector (rather than a class selector) is used to ensure this runs quickly even on pages
+ * with thousands of task rows (ID selectors are much faster than class selectors). */
+function stripeSummaryTable() {
+    $("#task-summary-table").find("tr:not(:hidden)").each(function (index) {


just wondering - is using this selector more expensive than just traversing elements in children() and checking manually for each one and then maintaining your own counter? It seems like interpreting this string could be expensive inside of javascript.

This is a table that has at most 6 or 7 rows (it's the summary table for each metric) -- so I don't think that level of optimization matters (the most important thing here is the ID selector, so you don't need to traverse the task table, which can have infinitely many rows)

Didn't follow that. Thanks for explaining.

The commit changes the java script used to show/hide additional metrics in order to reduce page load time. SPARK-4016 significantly increased page load time for the stage page when stages had a lot (thousands or tens of thousands) of tasks, due to the additional Javascript to hide some metrics by default and stripe the tables. This commit reduces page load time in two ways: (1) Now, all of the metrics that are hidden by default are hidden by setting "display: none;" using CSS for the page, rather than hiding them using javascript after the page loads. Without this change, for stages with thousands of tasks, there was a few second delay after page load, where first the additional metrics were shown, and then after a delay were hidden once the relevant JS finished running. (2) CSS is used to stripe all of the tables except for the summary table. The summary table needs javascript to do the striping because some rows are hidden, but the javascript striping is slower, which again resulted in a delay when it was used for the task table (where for a few seconds after page load, all of the rows in the task table would be white, while the browser finished running the JS to stripe the table).

kayousterhout · 2014-11-24T23:20:44Z

Ok I rebased this and will let Jenkins test it one more time (since it was last tested 5 days ago and I know lots has been merged since then) -- then I'll merge it assuming all of the tests pass.

SparkQA · 2014-11-24T23:22:55Z

Test build #23799 has started for PR 3328 at commit f964091.

This patch merges cleanly.

SparkQA · 2014-11-25T00:47:30Z

Test build #23799 has finished for PR 3328 at commit f964091.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-25T00:47:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23799/
Test PASSed.

The commit changes the java script used to show/hide additional metrics in order to reduce page load time. SPARK-4016 significantly increased page load time for the stage page when stages had a lot (thousands or tens of thousands) of tasks, due to the additional Javascript to hide some metrics by default and stripe the tables. This commit reduces page load time in two ways: (1) Now, all of the metrics that are hidden by default are hidden by setting "display: none;" using CSS for the page, rather than hiding them using javascript after the page loads. Without this change, for stages with thousands of tasks, there was a few second delay after page load, where first the additional metrics were shown, and then after a delay were hidden once the relevant JS finished running. (2) CSS is used to stripe all of the tables except for the summary table. The summary table needs javascript to do the striping because some rows are hidden, but the javascript striping is slower, which again resulted in a delay when it was used for the task table (where for a few seconds after page load, all of the rows in the task table would be white, while the browser finished running the JS to stripe the table). cc pwendell This change is intended to be backported to 1.2 to avoid a regression in UI performance when users run large jobs. Author: Kay Ousterhout <kayousterhout@gmail.com> Closes #3328 from kayousterhout/SPARK-4266 and squashes the following commits: f964091 [Kay Ousterhout] [SPARK-4266] [Web-UI] Reduce stage page load time. (cherry picked from commit d24d5bf) Signed-off-by: Kay Ousterhout <kayousterhout@gmail.com>

andrewor14 reviewed Nov 19, 2014
View reviewed changes

kayousterhout force-pushed the SPARK-4266 branch from e2d2bb1 to 9470b4f Compare November 19, 2014 21:16

pwendell reviewed Nov 24, 2014
View reviewed changes

kayousterhout force-pushed the SPARK-4266 branch from 9470b4f to f964091 Compare November 24, 2014 23:20

asfgit closed this in d24d5bf Nov 25, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-4266] [Web-UI] Reduce stage page load time. #3328

[SPARK-4266] [Web-UI] Reduce stage page load time. #3328

kayousterhout commented Nov 17, 2014

SparkQA commented Nov 17, 2014

SparkQA commented Nov 18, 2014

AmplabJenkins commented Nov 18, 2014

andrewor14 Nov 19, 2014

kayousterhout commented Nov 19, 2014

SparkQA commented Nov 19, 2014

SparkQA commented Nov 19, 2014

AmplabJenkins commented Nov 19, 2014

SparkQA commented Nov 19, 2014

SparkQA commented Nov 19, 2014

AmplabJenkins commented Nov 19, 2014

andrewor14 commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

kayousterhout commented Nov 24, 2014

pwendell commented Nov 24, 2014

andrewor14 commented Nov 24, 2014

pwendell Nov 24, 2014

kayousterhout Nov 24, 2014

pwendell Nov 24, 2014

kayousterhout commented Nov 24, 2014

SparkQA commented Nov 24, 2014

SparkQA commented Nov 25, 2014

AmplabJenkins commented Nov 25, 2014

[SPARK-4266] [Web-UI] Reduce stage page load time. #3328

[SPARK-4266] [Web-UI] Reduce stage page load time. #3328

Conversation

kayousterhout commented Nov 17, 2014

SparkQA commented Nov 17, 2014

SparkQA commented Nov 18, 2014

AmplabJenkins commented Nov 18, 2014

andrewor14 Nov 19, 2014

Choose a reason for hiding this comment

kayousterhout commented Nov 19, 2014

SparkQA commented Nov 19, 2014

SparkQA commented Nov 19, 2014

AmplabJenkins commented Nov 19, 2014

SparkQA commented Nov 19, 2014

SparkQA commented Nov 19, 2014

AmplabJenkins commented Nov 19, 2014

andrewor14 commented Nov 20, 2014

SparkQA commented Nov 20, 2014

SparkQA commented Nov 20, 2014

AmplabJenkins commented Nov 20, 2014

kayousterhout commented Nov 24, 2014

pwendell commented Nov 24, 2014

andrewor14 commented Nov 24, 2014

pwendell Nov 24, 2014

Choose a reason for hiding this comment

kayousterhout Nov 24, 2014

Choose a reason for hiding this comment

pwendell Nov 24, 2014

Choose a reason for hiding this comment

kayousterhout commented Nov 24, 2014

SparkQA commented Nov 24, 2014

SparkQA commented Nov 25, 2014

AmplabJenkins commented Nov 25, 2014