-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-21254] [WebUI] History UI performance fixes #18783
[SPARK-21254] [WebUI] History UI performance fixes #18783
Conversation
…ore processing Currently all the DOM manipulations are handled in a loop after Mustache template is parsed. This causes severe performance issues especially within loops iteration over thousands of (attempt/application) records and causing all kinds of unnecessary browser work: reflow, repaint, etc. This could be easily fixed by preparing a DOM node beforehand and doing all manipulations within the loops on detached node, reattaching it to the document only after the work is done. The most costly operation in this case was setting innerHTML for `duration` column within a loop, which is extremely unperformant: https://jsperf.com/jquery-append-vs-html-list-performance/24 While duration parsing could be done before mustache-template processing without any additional DOM alteratoins.
Check whether to display pagination or not on large data sets (10-20k rows) was taking up to 50ms because it was iterating over all rows. This could be easily done by testing length of array before passing it to mustache.
Logic related to `hasMultipleAttempts` flag: - Hiding attmptId column (if `hasMultipleAttempts = false`) - Seting white background color for first 2 columns (if `hasMultipleAttempts = true`) was updating DOM after mustache template processing, which was causing 2 unnecessary iterations over full data set (first through jquery selector, than through for-loop). Refactoring it inside mustache template helps saving 80-90ms on large data sets (10k+ rows)
Refactoring incomplete requests filter behavior due to inefficency in DOM manipulations. We were traversing DOM multiple times just to hide columns that we could have avoided rendering in mustache (end date, duration). Factoring this logic in mustache template (`showCompletedColumn`) saves 70-80ms on 10k+ rows.
Detaching history table wrapper from document before parsing it with DataTables plugin and reattaching back right after plugin has processed nested DOM. This allows to avoid huge amount of browser repaints and reflows, reducing initial page load time in Chrome from 15s to 4s for 20k+ rows
Looks like you opened a duplicate of #18777 |
It would have to go in master first so close the 2.1one |
@srowen Done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, LGTM! This is a fantastic refactor, even just from a code perspective it's much cleaner and more efficient.
Thank you @ajbozarth |
It looks a lot simpler. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like it if @ajbozarth does
pom.xml
Outdated
@@ -179,7 +179,7 @@ | |||
<jpam.version>1.1</jpam.version> | |||
<selenium.version>2.52.0</selenium.version> | |||
<paranamer.version>2.6</paranamer.version> | |||
<maven-antrun.version>1.8</maven-antrun.version> | |||
<selenium.htmlunitdriver.version>2.27</selenium.htmlunitdriver.version> <maven-antrun.version>1.8</maven-antrun.version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Add back a newline at the end before the next property)
Selenium's version don't necessarily match with its htmlunitdriver? just checking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi,@srowen
Is the version of "selenium htmlunitdriver" too low?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@highfei2011 @srowen Reverted all pom.xml change since they appear not to be mandatory for this PR
Indeed it seems that selenium version should have been updated as well.
…ependency" Upgrade of HtmlUnit seems to bring new dependencies and since it is not required for the purposes of PR (`HistoryServerSuite` tests pass anyway on our Jenkins), we revert HtmlUnit version change This reverts commit 96598ab.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Revering pom.xml changes
pom.xml
Outdated
@@ -179,7 +179,7 @@ | |||
<jpam.version>1.1</jpam.version> | |||
<selenium.version>2.52.0</selenium.version> | |||
<paranamer.version>2.6</paranamer.version> | |||
<maven-antrun.version>1.8</maven-antrun.version> | |||
<selenium.htmlunitdriver.version>2.27</selenium.htmlunitdriver.version> <maven-antrun.version>1.8</maven-antrun.version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@highfei2011 @srowen Reverted all pom.xml change since they appear not to be mandatory for this PR
Indeed it seems that selenium version should have been updated as well.
Could we try to run tests and build on Jenkins? |
Jenkins test this please |
Test build #80133 has finished for PR 18783 at commit
|
Thank you @srowen. Seems like jenkins is happy. Should we merge in this case? |
Merged to master |
## This is a backport of PR #18783 to the latest released branch 2.2. ## What changes were proposed in this pull request? As described in JIRA ticket, History page is taking ~1min to load for cases when amount of jobs is 10k+. Most of the time is currently being spent on DOM manipulations and all additional costs implied by this (browser repaints and reflows). PR's goal is not to change any behavior but to optimize time of History UI rendering: 1. The most costly operation is setting `innerHTML` for `duration` column within a loop, which is [extremely unperformant](https://jsperf.com/jquery-append-vs-html-list-performance/24). [Refactoring ](criteo-forks@b7e56ee) this helped to get page load time **down to 10-15s** 2. Second big gain bringing page load time **down to 4s** was [was achieved](criteo-forks@3630ca2) by detaching table's DOM before parsing it with DataTables jQuery plugin. 3. Another chunk of improvements ([1]criteo-forks@aeeeeb5), [2](criteo-forks@e25be9a), [3](criteo-forks@9169707)) was focused on removing unnecessary DOM manipulations that in total contributed ~250ms to page load time. ## How was this patch tested? Tested by existing Selenium tests in `org.apache.spark.deploy.history.HistoryServerSuite`. Changes were also tested on Criteo's spark-2.1 fork with 20k+ number of rows in the table, reducing load time to 4s. Author: Dmitry Parfenchik <d.parfenchik@criteo.com> Closes #18860 from 2ooom/history-ui-perf-fix-2.2.
## This is a backport of PR apache#18783 to the latest released branch 2.2. ## What changes were proposed in this pull request? As described in JIRA ticket, History page is taking ~1min to load for cases when amount of jobs is 10k+. Most of the time is currently being spent on DOM manipulations and all additional costs implied by this (browser repaints and reflows). PR's goal is not to change any behavior but to optimize time of History UI rendering: 1. The most costly operation is setting `innerHTML` for `duration` column within a loop, which is [extremely unperformant](https://jsperf.com/jquery-append-vs-html-list-performance/24). [Refactoring ](criteo-forks@b7e56ee) this helped to get page load time **down to 10-15s** 2. Second big gain bringing page load time **down to 4s** was [was achieved](criteo-forks@3630ca2) by detaching table's DOM before parsing it with DataTables jQuery plugin. 3. Another chunk of improvements ([1]criteo-forks@aeeeeb5), [2](criteo-forks@e25be9a), [3](criteo-forks@9169707)) was focused on removing unnecessary DOM manipulations that in total contributed ~250ms to page load time. ## How was this patch tested? Tested by existing Selenium tests in `org.apache.spark.deploy.history.HistoryServerSuite`. Changes were also tested on Criteo's spark-2.1 fork with 20k+ number of rows in the table, reducing load time to 4s. Author: Dmitry Parfenchik <d.parfenchik@criteo.com> Closes apache#18860 from 2ooom/history-ui-perf-fix-2.2.
What changes were proposed in this pull request?
As described in JIRA ticket, History page is taking ~1min to load for cases when amount of jobs is 10k+.
Most of the time is currently being spent on DOM manipulations and all additional costs implied by this (browser repaints and reflows).
PR's goal is not to change any behavior but to optimize time of History UI rendering:
The most costly operation is setting
innerHTML
forduration
column within a loop, which is extremely unperformant. Refactoring this helped to get page load time down to 10-15sSecond big gain bringing page load time down to 4s was was achieved by detaching table's DOM before parsing it with DataTables jQuery plugin.
Another chunk of improvements (1, 2, 3) was focused on removing unnecessary DOM manipulations that in total contributed ~250ms to page load time.
How was this patch tested?
Tested by existing Selenium tests in
org.apache.spark.deploy.history.HistoryServerSuite
.Changes were also tested on Criteo's spark-2.1 fork with 20k+ number of rows in the table, reducing load time to 4s.