[SPARK-10873] Support column sort and search for History Server. #10648

zhuoliu · 2016-01-07T19:31:33Z

[SPARK-10873] Support column sort and search for History Server using jQuery DataTable and REST API. Before this commit, the history server was generated hard-coded html and can not support search, also, the sorting was disabled if there is any application that has more than one attempt. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve user experience.

Create the historypage-template.html for displaying application information in datables.
historypage.js uses jQuery to access the data from /api/v1/applications REST API, and use DataTable to display each application's information. For application that has more than one attempt, the RowsGroup is used to merge such entries while at the same time supporting sort and search.
"duration" and "lastUpdated" rest API are added to application's "attempts".
External javascirpt and css files for datatables, RowsGroup and jquery plugins are added with licenses clarified.

Snapshots for how it looks like now:

History page view:

Search:

Sort by started time:

… datatable and REST API.

vanzin · 2016-01-07T19:34:37Z

ok to test

vanzin · 2016-01-07T19:35:35Z

Haven't looked at the code, but the zebra striping is messed up when you have multiple attempts.

tgravescs · 2016-01-07T19:46:37Z

the striping is the same as the current history server (1.5) when you have multiple attempts. The only difference I see if the striping on the first 2 columns is disabled on the current version. I'm fine with changing it if its not to much hassle.

SparkQA · 2016-01-07T19:51:26Z

Test build #48961 has finished for PR 10648 at commit 808558f.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

zhuoliu · 2016-01-07T20:18:58Z

core/src/main/resources/org/apache/spark/ui/static/dataTables.rowsGroup.js

+	search: 'applied',
+}
+
+GroupedColumnsOrderDir = 'desc'; // change


This external js file fro RowsGroup is using MIT license. I modified this line to set the default order from "asc" to "desc". This change is necessary that by default the latest applications will be displayed on the first page.
Also, I commented line 181out so that it will not disable sorting on the last 6 columns.
Do we need to add an additional license for this two minor changes?

SparkQA · 2016-01-07T20:46:46Z

Test build #48966 has finished for PR 10648 at commit f1f8feb.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-07T21:13:24Z

Test build #48968 has finished for PR 10648 at commit 2abe73c.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

zhuoliu · 2016-01-07T21:33:38Z

Hi @vanzin , I added two new fields (duration Long, lastUpdated Date) into ApplicationAttemptInfo in api.scala.
The jenkins compile is complaining about mima binary check failures. Did you have a clue for this? Thank you!

[info] Compiling 16 Scala sources and 2 Java sources to /home/jenkins/workspace/SparkPullRequestBuilder@3/sql/core/target/scala-2.10/classes...
[info] spark-core: found 2 potential binary incompatibilities (filtered 621)
[error]  * synthetic method <init>$default$5()Boolean in object org.apache.spark.status.api.v1.ApplicationAttemptInfo does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.<init>$default$5")
[error]  * method this(scala.Option,java.util.Date,java.util.Date,java.lang.String,Boolean)Unit in class org.apache.spark.status.api.v1.ApplicationAttemptInfo does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.ApplicationAttemptInfo.this")
...
...
[error] (core/*:mimaReportBinaryIssues) spark-core: Binary compatibility check failed!
[error] Total time: 111 s, completed Jan 7, 2016 1:13:24 PM
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@3/dev/mima ; received return code 1

vanzin · 2016-01-07T21:38:22Z

Since the constructor is private anyway, it's fine to just do what the error message says and add the exclusions to MimaExcludes.scala.

zhuoliu · 2016-01-07T22:23:22Z

Hi @vanzin , thanks for the info.
BTW, I have fixed the issue you mentioned about messed zebra striping. Now the zebra striping is exactly as what we have originally (the first two columns display pure white if any application has more than one attempt). Please check the updated snapshot figures above.

SparkQA · 2016-01-07T23:46:43Z

Test build #48976 has finished for PR 10648 at commit 4322851.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…nger applicable href link test

SparkQA · 2016-01-08T22:05:31Z

Test build #49025 has finished for PR 10648 at commit d7cc895.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-12T15:14:35Z

@vanzin @srowen @JoshRosen @steveloughran @andrewor14 @rxin I want to get input on what people think of using the datatables? If everyone is ok with it we can review in more detail. I think starting out with history page and then perhaps changing all the UI to use datatables to its consistent across the board.

vanzin · 2016-01-13T01:27:33Z

I don't have a problem with it if the license are ok; I just don't have the cycles right now to review the code. :-(

andrewor14 · 2016-01-13T20:36:58Z

Screenshot looks amazing, thanks @zhuoliu!

I think @JoshRosen has some opinions on using a database for the history server.

steveloughran · 2016-01-14T04:15:40Z

I have some opinions too, as with YARN timeline service integration, it's essentially hooked up to a database, both for publishing and retrieval. It might be interesting to consider a dual mode, where things are published to HDFS/s3/WASB during a run (availability will match that of the analytics work itself), and for retrieval, after any initial load & parse, persist in a DB. You could do that with the current FS records, extracting useful summary data "completed" "incompatible-version" in the process.

tgravescs · 2016-01-19T15:41:18Z

The use of a database seems like it could be separate from this pr. This pr does change it to use the rest api but the main change here is the use of datatables to allow for sorting, searching, etc.

I haven't heard any issue with using that so I'll go ahead with a more thorough review. If I missed someone concern with that please let me know ASAP.

tgravescs · 2016-01-21T16:46:58Z

Jenkins, test this please

SparkQA · 2016-01-21T18:41:11Z

Test build #49878 has finished for PR 10648 at commit ad6ce01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-21T21:59:03Z

core/src/main/resources/org/apache/spark/ui/static/historypage.js

+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+


can you add a comment saying this is same as UIUtils.formatDuration

SparkQA · 2016-01-22T05:48:52Z

Test build #49899 has finished for PR 10648 at commit b62d77b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-26T15:58:40Z

LICENSE

@@ -291,3 +291,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
     (MIT License) dagre-d3 (https://github.com/cpettitt/dagre-d3)
     (MIT License) sorttable (https://github.com/stuartlangridge/sorttable)
     (MIT License) boto (https://github.com/boto/boto/blob/develop/LICENSE)
+     (MIT/GPL License) datatables (http://datatables.net/license)


the version we pull in is MIT licenses so lets removed GPL license

It doesn't matter much since it's MIT or GPL, and I think this 'notation' was started by the automated output of a Maven plugin.

This is sufficient for MIT-licensed code. I'm a bit concerned about pulling in a lot of third party source just for this feature, but hey.

It looks like you're pulling in an inconsistent set of jquery files though. 1.11.1 vs 1.10.4? or maybe I'm misreading this. They show up as 1-line diffs.

tgravescs · 2016-01-26T16:47:53Z

LICENSE

@@ -291,3 +291,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
     (MIT License) dagre-d3 (https://github.com/cpettitt/dagre-d3)
     (MIT License) sorttable (https://github.com/stuartlangridge/sorttable)
     (MIT License) boto (https://github.com/boto/boto/blob/develop/LICENSE)
+     (MIT/GPL License) datatables (http://datatables.net/license)
+     (MIT/GPL License) blockUI (http://jquery.malsup.com/block/)


similar here since it allows user to choose MIT or GPL lets just put MIT.

SparkQA · 2016-01-26T18:05:14Z

Test build #50109 has finished for PR 10648 at commit 638a903.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-26T19:05:58Z

core/src/main/resources/org/apache/spark/ui/static/historypage.js

+
+    historySummary = $("#history-summary");
+    searchString = JSON.stringify(historySummary["context"]["location"]["search"]);
+    requestedIncomplete = searchString.indexOf("true") >= 0 ? true : false;


I think this would be better if we specifically look for the paramter showIncomplete=true or false here

SparkQA · 2016-01-27T03:38:37Z

Test build #50155 has finished for PR 10648 at commit d7e679e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-27T15:32:09Z

core/src/main/resources/org/apache/spark/ui/static/historypage.js

+}
+
+function getParameterByName(name, searchString) {
+  name = name.replace(/[\[]/, "\\[").replace(/[\]]/, "\\]");


we don't need this.

tgravescs · 2016-01-27T17:04:12Z

The paging doesn't seem to be working can you take a look.

Then could you also line up the table indentation with the other text for "Event log directory"

SparkQA · 2016-01-27T19:33:52Z

Test build #50196 has finished for PR 10648 at commit 1d3343f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhuoliu · 2016-01-28T04:25:37Z

Hi @tgravescs , finally fixed the paging stuff in RowsGrouping. :)

SparkQA · 2016-01-28T06:25:49Z

Test build #50248 has finished for PR 10648 at commit 430d283.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-28T06:41:04Z

@tgravescs @zhuoliu are you guys interested in more UI work? I have some ideas that I never found time / people to work on ... I think they will make the UI a lot more useful.

tgravescs · 2016-01-28T14:25:19Z

@rxin I would be interested in hearing them if they make debugging easier and usability better. We'll have to prioritize it with the rest of the work we have. We would like to change most of the pages to use the datatables to keep things consistent so if its things we can do at the same time it might be easier.

tgravescs · 2016-01-28T17:43:47Z

mostly looks good a few more minor things.

can you highlight the page you are on
file a jira to have attemptid always show up in rest api so spark shell reports attempt id
when you click an application id with multiple attempts, open the latest (ie 3 if 3 attempts).

zhuoliu · 2016-01-28T17:51:02Z

JIRA filed here: https://issues.apache.org/jira/browse/SPARK-13064

tgravescs · 2016-01-28T22:37:35Z

Jenkins, test this please

SparkQA · 2016-01-29T03:04:28Z

Test build #50306 has finished for PR 10648 at commit 0c55712.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2016-01-29T13:53:30Z

Jenkins, test this please

tgravescs · 2016-01-29T16:55:10Z

+1 for the latest changes, the unit tests failures are unrelated. Going to give this a little while but if they are hung again commit this anyway since the latest changes made don't affect unit tests.

[SPARK-10873] Support column sort and search for History Server using jQuery DataTable and REST API. Before this commit, the history server was generated hard-coded html and can not support search, also, the sorting was disabled if there is any application that has more than one attempt. Supporting search and sort (over all applications rather than the 20 entries in the current page) in any case will greatly improve user experience. 1. Create the historypage-template.html for displaying application information in datables. 2. historypage.js uses jQuery to access the data from /api/v1/applications REST API, and use DataTable to display each application's information. For application that has more than one attempt, the RowsGroup is used to merge such entries while at the same time supporting sort and search. 3. "duration" and "lastUpdated" rest API are added to application's "attempts". 4. External javascirpt and css files for datatables, RowsGroup and jquery plugins are added with licenses clarified. Snapshots for how it looks like now: History page view: ![historypage](https://cloud.githubusercontent.com/assets/11683054/12184383/89bad774-b55a-11e5-84e4-b0276172976f.png) Search: ![search](https://cloud.githubusercontent.com/assets/11683054/12184385/8d3b94b0-b55a-11e5-869a-cc0ef0a4242a.png) Sort by started time: ![sort-by-started-time](https://cloud.githubusercontent.com/assets/11683054/12184387/8f757c3c-b55a-11e5-98c8-577936366566.png) Author: zhuol <zhuol@yahoo-inc.com> Closes apache#10648 from zhuoliu/10873.

[SPARK-10873] Support column sort and search for History Server using…

808558f

… datatable and REST API.

zhuoliu reviewed Jan 7, 2016
View reviewed changes

Add rat excludes for the added js and css files

f1f8feb

Minor format issue

2abe73c

Zebra View exactly as it was and fix Mima exclude

4322851

Deal with lastUpdated and duration in HS suite test, delete the no-lo…

d7cc895

…nger applicable href link test

Solve the 20 show issue, remove the redundant page index

ad6ce01

tgravescs reviewed Jan 21, 2016
View reviewed changes

Add one comment for formatDuration

b62d77b

tgravescs reviewed Jan 26, 2016
View reviewed changes

Deal with licenses

638a903

tgravescs reviewed Jan 26, 2016
View reviewed changes

Licenses for cookies & mustache, showInComplete, remove org page index.

d7e679e

tgravescs reviewed Jan 27, 2016
View reviewed changes

Removed page num in link, fix paging, adjust indention

1d3343f

Fix a tough bug in DataGroup.js about paging.

430d283

Highlight current page button, link to the most recent attempt

0c55712

asfgit closed this in e4c1162 Jan 29, 2016

sarutak mentioned this pull request Mar 11, 2016

[SPARK-10775][UI]add search keywords in history page ui #8885

Closed

[SPARK-10873] Support column sort and search for History Server. #10648

[SPARK-10873] Support column sort and search for History Server. #10648

Conversation

zhuoliu commented Jan 7, 2016

vanzin commented Jan 7, 2016

vanzin commented Jan 7, 2016

tgravescs commented Jan 7, 2016

SparkQA commented Jan 7, 2016

zhuoliu Jan 7, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 7, 2016

SparkQA commented Jan 7, 2016

zhuoliu commented Jan 7, 2016

vanzin commented Jan 7, 2016

zhuoliu commented Jan 7, 2016

SparkQA commented Jan 7, 2016

SparkQA commented Jan 8, 2016

tgravescs commented Jan 12, 2016

vanzin commented Jan 13, 2016

andrewor14 commented Jan 13, 2016

steveloughran commented Jan 14, 2016

tgravescs commented Jan 19, 2016

tgravescs commented Jan 21, 2016

SparkQA commented Jan 21, 2016

tgravescs Jan 21, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 22, 2016

tgravescs Jan 26, 2016

Choose a reason for hiding this comment

srowen Jan 26, 2016

Choose a reason for hiding this comment

tgravescs Jan 26, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 26, 2016

tgravescs Jan 26, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 27, 2016

tgravescs Jan 27, 2016

Choose a reason for hiding this comment

tgravescs commented Jan 27, 2016

SparkQA commented Jan 27, 2016

zhuoliu commented Jan 28, 2016

SparkQA commented Jan 28, 2016

rxin commented Jan 28, 2016

tgravescs commented Jan 28, 2016

tgravescs commented Jan 28, 2016

zhuoliu commented Jan 28, 2016

tgravescs commented Jan 28, 2016

SparkQA commented Jan 29, 2016

tgravescs commented Jan 29, 2016

tgravescs commented Jan 29, 2016