[LIVY-344] [Web UI] Create Session Log Page #25

ajbozarth · 2017-07-26T21:11:34Z

Created a Session Log page for viewing the session log.
Updated All Sessions Page and Session Page with links to the Log Page.
Log Page exists for both Interactive Sessions and Batches.

Tested Manually, screenshots below

Created a Session Log page for viewing the session log. Updated All Sessions Page and Session Page with links to the Log Page. Log Page exists for both Interactive Sessions and Batches.

ajbozarth · 2017-07-26T21:15:26Z

Feedback Question:
Currently I'm just displaying the log as a text dump, but should I address the following:

Parse the log for stdout, stderr, and YARN Diagnostics so as to separate them
Restrict the log size and implement a form of scrolling (like Spark's Log Page)

ajbozarth · 2017-07-26T21:16:53Z

@jerryshao Here's the next UI PR

jerryshao

Generally LGTM, just have two minor questions.

jerryshao · 2017-07-27T02:30:11Z

server/src/main/resources/org/apache/livy/server/ui/static/js/session-log.js

+  var type = pathArr.shift();
+  var id = pathArr.shift();
+
+  $.getJSON(location.origin + getLogPath(type, id), {from: 0, size: 100}, function(response) {


Based on the definition of from and size, here with your settings, it will always only show first 100 lines of log. I would instead use the default one. Please see the implementation of serializeLogs in SessionServlet.scala.

I put 100 here as a placeholder until we address whether we want scrolling, as for why 100, it's the default value in the function you mentioned.

My concern is actually from: 0, by default it is -1 which has different meanings.

Ah, that was so it always displays the first 100 lines, whereas using -1 it will display the last 100 lines. Either way I'll be addressing this based on my comment below since we'll need to read the whole log.

jerryshao · 2017-07-27T02:35:24Z

server/src/main/scala/org/apache/livy/server/ui/UIServlet.scala

+          <li class="active"><a href="#">{sessionPage.name}</a></li>
+      }
+      case logPage: LogPage => {
+        val sessionLink = if (logPage.sessionType == "Session") "/ui/session/" + logPage.id else "#"


does it mean that log will only be shown for interactive session?

This is the link used in the header. You'll notice when on the Log Page the Header will have the tabs: Sessions | Session # | Log clicking Session # will go to the session summary page. In the case of a Batch log the Header will look like Sessions | Batch # | Log and click Batch # will do nothing (href="#") since there's no Session Pages for Batches. I chose this option over linking back to the All Sessions Page, which was my other idea.

jerryshao · 2017-07-27T02:39:38Z

Parse the log for stdout, stderr, and YARN Diagnostics so as to separate them

This would be good if it is not difficult to achieve.

Restrict the log size and implement a form of scrolling (like Spark's Log Page)

Here based on you implementation, we will only show 100 lines of logs at most, do you think it is necessary to add a form scrolling?

ajbozarth · 2017-07-27T02:51:07Z

I'll separate out stderr/stdout/YARN tomorrow.

And by scrolling I meant a form of infinite scrolling (which I implemented on Sparks Log Page), though after your comment about 100 lines and my suggestion on splitting up the log I've realized that if stderr or stdout was too long YARN Diag won't even load. So I will probably implement a way to load the whole log no matter length and have scrolling within each of the "sub-logs"

ajbozarth · 2017-07-27T02:55:11Z

Also based on everything I'm realizing in this conversation I may toy will how the logs are stored and accessed in order to better facilitate this page. (Without changing current API functionality of course)

ajbozarth · 2017-07-28T20:45:01Z

Pushed an update addressing the log length, after a little work I found that separating out the different logs is simple, but from a UI perspective splitting them up doesn't do much without optimizing the UI for multiple logs, therefore I'm splitting off a followup JIRA to deal with splitting up the logs so we can merge more of the UI now and move forward with a 0.4 release faster.

ajbozarth · 2017-07-28T20:52:13Z

opened up followup JIRA mentioned above: https://issues.cloudera.org/browse/LIVY-387

codecov-io · 2017-07-28T21:20:26Z

Codecov Report

Merging #25 into master will decrease coverage by 0.18%.
The diff coverage is 17.39%.

@@             Coverage Diff              @@
##             master      #25      +/-   ##
============================================
- Coverage     70.95%   70.76%   -0.19%     
- Complexity      778      781       +3     
============================================
  Files            97       97              
  Lines          5343     5364      +21     
  Branches        799      805       +6     
============================================
+ Hits           3791     3796       +5     
- Misses         1020     1033      +13     
- Partials        532      535       +3

Impacted Files	Coverage Δ	Complexity Δ
...in/scala/org/apache/livy/server/ui/UIServlet.scala	`5.19% <10.52%> (+1.91%)`	`1 <0> (ø)`	⬇️
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	`61.97% <100%> (ø)`	`29 <0> (ø)`	⬇️
.../scala/org/apache/livy/server/SessionServlet.scala	`64.77% <33.33%> (-1.51%)`	`27 <0> (ø)`
.../apache/livy/rsc/driver/RSCDriverBootstrapper.java	`85.71% <0%> (-4.61%)`	`6% <0%> (ø)`
rsc/src/main/java/org/apache/livy/rsc/Utils.java	`83.33% <0%> (-2.39%)`	`16% <0%> (ø)`
rsc/src/main/java/org/apache/livy/rsc/rpc/Rpc.java	`79.28% <0%> (-0.6%)`	`12% <0%> (ø)`
...in/java/org/apache/livy/rsc/driver/JobWrapper.java	`80.64% <0%> (ø)`	`8% <0%> (+1%)`	⬆️
...server/interactive/InteractiveSessionServlet.scala	`62.29% <0%> (+0.5%)`	`5% <0%> (ø)`	⬇️
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java	`80.25% <0%> (+1.26%)`	`43% <0%> (+2%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 087adb6...9c5c1a5. Read the comment docs.

jerryshao · 2017-08-01T02:27:04Z

server/src/main/scala/org/apache/livy/server/SessionServlet.scala

    var from = fromOpt.getOrElse(-1)
+    if (size < 0) {
+      size = lines.length


Please be aware even we set size to -1, still we cannot get the whole logs, by default it will only cache the latest 200 lines (livy.cache-log.size).

Oh, that code was merged after I wrote this (this has been sitting on my computer for a while waiting for the previous PRs to merge).

I feel like 200 is a really small default size for the logs, is there a reason that number was chosen @praveenkanamarlapudi ?

As for size=-1, since it's used in js the conf value wouldn't be available so it will work well for what ever livy.cache-log.size is set to.

Yeah, it's fine for JS. I just want to mention relying on this to show the whole log is not a good idea.

@ajbozarth we use livy a lot. We have seen memory issues due to logs(as it stored in memory). So, we set the default log size to 200 to resolve the memory issues.

So I read through #14 to see exactly how livy.cache-log.size is used and it's a line limit on each of the stdout and stderr sections of the log, not the log length itself. So the default max log size is actually (livy.cache-log.size * 2) + yarnDiagnostics.size + 3 (3 for each log header) and yarnDiagnostics has no size limit that I can find. So I'm ok with that default size I think.

thanks @praveenkanamarlapudi

jerryshao · 2017-08-01T02:28:51Z

LGTM based on my limited knowledge of JS :).

ajbozarth · 2017-08-01T04:00:54Z

I pushed a small update to the TODO comment, shouldn't affect travis results

jerryshao · 2017-08-01T04:02:24Z

Thanks, will merge after travis finished.

ajbozarth · 2017-08-01T04:40:14Z

Travis failure are unrelated and passed before the update to the comment, is it ok to merge or should I re-trigger travis?

jerryshao · 2017-08-01T05:31:34Z

Yes, please try again, thank you!

jerryshao · 2017-08-01T05:31:47Z

Yes, please try again, thank you!

[LIVY-344] [Web UI] Create Session Log Page

b3b1d7e

Created a Session Log page for viewing the session log. Updated All Sessions Page and Session Page with links to the Log Page. Log Page exists for both Interactive Sessions and Batches.

jerryshao reviewed Jul 27, 2017

View reviewed changes

Updated to show entire log and added TODO for followup JIRA

9334f31

jerryshao reviewed Aug 1, 2017

View reviewed changes

nit: added followup JIRA number to TODO for reference

9c5c1a5

ajbozarth closed this Aug 1, 2017

ajbozarth reopened this Aug 1, 2017

asfgit closed this in 69300a2 Aug 1, 2017

ajbozarth deleted the log-ui branch August 1, 2017 15:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LIVY-344] [Web UI] Create Session Log Page #25

[LIVY-344] [Web UI] Create Session Log Page #25

ajbozarth commented Jul 26, 2017

ajbozarth commented Jul 26, 2017

ajbozarth commented Jul 26, 2017

jerryshao left a comment

jerryshao Jul 27, 2017

ajbozarth Jul 27, 2017

jerryshao Jul 27, 2017

ajbozarth Jul 27, 2017

jerryshao Jul 27, 2017

ajbozarth Jul 27, 2017

jerryshao commented Jul 27, 2017

ajbozarth commented Jul 27, 2017

ajbozarth commented Jul 27, 2017

ajbozarth commented Jul 28, 2017

ajbozarth commented Jul 28, 2017

codecov-io commented Jul 28, 2017 •

edited

Loading

jerryshao Aug 1, 2017

ajbozarth Aug 1, 2017

jerryshao Aug 1, 2017

praveen-kanamarlapudi Aug 1, 2017

ajbozarth Aug 1, 2017

ajbozarth Aug 1, 2017

jerryshao commented Aug 1, 2017

ajbozarth commented Aug 1, 2017

jerryshao commented Aug 1, 2017

ajbozarth commented Aug 1, 2017

jerryshao commented Aug 1, 2017

jerryshao commented Aug 1, 2017

[LIVY-344] [Web UI] Create Session Log Page #25

[LIVY-344] [Web UI] Create Session Log Page #25

Conversation

ajbozarth commented Jul 26, 2017

ajbozarth commented Jul 26, 2017

ajbozarth commented Jul 26, 2017

jerryshao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao commented Jul 27, 2017

ajbozarth commented Jul 27, 2017

ajbozarth commented Jul 27, 2017

ajbozarth commented Jul 28, 2017

ajbozarth commented Jul 28, 2017

codecov-io commented Jul 28, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao commented Aug 1, 2017

ajbozarth commented Aug 1, 2017

jerryshao commented Aug 1, 2017

ajbozarth commented Aug 1, 2017

jerryshao commented Aug 1, 2017

jerryshao commented Aug 1, 2017

codecov-io commented Jul 28, 2017 •

edited

Loading