Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-6939][Streaming][WebUI] Add timeline and histogram graphs for streaming statistics #5533

Closed
wants to merge 37 commits into from

Conversation

zsxwing
Copy link
Member

@zsxwing zsxwing commented Apr 15, 2015

This is the initial work of SPARK-6939. Not yet ready for code review. Here are the screenshots:

graph1

graph2

TODOs:

  • Display more information on mouse hover
  • Align the timeline and distribution graphs
  • Clean up the codes

@SparkQA
Copy link

SparkQA commented Apr 15, 2015

Test build #30361 has started for PR 5533 at commit dd653a1.

@tdas
Copy link
Contributor

tdas commented Apr 15, 2015

Awesome start!!! I think it can polished a little bit more. Here is some high level feedback.

  1. the graphs should be thinner, that is, less tall. It should be closer to sparklines than charts.
  2. The histogram can also be a line graph.
  3. Not sure how much it is possible, but can you make the horizontal axis of side-by-side graphs horizontal? I think the problem is the time in the timeline, they take too much vertical space. What you could try is not put the time horizontally but not on all the tick marks. Only the first
  4. The y axis in timeline looks very dense. Give more space between the tick marks. In fact both axes, the
  5. I got feedback from others that the tick labels (time, etc.) is too much information. One idea is to make the label grey, to make them less distracting.

@SparkQA
Copy link

SparkQA commented Apr 15, 2015

Test build #30361 has finished for PR 5533 at commit dd653a1.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30361/
Test FAILed.

@zsxwing
Copy link
Member Author

zsxwing commented Apr 16, 2015

@tdas, here is the new screenshot. Now it only shows the first time and the last time in the timeline. But it's weird to use a line graph for distribution. Is it what you want? Or something else?

g2

@tdas
Copy link
Contributor

tdas commented Apr 20, 2015

Some more thoughts.

  • Make the text in first column (that is " Input Rate \n avg: ... " middle-aligned in row.
  • Replace the "#" in the top figure with "records". And the "events/sec" to "records/sec" for consistency.

Also spoke offline,

  • Try making retainedBatches = 1000 and see the trendline. Probably have to remove dots/markers for each point.
  • Change bin count for distributions to 100 points. If there are less than 100 points, then binCount = number of points. And no markers for each point.
  • Make the y-axis ranges same for all delays, and they should start at zero.

@SparkQA
Copy link

SparkQA commented Apr 21, 2015

Test build #30680 has started for PR 5533 at commit b82ed1e.

@zsxwing
Copy link
Member Author

zsxwing commented Apr 21, 2015

Replace the "#" in the top figure with "records". And the "events/sec" to "records/sec" for consistency.

We have already decided to use events rather than records in #5434

Here is the new screenshot after updating the PR:

g4

@zsxwing zsxwing changed the title [SPARK-6939][Streaming][WebUI][WIP] Add timeline and histogram graphs for streaming statistics [SPARK-6939][Streaming][WebUI] Add timeline and histogram graphs for streaming statistics Apr 21, 2015
@SparkQA
Copy link

SparkQA commented Apr 21, 2015

Test build #30680 has finished for PR 5533 at commit b82ed1e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30680/
Test FAILed.

@tdas
Copy link
Contributor

tdas commented Apr 21, 2015

Fantastic! This is looking better and better.
I also got some more feedback on our company expert on data analytics graphs.

  • I was wrong about not using histogram. Histogram is the standard way to show this distribution beside the timeline. So better use that.
  • Also the standard way to show these graphs is to align the time line + distribution very close. The vertical axis of the histogram will not have numbers and will be very close to the right side of the timelines. In fact there can be a extra y-axis on the right side of the timeline, which will be the vertical axis of the histogram. I am not sure whether it can be done easily with two columns in the table. So maybe it will be good to try an alternate version where the timeline and distribution are in one column and closely aligned to each other.
  • Same horizontal axis ranges for distributions of inputs and delays
  • Color of axes = grey (same grey as the labels)

@SparkQA
Copy link

SparkQA commented Apr 22, 2015

Test build #30735 has started for PR 5533 at commit ddf741a.

@SparkQA
Copy link

SparkQA commented Apr 22, 2015

Test build #30735 has finished for PR 5533 at commit ddf741a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30735/
Test FAILed.

@zsxwing
Copy link
Member Author

zsxwing commented Apr 23, 2015

New screenshot:
g5

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30833 has started for PR 5533 at commit d78672a.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30833 has finished for PR 5533 at commit d78672a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30833/
Test PASSed.

@zsxwing
Copy link
Member Author

zsxwing commented Apr 23, 2015

Another screenshot as per our discussion offline (my local branch):
g6

@SparkQA
Copy link

SparkQA commented Apr 27, 2015

Test build #31008 has started for PR 5533 at commit d78672a.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31838/
Test FAILed.

* Find the best `TimeUnit` for converting milliseconds to a friendly string. Return the value
* after converting, also with its TimeUnit.
*/
def normalizeDuration(milliseconds: Long): (Double, TimeUnit) = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have utility methods for this in Utils, no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but this one requires normalizing and getting the unit by which it is normalized. And later we will use the same unit to normalize others time intervals. This is so that we can set the max Y-axis range and corresponding time unit for all the histograms and then map all other values using the same unit.

@tdas
Copy link
Contributor

tdas commented May 5, 2015

@zsxwing I merged the Direct Kafka patch before this because it is easier for you update this PR. Mind updating it :) Will merge as soon as you update.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31880 has started for PR 5533 at commit 9f7cd19.

@SparkQA
Copy link

SparkQA commented May 5, 2015

Test build #31880 has finished for PR 5533 at commit 9f7cd19.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public class EnumUtil

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31880/
Test PASSed.

@zsxwing
Copy link
Member Author

zsxwing commented May 5, 2015

Fixed the conflicts

@tdas
Copy link
Contributor

tdas commented May 5, 2015

Merging this! Thanks @zsxwing for all the effort!

asfgit pushed a commit that referenced this pull request May 5, 2015
…r streaming statistics

This is the initial work of SPARK-6939. Not yet ready for code review. Here are the screenshots:

![graph1](https://cloud.githubusercontent.com/assets/1000778/7165766/465942e0-e3dc-11e4-9b05-c184b09d75dc.png)

![graph2](https://cloud.githubusercontent.com/assets/1000778/7165779/53f13f34-e3dc-11e4-8714-a4a75b7e09ff.png)

TODOs:
- [x] Display more information on mouse hover
- [x] Align the timeline and distribution graphs
- [x] Clean up the codes

Author: zsxwing <zsxwing@gmail.com>

Closes #5533 from zsxwing/SPARK-6939 and squashes the following commits:

9f7cd19 [zsxwing] Merge branch 'master' into SPARK-6939
deacc3f [zsxwing] Remove unused import
cd03424 [zsxwing] Fix .rat-excludes
70cc87d [zsxwing] Streaming Scheduling Delay => Scheduling Delay
d457277 [zsxwing] Fix UIUtils in BatchPage
b3f303e [zsxwing] Add comments for unclear classes and methods
ff0bff8 [zsxwing] Make InputDStream.name private[streaming]
cc392c5 [zsxwing] Merge branch 'master' into SPARK-6939
e275e23 [zsxwing] Move time related methods to Streaming's UIUtils
d5d86f6 [zsxwing] Fix incorrect lastErrorTime
3be4b7a [zsxwing] Use InputInfo
b50fa32 [zsxwing] Jump to the batch page when clicking a point in the timeline graphs
203605d [zsxwing] Merge branch 'master' into SPARK-6939
74307cf [zsxwing] Reuse the data for histogram graphs to reduce the page size
2586916 [zsxwing] Merge branch 'master' into SPARK-6939
70d8533 [zsxwing] Remove BatchInfo.numRecords and a few renames
7bbdc0a [zsxwing] Hide the receiver sub table if no receiver
a2972e9 [zsxwing] Add some ui tests for StreamingPage
fd03ad0 [zsxwing] Add a test to verify no memory leak
4a8f886 [zsxwing] Merge branch 'master' into SPARK-6939
18607a1 [zsxwing] Merge branch 'master' into SPARK-6939
d0b0aec [zsxwing] Clean up the codes
a459f49 [zsxwing] Add a dash line to processing time graphs
8e4363c [zsxwing] Prepare for the demo
c81a1ee [zsxwing] Change time unit in the graphs automatically
4c0b43f [zsxwing] Update Streaming UI
04c7500 [zsxwing] Make the server and client use the same timezone
fed8219 [zsxwing] Move the x axis at the top and show a better tooltip
c23ce10 [zsxwing] Make two graphs close
d78672a [zsxwing] Make the X axis use the same range
881c907 [zsxwing] Use histogram for distribution
5688702 [zsxwing] Fix the unit test
ddf741a [zsxwing] Fix the unit test
ad93295 [zsxwing] Remove unnecessary codes
a0458f9 [zsxwing] Clean the codes
b82ed1e [zsxwing] Update the graphs as per comments
dd653a1 [zsxwing] Add timeline and histogram graphs for streaming statistics

(cherry picked from commit 489700c)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 489700c May 5, 2015
asfgit pushed a commit that referenced this pull request May 5, 2015
…s to docs

The default value will be changed to `1000` in #5533. So here I just used `1000`.

Author: zsxwing <zsxwing@gmail.com>

Closes #5899 from zsxwing/SPARK-7351 and squashes the following commits:

e1ec515 [zsxwing] [SPARK-7351][Streaming][Docs] Add spark.streaming.ui.retainedBatches to docs
asfgit pushed a commit that referenced this pull request May 5, 2015
…s to docs

The default value will be changed to `1000` in #5533. So here I just used `1000`.

Author: zsxwing <zsxwing@gmail.com>

Closes #5899 from zsxwing/SPARK-7351 and squashes the following commits:

e1ec515 [zsxwing] [SPARK-7351][Streaming][Docs] Add spark.streaming.ui.retainedBatches to docs

(cherry picked from commit fec7b29)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@zsxwing zsxwing deleted the SPARK-6939 branch May 5, 2015 20:57
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
…r streaming statistics

This is the initial work of SPARK-6939. Not yet ready for code review. Here are the screenshots:

![graph1](https://cloud.githubusercontent.com/assets/1000778/7165766/465942e0-e3dc-11e4-9b05-c184b09d75dc.png)

![graph2](https://cloud.githubusercontent.com/assets/1000778/7165779/53f13f34-e3dc-11e4-8714-a4a75b7e09ff.png)

TODOs:
- [x] Display more information on mouse hover
- [x] Align the timeline and distribution graphs
- [x] Clean up the codes

Author: zsxwing <zsxwing@gmail.com>

Closes apache#5533 from zsxwing/SPARK-6939 and squashes the following commits:

9f7cd19 [zsxwing] Merge branch 'master' into SPARK-6939
deacc3f [zsxwing] Remove unused import
cd03424 [zsxwing] Fix .rat-excludes
70cc87d [zsxwing] Streaming Scheduling Delay => Scheduling Delay
d457277 [zsxwing] Fix UIUtils in BatchPage
b3f303e [zsxwing] Add comments for unclear classes and methods
ff0bff8 [zsxwing] Make InputDStream.name private[streaming]
cc392c5 [zsxwing] Merge branch 'master' into SPARK-6939
e275e23 [zsxwing] Move time related methods to Streaming's UIUtils
d5d86f6 [zsxwing] Fix incorrect lastErrorTime
3be4b7a [zsxwing] Use InputInfo
b50fa32 [zsxwing] Jump to the batch page when clicking a point in the timeline graphs
203605d [zsxwing] Merge branch 'master' into SPARK-6939
74307cf [zsxwing] Reuse the data for histogram graphs to reduce the page size
2586916 [zsxwing] Merge branch 'master' into SPARK-6939
70d8533 [zsxwing] Remove BatchInfo.numRecords and a few renames
7bbdc0a [zsxwing] Hide the receiver sub table if no receiver
a2972e9 [zsxwing] Add some ui tests for StreamingPage
fd03ad0 [zsxwing] Add a test to verify no memory leak
4a8f886 [zsxwing] Merge branch 'master' into SPARK-6939
18607a1 [zsxwing] Merge branch 'master' into SPARK-6939
d0b0aec [zsxwing] Clean up the codes
a459f49 [zsxwing] Add a dash line to processing time graphs
8e4363c [zsxwing] Prepare for the demo
c81a1ee [zsxwing] Change time unit in the graphs automatically
4c0b43f [zsxwing] Update Streaming UI
04c7500 [zsxwing] Make the server and client use the same timezone
fed8219 [zsxwing] Move the x axis at the top and show a better tooltip
c23ce10 [zsxwing] Make two graphs close
d78672a [zsxwing] Make the X axis use the same range
881c907 [zsxwing] Use histogram for distribution
5688702 [zsxwing] Fix the unit test
ddf741a [zsxwing] Fix the unit test
ad93295 [zsxwing] Remove unnecessary codes
a0458f9 [zsxwing] Clean the codes
b82ed1e [zsxwing] Update the graphs as per comments
dd653a1 [zsxwing] Add timeline and histogram graphs for streaming statistics
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
…s to docs

The default value will be changed to `1000` in apache#5533. So here I just used `1000`.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#5899 from zsxwing/SPARK-7351 and squashes the following commits:

e1ec515 [zsxwing] [SPARK-7351][Streaming][Docs] Add spark.streaming.ui.retainedBatches to docs
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
…r streaming statistics

This is the initial work of SPARK-6939. Not yet ready for code review. Here are the screenshots:

![graph1](https://cloud.githubusercontent.com/assets/1000778/7165766/465942e0-e3dc-11e4-9b05-c184b09d75dc.png)

![graph2](https://cloud.githubusercontent.com/assets/1000778/7165779/53f13f34-e3dc-11e4-8714-a4a75b7e09ff.png)

TODOs:
- [x] Display more information on mouse hover
- [x] Align the timeline and distribution graphs
- [x] Clean up the codes

Author: zsxwing <zsxwing@gmail.com>

Closes apache#5533 from zsxwing/SPARK-6939 and squashes the following commits:

9f7cd19 [zsxwing] Merge branch 'master' into SPARK-6939
deacc3f [zsxwing] Remove unused import
cd03424 [zsxwing] Fix .rat-excludes
70cc87d [zsxwing] Streaming Scheduling Delay => Scheduling Delay
d457277 [zsxwing] Fix UIUtils in BatchPage
b3f303e [zsxwing] Add comments for unclear classes and methods
ff0bff8 [zsxwing] Make InputDStream.name private[streaming]
cc392c5 [zsxwing] Merge branch 'master' into SPARK-6939
e275e23 [zsxwing] Move time related methods to Streaming's UIUtils
d5d86f6 [zsxwing] Fix incorrect lastErrorTime
3be4b7a [zsxwing] Use InputInfo
b50fa32 [zsxwing] Jump to the batch page when clicking a point in the timeline graphs
203605d [zsxwing] Merge branch 'master' into SPARK-6939
74307cf [zsxwing] Reuse the data for histogram graphs to reduce the page size
2586916 [zsxwing] Merge branch 'master' into SPARK-6939
70d8533 [zsxwing] Remove BatchInfo.numRecords and a few renames
7bbdc0a [zsxwing] Hide the receiver sub table if no receiver
a2972e9 [zsxwing] Add some ui tests for StreamingPage
fd03ad0 [zsxwing] Add a test to verify no memory leak
4a8f886 [zsxwing] Merge branch 'master' into SPARK-6939
18607a1 [zsxwing] Merge branch 'master' into SPARK-6939
d0b0aec [zsxwing] Clean up the codes
a459f49 [zsxwing] Add a dash line to processing time graphs
8e4363c [zsxwing] Prepare for the demo
c81a1ee [zsxwing] Change time unit in the graphs automatically
4c0b43f [zsxwing] Update Streaming UI
04c7500 [zsxwing] Make the server and client use the same timezone
fed8219 [zsxwing] Move the x axis at the top and show a better tooltip
c23ce10 [zsxwing] Make two graphs close
d78672a [zsxwing] Make the X axis use the same range
881c907 [zsxwing] Use histogram for distribution
5688702 [zsxwing] Fix the unit test
ddf741a [zsxwing] Fix the unit test
ad93295 [zsxwing] Remove unnecessary codes
a0458f9 [zsxwing] Clean the codes
b82ed1e [zsxwing] Update the graphs as per comments
dd653a1 [zsxwing] Add timeline and histogram graphs for streaming statistics
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
…s to docs

The default value will be changed to `1000` in apache#5533. So here I just used `1000`.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#5899 from zsxwing/SPARK-7351 and squashes the following commits:

e1ec515 [zsxwing] [SPARK-7351][Streaming][Docs] Add spark.streaming.ui.retainedBatches to docs
@hotienvu
Copy link

Hi, I'm running Spark job UI under virtual hosts since I can't open any port rather than 80 for security reason. Using spark.ui.proxyBase has been working at least for static files but doesn't work with the Streaming tab since it pulling static files from /static/streaming. My current workaround is copy org/apache/spark/streaming/ui/static/ to org/apache/spark/ui/static/streaming but this introduces duplication.

@zsxwing
Copy link
Member Author

zsxwing commented Jun 15, 2015

@hotienvu could you open a JIRA and describe more details? Such as the error information and the invalid link?

nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
…r streaming statistics

This is the initial work of SPARK-6939. Not yet ready for code review. Here are the screenshots:

![graph1](https://cloud.githubusercontent.com/assets/1000778/7165766/465942e0-e3dc-11e4-9b05-c184b09d75dc.png)

![graph2](https://cloud.githubusercontent.com/assets/1000778/7165779/53f13f34-e3dc-11e4-8714-a4a75b7e09ff.png)

TODOs:
- [x] Display more information on mouse hover
- [x] Align the timeline and distribution graphs
- [x] Clean up the codes

Author: zsxwing <zsxwing@gmail.com>

Closes apache#5533 from zsxwing/SPARK-6939 and squashes the following commits:

9f7cd19 [zsxwing] Merge branch 'master' into SPARK-6939
deacc3f [zsxwing] Remove unused import
cd03424 [zsxwing] Fix .rat-excludes
70cc87d [zsxwing] Streaming Scheduling Delay => Scheduling Delay
d457277 [zsxwing] Fix UIUtils in BatchPage
b3f303e [zsxwing] Add comments for unclear classes and methods
ff0bff8 [zsxwing] Make InputDStream.name private[streaming]
cc392c5 [zsxwing] Merge branch 'master' into SPARK-6939
e275e23 [zsxwing] Move time related methods to Streaming's UIUtils
d5d86f6 [zsxwing] Fix incorrect lastErrorTime
3be4b7a [zsxwing] Use InputInfo
b50fa32 [zsxwing] Jump to the batch page when clicking a point in the timeline graphs
203605d [zsxwing] Merge branch 'master' into SPARK-6939
74307cf [zsxwing] Reuse the data for histogram graphs to reduce the page size
2586916 [zsxwing] Merge branch 'master' into SPARK-6939
70d8533 [zsxwing] Remove BatchInfo.numRecords and a few renames
7bbdc0a [zsxwing] Hide the receiver sub table if no receiver
a2972e9 [zsxwing] Add some ui tests for StreamingPage
fd03ad0 [zsxwing] Add a test to verify no memory leak
4a8f886 [zsxwing] Merge branch 'master' into SPARK-6939
18607a1 [zsxwing] Merge branch 'master' into SPARK-6939
d0b0aec [zsxwing] Clean up the codes
a459f49 [zsxwing] Add a dash line to processing time graphs
8e4363c [zsxwing] Prepare for the demo
c81a1ee [zsxwing] Change time unit in the graphs automatically
4c0b43f [zsxwing] Update Streaming UI
04c7500 [zsxwing] Make the server and client use the same timezone
fed8219 [zsxwing] Move the x axis at the top and show a better tooltip
c23ce10 [zsxwing] Make two graphs close
d78672a [zsxwing] Make the X axis use the same range
881c907 [zsxwing] Use histogram for distribution
5688702 [zsxwing] Fix the unit test
ddf741a [zsxwing] Fix the unit test
ad93295 [zsxwing] Remove unnecessary codes
a0458f9 [zsxwing] Clean the codes
b82ed1e [zsxwing] Update the graphs as per comments
dd653a1 [zsxwing] Add timeline and histogram graphs for streaming statistics
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
…s to docs

The default value will be changed to `1000` in apache#5533. So here I just used `1000`.

Author: zsxwing <zsxwing@gmail.com>

Closes apache#5899 from zsxwing/SPARK-7351 and squashes the following commits:

e1ec515 [zsxwing] [SPARK-7351][Streaming][Docs] Add spark.streaming.ui.retainedBatches to docs
@hotienvu
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants