[SPARK-15373][WEB UI] Spark UI should show consistent timezones. #13158

dongjoon-hyun · 2016-05-18T00:30:38Z

What changes were proposed in this pull request?

Currently, SparkUI shows two timezones in a single page when the timezone of browser is different from the server JVM timezone. The following is an example on Databricks CE which uses 'Etc/UTC' timezone.

The time of submitted column of list and pop-up description shows 2016/05/18 00:03:07
The time of timeline chart shows 2016/05/17 17:03:07.

This PR fixes the timeline chart to use the same timezone by the followings.

Upgrade vis from 3.9.0(2015-01-16) to 4.16.1(2016-04-18)
Override moment of vis to get offset
Update AllJobsPage, JobPage, and StagePage.

How was this patch tested?

Manual. Run the following command and see the Spark UI's event timelines.

$ SPARK_SUBMIT_OPTS="-Dscala.usejavacp=true -Duser.timezone=Etc/UTC" bin/spark-submit --class org.apache.spark.repl.Main
...
scala> sql("select 1").head

dongjoon-hyun · 2016-05-18T00:32:03Z

cc @davies

SparkQA · 2016-05-18T02:28:12Z

Test build #58729 has finished for PR 13158 at commit d1bdb32.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2016-05-18T05:51:19Z

cc @andrewor14 @zsxwing

srowen · 2016-05-18T08:53:19Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

@@ -172,6 +172,8 @@ private[ui] class AllJobsPage(parent: JobsTab) extends WebUIPage("") {

    val jobEventJsonAsStrSeq = makeJobEvent(jobs)
    val executorEventJsonAsStrSeq = makeExecutorEvent(executors)
+    val offset = TimeZone.getTimeZone(System.getProperty("user.timezone"))


I think this is what TimeZone.getDefault does, and it takes care of a few other details

Thank you for review, @srowen . I'll replace that.

dongjoon-hyun · 2016-05-18T09:47:31Z

@srowen . I updated the PR to use TimeZone.getDefault(). Thank you again.

srowen · 2016-05-18T09:49:54Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala

@@ -172,6 +172,7 @@ private[ui] class AllJobsPage(parent: JobsTab) extends WebUIPage("") {

    val jobEventJsonAsStrSeq = makeJobEvent(jobs)
    val executorEventJsonAsStrSeq = makeExecutorEvent(executors)
+    val offset = TimeZone.getDefault().getOffset(System.currentTimeMillis()) / 1000 / 60


How about getRawOffset here? I think that's what you are looking for?

Ur, I think you might miss my previous comment about this . Or, did I really miss something?

Since getRawOffset does not consider Daylight Saving Time, I needed to use the above.

For example, the second one is correct.

scala> java.util.TimeZone.getTimeZone("America/New_York").getRawOffset() res0: Int = -18000000 scala> java.util.TimeZone.getTimeZone("America/New_York").getOffset(System.currentTimeMillis) res1: Int = -14400000

Oh I did miss that yet. THis LGTM

nit: could you add a new method for this line in UIUtils and reuse it?

Sure. That would be great. I'll refactor them like this.

def getTimeZoneOffset() : Int = TimeZone.getDefault().getOffset(System.currentTimeMillis()) / 1000 / 60

SparkQA · 2016-05-18T11:36:32Z

Test build #58767 has finished for PR 13158 at commit 38525b3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-05-18T16:05:21Z

retest this please

SparkQA · 2016-05-18T17:45:41Z

Test build #58789 has finished for PR 13158 at commit 38525b3.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-05-18T17:48:36Z

retest this please

zsxwing · 2016-05-18T17:50:07Z

@dongjoon-hyun could you revert the changes to the file permission?

dongjoon-hyun · 2016-05-18T17:51:21Z

Oh, did I do that? Sorry, but which file did I do that?

dongjoon-hyun · 2016-05-18T17:53:40Z

I see. The js files. I'll fix them. Thank you.

zsxwing · 2016-05-18T17:59:48Z

@dongjoon-hyun looks pretty good. Let me try locally.

dongjoon-hyun · 2016-05-18T18:03:31Z

Oh, wait a moment. I'm testing on refactored method. I'll update the PR very soon again.

dongjoon-hyun · 2016-05-18T18:06:38Z

I'm done. You can test locally now with the up-to-date code.
Thank you for review and local testing, @zsxwing !

zsxwing · 2016-05-18T18:13:52Z

LGTM pending tests

SparkQA · 2016-05-18T19:46:01Z

Test build #58800 has finished for PR 13158 at commit 38525b3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-18T19:51:57Z

Test build #58804 has finished for PR 13158 at commit 206171a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-18T20:14:15Z

Test build #58801 has finished for PR 13158 at commit e170041.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-18T20:25:14Z

Test build #58803 has finished for PR 13158 at commit 0fa7d5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-18T22:07:08Z

Test build #2993 has finished for PR 13158 at commit 206171a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-05-18T22:10:48Z

Hi, @zsxwing .
Finally, it passes the Jenkins test. :)

srowen · 2016-05-18T22:20:30Z

Merged to master/2.0

## What changes were proposed in this pull request? Currently, SparkUI shows two timezones in a single page when the timezone of browser is different from the server JVM timezone. The following is an example on Databricks CE which uses 'Etc/UTC' timezone. - The time of `submitted` column of list and pop-up description shows `2016/05/18 00:03:07` - The time of `timeline chart` shows `2016/05/17 17:03:07`. ![Different Timezone](https://issues.apache.org/jira/secure/attachment/12804553/12804553_timezone.png) This PR fixes the **timeline chart** to use the same timezone by the followings. - Upgrade `vis` from 3.9.0(2015-01-16) to 4.16.1(2016-04-18) - Override `moment` of `vis` to get `offset` - Update `AllJobsPage`, `JobPage`, and `StagePage`. ## How was this patch tested? Manual. Run the following command and see the Spark UI's event timelines. ``` $ SPARK_SUBMIT_OPTS="-Dscala.usejavacp=true -Duser.timezone=Etc/UTC" bin/spark-submit --class org.apache.spark.repl.Main ... scala> sql("select 1").head ``` Author: Dongjoon Hyun <dongjoon@apache.org> Closes #13158 from dongjoon-hyun/SPARK-15373. (cherry picked from commit cc6a47d) Signed-off-by: Sean Owen <sowen@cloudera.com>

dongjoon-hyun · 2016-05-18T22:22:20Z

Oh, thank you, @srowen !

## What changes were proposed in this pull request? SPARK-15373 (#13158) updated the version of vis.js to 4.16.1. As of 4.0.0, some class was renamed like 'timeline to vis-timeline' but that ticket didn't care and now style is broken. In this PR, I've restored the style by modifying `timeline-view.css` and `timeline-view.js`. ## How was this patch tested? manual tests. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) * Before <img width="1258" alt="2016-09-01 1 38 31" src="https://cloud.githubusercontent.com/assets/4736016/18141311/fddf1bac-6ff3-11e6-935f-28b389073b39.png"> * After <img width="1256" alt="2016-09-01 3 30 19" src="https://cloud.githubusercontent.com/assets/4736016/18141394/49af65dc-6ff4-11e6-8640-70e20300f3c3.png"> Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #14900 from sarutak/SPARK-17342. (cherry picked from commit 2ab8dbd) Signed-off-by: Sean Owen <sowen@cloudera.com>

## What changes were proposed in this pull request? SPARK-15373 (apache#13158) updated the version of vis.js to 4.16.1. As of 4.0.0, some class was renamed like 'timeline to vis-timeline' but that ticket didn't care and now style is broken. In this PR, I've restored the style by modifying `timeline-view.css` and `timeline-view.js`. ## How was this patch tested? manual tests. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) * Before <img width="1258" alt="2016-09-01 1 38 31" src="https://cloud.githubusercontent.com/assets/4736016/18141311/fddf1bac-6ff3-11e6-935f-28b389073b39.png"> * After <img width="1256" alt="2016-09-01 3 30 19" src="https://cloud.githubusercontent.com/assets/4736016/18141394/49af65dc-6ff4-11e6-8640-70e20300f3c3.png"> Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes apache#14900 from sarutak/SPARK-17342.

srowen reviewed May 18, 2016
View reviewed changes

dongjoon-hyun added 2 commits May 18, 2016 10:52

[SPARK-15373][WEB UI] Spark UI should show consistent timezones.

4f4e605

Use TimeZone.getDefault().

67c608e

Revert the file permission on vis.min.js and vis.min.css.

e170041

Make UIUtils.getTimeZoneOffset function.

0fa7d5b

Remove unused imports

206171a

asfgit closed this in cc6a47d May 18, 2016

dongjoon-hyun deleted the SPARK-15373 branch July 20, 2016 07:35

sarutak mentioned this pull request Aug 31, 2016

[WEBUI][SPARK-17342] Style of event timeline is broken #14900

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-15373][WEB UI] Spark UI should show consistent timezones. #13158

[SPARK-15373][WEB UI] Spark UI should show consistent timezones. #13158

dongjoon-hyun commented May 18, 2016 •

edited

Loading

dongjoon-hyun commented May 18, 2016

SparkQA commented May 18, 2016

davies commented May 18, 2016

srowen May 18, 2016

dongjoon-hyun May 18, 2016

dongjoon-hyun commented May 18, 2016

srowen May 18, 2016

dongjoon-hyun May 18, 2016

dongjoon-hyun May 18, 2016

srowen May 18, 2016

zsxwing May 18, 2016

dongjoon-hyun May 18, 2016

SparkQA commented May 18, 2016

dongjoon-hyun commented May 18, 2016

SparkQA commented May 18, 2016

dongjoon-hyun commented May 18, 2016

zsxwing commented May 18, 2016

dongjoon-hyun commented May 18, 2016

dongjoon-hyun commented May 18, 2016

zsxwing commented May 18, 2016

dongjoon-hyun commented May 18, 2016

dongjoon-hyun commented May 18, 2016

zsxwing commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

dongjoon-hyun commented May 18, 2016

srowen commented May 18, 2016

dongjoon-hyun commented May 18, 2016

[SPARK-15373][WEB UI] Spark UI should show consistent timezones. #13158

[SPARK-15373][WEB UI] Spark UI should show consistent timezones. #13158

Conversation

dongjoon-hyun commented May 18, 2016 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

dongjoon-hyun commented May 18, 2016

SparkQA commented May 18, 2016

davies commented May 18, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented May 18, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented May 18, 2016

dongjoon-hyun commented May 18, 2016

SparkQA commented May 18, 2016

dongjoon-hyun commented May 18, 2016

zsxwing commented May 18, 2016

dongjoon-hyun commented May 18, 2016

dongjoon-hyun commented May 18, 2016

zsxwing commented May 18, 2016

dongjoon-hyun commented May 18, 2016

dongjoon-hyun commented May 18, 2016

zsxwing commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

SparkQA commented May 18, 2016

dongjoon-hyun commented May 18, 2016

srowen commented May 18, 2016

dongjoon-hyun commented May 18, 2016

dongjoon-hyun commented May 18, 2016 •

edited

Loading