[SPARK-3454] separate json endpoints for data in the UI #4435

squito · 2015-02-06T21:35:40Z

Exposes data available in the UI as json over http. Key points:

new endpoints, handled independently of existing XyzPage classes. Root entrypoint is JsonRootResource
Uses jersey + jackson for routing & converting POJOs into json
tests against known results in HistoryServerSuite
also fixes some minor issues w/ the UI -- synchronizing on access to StorageListener & StorageStatusListener, and fixing some inconsistencies w/ the way we handle retained jobs & stages.

…rs in sbt

SparkQA · 2015-02-06T21:37:46Z

Test build #26945 has finished for PR 4435 at commit b252e7a.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AllApplicationsJsonRoute(val uiRoot: UIRoot) extends StatusJsonRoute[Seq[ApplicationInfo]]
- class OneApplicationJsonRoute(val uiRoot: UIRoot) extends StatusJsonRoute[ApplicationInfo]
- class MasterApplicationJsonRoute(val parent: MasterWebUI) extends StatusJsonRoute[ApplicationInfo]
- class MasterJsonRoute(parent: MasterWebUI) extends StatusJsonRoute[Seq[ApplicationInfo]]
- case class ApplicationInfo(
- case class ExecutorStageSummary(
- case class ExecutorSummary(
- case class JobData(
- case class RDDStorageInfo(
- case class RDDDataDistribution(
- case class RDDPartitionInfo(
- case class StageData(
- case class TaskData(
- case class TaskMetrics(
- case class InputMetrics(
- case class OutputMetrics(
- case class ShuffleReadMetrics(
- case class ShuffleWriteMetrics(
- class ExecutorsJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[ExecutorSummary]]
- class AllJobsJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[JobData]]
- class AllStagesJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[StageData]]
- class OneStageJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[StageData]
- class AllRDDJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[RDDStorageInfo]]
- class RDDJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[RDDStorageInfo]

squito · 2015-02-06T22:40:01Z

.rat-excludes

@@ -65,3 +65,6 @@ logs
 .*scalastyle-output.xml
 .*dependency-reduced-pom.xml
 known_translations
+json_expectation
+local-1422981759269/*
+local-1422981780767/*


is this OK? or should the apache header go in the test resource files as well?

the header shouldn't go into those files

SparkQA · 2015-02-06T22:44:25Z

Test build #26965 has finished for PR 4435 at commit bceb3a9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AllApplicationsJsonRoute(val uiRoot: UIRoot) extends StatusJsonRoute[Seq[ApplicationInfo]]
- class OneApplicationJsonRoute(val uiRoot: UIRoot) extends StatusJsonRoute[ApplicationInfo]
- class MasterApplicationJsonRoute(val parent: MasterWebUI) extends StatusJsonRoute[ApplicationInfo]
- class MasterJsonRoute(parent: MasterWebUI) extends StatusJsonRoute[Seq[ApplicationInfo]]
- case class ApplicationInfo(
- case class ExecutorStageSummary(
- case class ExecutorSummary(
- case class JobData(
- case class RDDStorageInfo(
- case class RDDDataDistribution(
- case class RDDPartitionInfo(
- case class StageData(
- case class TaskData(
- case class TaskMetrics(
- case class InputMetrics(
- case class OutputMetrics(
- case class ShuffleReadMetrics(
- case class ShuffleWriteMetrics(
- class ExecutorsJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[ExecutorSummary]]
- class AllJobsJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[JobData]]
- class AllStagesJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[StageData]]
- class OneStageJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[StageData]
- class AllRDDJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[Seq[RDDStorageInfo]]
- class RDDJsonRoute(parent: JsonRequestHandler) extends StatusJsonRoute[RDDStorageInfo]

rxin · 2015-02-06T23:55:32Z

core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala

@@ -71,6 +74,7 @@ class HistoryServer(
    protected override def doGet(req: HttpServletRequest, res: HttpServletResponse): Unit = {
      val parts = Option(req.getPathInfo()).getOrElse("").split("/")
      if (parts.length < 2) {
+        logError("bad path info!")


log the path too?

this

andrewor14 · 2015-02-19T22:47:43Z

Hi @squito thanks for working on this. Usually we require a design doc for large features such as this one. Would you mind posting one on the JIRA?

squito · 2015-02-20T15:59:16Z

Hi @andrewor14 , there is already a doc on the JIRA: https://issues.apache.org/jira/secure/attachment/12695540/sparkmonitoringjsondesign.pdf

I changed the names slightly in the actual implementation, but of course I can change the naming however we want. Let me know if you want any other info in that doc.

JoshRosen · 2015-03-09T18:16:28Z

core/src/test/scala/org/apache/spark/status/JsonRequestHandlerTest.scala

+import org.apache.spark.status.api.StageStatus
+import org.scalatest.{Matchers, FunSuite}
+
+class JsonRequestHandlerTest extends FunSuite with Matchers {


This should be named JsonRequestHandlerSuite to be consistent with the *Suite naming convention that we use for our tests.

JoshRosen · 2015-03-09T18:53:16Z

Hi @squito,

I took a quick pass through this PR and design doc. I like the overall approach here:

Structuring the hierarchy around applications / jobs / stages / tasks is a good design.
Using our existing binary-compatibility checking tools to ensure JSON compatibility is a good idea. The end-to-end "golden file" tests are good, too, since this guards against breakage unexpected changes in the JSON serialization / translation layer.
Building explicit versioning into the API from day one is a good idea.

A couple of high-level questions:

Does this work significantly differently in the HistoryServer versus a live application UI?
How do the JSON endpoints respond to invalid requests or requests for stages / jobs / tasks whose information has been garbage-collected? We should add a test to ensure that these endpoints return the right HTTP error codes.
Will adding / changing the Jackson dependency create a dependency-hell mess for us? Should we pre-emptively resort to shading or a different packaging strategy?

General code review comments:

I don't think that the public JSON response classes should be case classes: the addition of new fields will break binary compatiblity (since apply / unapply will have their signatures change, breaking users' ability to pattern-match on these objects). We've already hit this problem with SparkListenerEvent, so I think we should seek to avoid it here.
Restrict the visibility of components as much as possible. For example, JsonRoute and all of its subclasses should probably be private[ui].
Imports need to be re-organized in several files.
The WorkerJsonRoute.scala file is empty.
I haven't been following the mailing list thread about enumerations very closely, but we should figure out whether the new enum is compatible with that decision.
The debugging printlns should become log statements or be removed.
Minor code style nit, but there are a bunch of style violations here that aren't caught by our checker. In general, you need extra whitespace around operators, braces, etc, e.g. .map{case(k,v) => k should be .map { case (k, v) => k.
I see unit tests for the individual components, but it would nice to add some end-to-end tests which can also serve as usage examples. Take a look at the tests in UISeleniumSuite for inspiration. It might be nice to add a test which spins up a driver, runs some long-running tasks / jobs, and polls the REST api to get their statuses.

squito · 2015-03-09T21:27:13Z

Hi @JoshRosen , thanks for the review. Sorry for all the style details wrong -- I will take a closer look at all the code, just wanted to get some validation of the overall approach. Some answers to specific questions:

It works almost exactly the same in the HistoryServer & the live UI, though to be honest I've only done very light testing of the live UI. Also I mentioned one issue on the JIRA -- I'd like the URLs the same in the live UI & the history server, but this does mean you need to put in the app_id even when you're dealing w/ the live UI.
I have a few test cases cover missing resources & response codes for the history server (looks like I didn't push them all, sorry ...), but I suppose there should probably be more for the live UI. I certainly hadn't thought about the live UI purging old jobs & stages.
Its a good question about jackson -- the dependency was already introduced here: 1390e56f , but it might still be worth considering. Just anecdotally, I've mostly been able to upgrade jackson in the past but I've occasionally run into issues when upgrading jackson, and having different versions of the various jackson modules.
Good point on the case classes -- I will change that
also agreed on the other style issues, I will clean those things up.

Also I've got another version which uses Jersey for the internal routing, it cleans things up. I'll update with your feedback and ping you here -- will be a few days as I am in the weeds on another issue.

thanks

Conflicts: core/src/main/scala/org/apache/spark/deploy/master/Master.scala core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala core/src/main/scala/org/apache/spark/ui/storage/RDDPage.scala core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala

Conflicts: core/src/main/scala/org/apache/spark/ui/SparkUI.scala core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala

squito · 2015-05-05T02:24:15Z

Hi @pwendell , I added a section to the monitoring docs on this, including a list of endpoints. It also includes a section on the guarantees of versioning (it might be too strong -- I'll highlight the line to take a look at). And I created @Private and used it on EnumUtil.

Also @vanzin and I just realized that the constructors of all the POJOs aren't really meant to get included in the mima checks, just all the fields, so I marked all constructors as private[spark]

squito · 2015-05-05T02:26:04Z

docs/monitoring.md

+* New endpoints may be added
+* New fields may be added to existing endpoints
+* New versions of the api may be added in the future at a separate endpoint (eg., `json/v2`).  New versions are *not* required to be backwards compatible.
+* Api versions may be dropped, but only after at least one minor release of existing beside a new api version


I wasn't sure if we want to make a stronger guarantee than this ... eg. maybe we should keep json/v1 around as long as we're on Spark 1.x. But I figured I'd word it like this for now to give us more room with the initial version. Definitely open to other opinions

SparkQA · 2015-05-05T04:02:43Z

Test build #31814 has finished for PR 4435 at commit f016182.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- public class EnumUtil

SparkQA · 2015-05-05T04:07:28Z

Test build #31816 has finished for PR 4435 at commit 5e78b4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class EnumUtil
- case class Not(child: Expression) extends UnaryExpression with Predicate with ExpectsInputTypes
- case class And(left: Expression, right: Expression)
- case class Or(left: Expression, right: Expression)
- abstract class BinaryComparison extends BinaryExpression with Predicate
- trait StringRegexExpression extends ExpectsInputTypes
- trait CaseConversionExpression extends ExpectsInputTypes
- case class Substring(str: Expression, pos: Expression, len: Expression)

SparkQA · 2015-05-05T04:23:16Z

Test build #31818 has finished for PR 4435 at commit da1e35f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class EnumUtil

pwendell · 2015-05-05T10:02:13Z

core/src/main/java/org/apache/spark/JobExecutionStatus.java

@@ -17,9 +17,15 @@

 package org.apache.spark;

+import org.apache.spark.util.EnumUtil;
+
 public enum JobExecutionStatus {


Should this be private?

pwendell · 2015-05-05T10:05:43Z

Hey Imran - I had a few comments about enums I'm not sure whether we want to annotate private (I'll leave it to to you based on your intentions). Overall LGTM, feel free to pull this into Spark 1.4 if you are able to get to it in the next day or so. If not, let me know and we can talk about whether it can still go on.

squito · 2015-05-05T10:40:04Z

Those enums were meant to be public -- they are part of the returned value from some endpoints, so we want them included in the mima checks (just like all the new pojos in api.scala).

btw, I also left one inline comment on the update to the docs that got hidden when I fixed a typo later on -- I'll make that comment again just to make sure you are OK with it.

squito · 2015-05-05T10:40:46Z

docs/monitoring.md

+* New endpoints may be added
+* New fields may be added to existing endpoints
+* New versions of the api may be added in the future at a separate endpoint (eg., `json/v2`).  New versions are *not* required to be backwards compatible.
+* Api versions may be dropped, but only after at least one minor release of co-existing with a new api version


I wasn't sure if we want to make a stronger guarantee than this ... eg. maybe we should keep json/v1 around as long as we're on Spark 1.x. But I figured I'd word it like this for now to give us more room with the initial version. Definitely open to other opinions

@Private

Exposes data available in the UI as json over http. Key points: * new endpoints, handled independently of existing XyzPage classes. Root entrypoint is `JsonRootResource` * Uses jersey + jackson for routing & converting POJOs into json * tests against known results in `HistoryServerSuite` * also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages. Author: Imran Rashid <irashid@cloudera.com> Closes #4435 from squito/SPARK-3454 and squashes the following commits: da1e35f [Imran Rashid] typos etc. 5e78b4f [Imran Rashid] fix rendering problems 5ae02ad [Imran Rashid] Merge branch 'master' into SPARK-3454 f016182 [Imran Rashid] change all constructors json-pojo class constructors to be private[spark] to protect us from mima-false-positives if we add fields 3347b72 [Imran Rashid] mark EnumUtil as @Private ec140a2 [Imran Rashid] create @Private cc1febf [Imran Rashid] add docs on the metrics-as-json api cbaf287 [Imran Rashid] Merge branch 'master' into SPARK-3454 56db31e [Imran Rashid] update tests for mulit-attempt 7f3bc4e [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" 67008b4 [Imran Rashid] rats 9e51400 [Imran Rashid] style c9bae1c [Imran Rashid] handle multiple attempts per app b87cd63 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 188762c [Imran Rashid] multi-attempt 2af11e5 [Imran Rashid] Merge branch 'master' into SPARK-3454 befff0c [Imran Rashid] review feedback 14ac3ed [Imran Rashid] jersey-core needs to be explicit; move version & scope to parent pom.xml f90680e [Imran Rashid] Merge branch 'master' into SPARK-3454 dc8a7fe [Imran Rashid] style, fix errant comments acb7ef6 [Imran Rashid] fix indentation 7bf1811 [Imran Rashid] move MetricHelper so mima doesnt think its exposed; comments 9d889d6 [Imran Rashid] undo some unnecessary changes f48a7b0 [Imran Rashid] docs 52bbae8 [Imran Rashid] StorageListener & StorageStatusListener needs to synchronize internally to be thread-safe 31c79ce [Imran Rashid] asm no longer needed for SPARK_PREPEND_CLASSES b2f8b91 [Imran Rashid] @DeveloperAPI 2e19be2 [Imran Rashid] lazily convert ApplicationInfo to avoid memory overhead ba3d9d2 [Imran Rashid] upper case enums 39ac29c [Imran Rashid] move EnumUtil d2bde77 [Imran Rashid] update error handling & scoping 4a234d3 [Imran Rashid] avoid jersey-media-json-jackson b/c of potential version conflicts a157a2f [Imran Rashid] style 7bd4d15 [Imran Rashid] delete security test, since it doesnt do anything a325563 [Imran Rashid] style a9c5cf1 [Imran Rashid] undo changes superceeded by master 0c6f968 [Imran Rashid] update deps 1ed0d07 [Imran Rashid] Merge branch 'master' into SPARK-3454 4c92af6 [Imran Rashid] style f2e63ad [Imran Rashid] Merge branch 'master' into SPARK-3454 c22b11f [Imran Rashid] fix compile error 9ea682c [Imran Rashid] go back to good ol' java enums cf86175 [Imran Rashid] style d493b38 [Imran Rashid] Merge branch 'master' into SPARK-3454 f05ae89 [Imran Rashid] add in ExecutorSummaryInfo for MiMa :( 101a698 [Imran Rashid] style d2ef58d [Imran Rashid] revert changes that had HistoryServer refresh the application listing more often b136e39 [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" e031719 [Imran Rashid] fixes from review 1f53a66 [Imran Rashid] style b4a7863 [Imran Rashid] fix compile error 2c8b7ee [Imran Rashid] rats 1578a4a [Imran Rashid] doc 674f8dc [Imran Rashid] more explicit about total numbers of jobs & stages vs. number retained 9922be0 [Imran Rashid] Merge branch 'master' into stage_distributions f5a5196 [Imran Rashid] undo removal of renderJson from MasterPage, since there is no substitute yet db61211 [Imran Rashid] get JobProgressListener directly from UI fdfc181 [Imran Rashid] stage/taskList 63eb4a6 [Imran Rashid] tests for taskSummary ad27de8 [Imran Rashid] error handling on quantile values b2efcaf [Imran Rashid] cleanup, combine stage-related paths into one resource aaba896 [Imran Rashid] wire up task summary a4b1397 [Imran Rashid] stage metric distributions e48ba32 [Imran Rashid] rename eaf3bbb [Imran Rashid] style 25cd894 [Imran Rashid] if only given day, assume GMT 51eaedb [Imran Rashid] more visibility fixes 9f28b7e [Imran Rashid] ack, more cleanup 99764e1 [Imran Rashid] Merge branch 'SPARK-3454_w_jersey' into SPARK-3454 a61a43c [Imran Rashid] oops, remove accidental checkin a066055 [Imran Rashid] set visibility on a lot of classes 1f361c8 [Imran Rashid] update rat-excludes 0be5120 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 2382bef [Imran Rashid] switch to using new "enum" fef6605 [Imran Rashid] some utils for working w/ new "enum" format dbfc7bf [Imran Rashid] style b86bcb0 [Imran Rashid] update test to look at one stage attempt 5f9df24 [Imran Rashid] style 7fd156a [Imran Rashid] refactor jsonDiff to avoid code duplication 73f1378 [Imran Rashid] test json; also add test cases for cleaned stages & jobs 97d411f [Imran Rashid] json endpoint for one job 0c96147 [Imran Rashid] better error msgs for bad stageId vs bad attemptId dddbd29 [Imran Rashid] stages have attempt; jobs are sorted; resource for all attempts for one stage 190c17a [Imran Rashid] StagePage should distinguish no task data, from unknown stage 84cd497 [Imran Rashid] AllJobsPage should still report correct completed & failed job count, even if some have been cleaned, to make it consistent w/ AllStagesPage 36e4062 [Imran Rashid] SparkUI needs to know about startTime, so it can list its own applicationInfo b4c75ed [Imran Rashid] fix merge conflicts; need to widen visibility in a few cases e91750a [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 56d2fc7 [Imran Rashid] jersey needs asm for SPARK_PREPEND_CLASSES to work f7df095 [Imran Rashid] add test for accumulables, and discover that I need update after all 9c0c125 [Imran Rashid] add accumulableInfo 00e9cc5 [Imran Rashid] more style 3377e61 [Imran Rashid] scaladoc d05f7a9 [Imran Rashid] dont use case classes for status api POJOs, since they have binary compatibility issues 654cecf [Imran Rashid] move all the status api POJOs to one file b86e2b0 [Imran Rashid] style 18a8c45 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 5598f19 [Imran Rashid] delete some unnecessary code, more to go 56edce0 [Imran Rashid] style 017c755 [Imran Rashid] add in metrics now available 1b78cb7 [Imran Rashid] fix some import ordering 0dc3ea7 [Imran Rashid] if app isnt found, reload apps from FS before giving up c7d884f [Imran Rashid] fix merge conflicts 0c12b50 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey b6a96a8 [Imran Rashid] compare json by AST, not string cd37845 [Imran Rashid] switch to using java.util.Dates for times a4ab5aa [Imran Rashid] add in explicit dependency on jersey 1.9 -- maven wasn't happy before this 4fdc39f [Imran Rashid] refactor case insensitive enum parsing cba1ef6 [Imran Rashid] add security (maybe?) for metrics json f0264a7 [Imran Rashid] switch to using jersey for metrics json bceb3a9 [Imran Rashid] set http response code on error, some testing e0356b6 [Imran Rashid] put new test expectation files in rat excludes (is this OK?) b252e7a [Imran Rashid] small cleanup of accidental changes d1a8c92 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 4b398d0 [Imran Rashid] expose UI data as json in new endpoints (cherry picked from commit d497358) Signed-off-by: Imran Rashid <irashid@cloudera.com>

rxin · 2015-05-05T21:38:39Z

I think Github is confused about the test files added in this pull request and now thinks 30% of Spark is written in PHP.

Can we rename the files or somehow making github not thinking we are using PHP?

scwf · 2015-05-06T01:56:51Z

+1 to rename the test files, since you added test file which names contain ?/& , this leads to git pull failure on windows system:

$ git pull origin master
From https://github.com/apache/spark
 * branch            master     -> FETCH_HEAD
Updating 8aa5aea..c688e3c
fatal: cannot create directory at 'core/src/test/resources/HistoryServerExpectat
ions/applications/local-1422981780767/jobs?status=succeeded&status=failed': Inva
lid argument

rxin · 2015-05-06T02:06:21Z

hmmm we might need to revert this for that and fix it.

rxin · 2015-05-06T02:25:49Z

@squito I'm going to revert the patch. Let's figure out how to deal with the file names properly. URLs are not proper filenames.

On top of that, is there any reason why we need a 12k line test file? I looked into it and most of it are just task scheduling events. It is so big that it is not feasible for humans to look at at all.

squito · 2015-05-06T13:39:45Z

@scwf @rxin oh, sorry about the file name issue, that is really unfortunate. I thought it was a limited enough set of URLs it worked out ok, didn't realize the windows problem. in any case I'll switch to sanitized file names

and you are right about the big test file -- I had it sitting around to test lots of different examples earlier, but I've paired it down to something much simpler. Will open a new PR shortly

This was originally added as part of #4435, which was reverted.

@Private

Exposes data available in the UI as json over http. Key points: * new endpoints, handled independently of existing XyzPage classes. Root entrypoint is `JsonRootResource` * Uses jersey + jackson for routing & converting POJOs into json * tests against known results in `HistoryServerSuite` * also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages. Author: Imran Rashid <irashid@cloudera.com> Closes apache#4435 from squito/SPARK-3454 and squashes the following commits: da1e35f [Imran Rashid] typos etc. 5e78b4f [Imran Rashid] fix rendering problems 5ae02ad [Imran Rashid] Merge branch 'master' into SPARK-3454 f016182 [Imran Rashid] change all constructors json-pojo class constructors to be private[spark] to protect us from mima-false-positives if we add fields 3347b72 [Imran Rashid] mark EnumUtil as @Private ec140a2 [Imran Rashid] create @Private cc1febf [Imran Rashid] add docs on the metrics-as-json api cbaf287 [Imran Rashid] Merge branch 'master' into SPARK-3454 56db31e [Imran Rashid] update tests for mulit-attempt 7f3bc4e [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" 67008b4 [Imran Rashid] rats 9e51400 [Imran Rashid] style c9bae1c [Imran Rashid] handle multiple attempts per app b87cd63 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 188762c [Imran Rashid] multi-attempt 2af11e5 [Imran Rashid] Merge branch 'master' into SPARK-3454 befff0c [Imran Rashid] review feedback 14ac3ed [Imran Rashid] jersey-core needs to be explicit; move version & scope to parent pom.xml f90680e [Imran Rashid] Merge branch 'master' into SPARK-3454 dc8a7fe [Imran Rashid] style, fix errant comments acb7ef6 [Imran Rashid] fix indentation 7bf1811 [Imran Rashid] move MetricHelper so mima doesnt think its exposed; comments 9d889d6 [Imran Rashid] undo some unnecessary changes f48a7b0 [Imran Rashid] docs 52bbae8 [Imran Rashid] StorageListener & StorageStatusListener needs to synchronize internally to be thread-safe 31c79ce [Imran Rashid] asm no longer needed for SPARK_PREPEND_CLASSES b2f8b91 [Imran Rashid] @DeveloperAPI 2e19be2 [Imran Rashid] lazily convert ApplicationInfo to avoid memory overhead ba3d9d2 [Imran Rashid] upper case enums 39ac29c [Imran Rashid] move EnumUtil d2bde77 [Imran Rashid] update error handling & scoping 4a234d3 [Imran Rashid] avoid jersey-media-json-jackson b/c of potential version conflicts a157a2f [Imran Rashid] style 7bd4d15 [Imran Rashid] delete security test, since it doesnt do anything a325563 [Imran Rashid] style a9c5cf1 [Imran Rashid] undo changes superceeded by master 0c6f968 [Imran Rashid] update deps 1ed0d07 [Imran Rashid] Merge branch 'master' into SPARK-3454 4c92af6 [Imran Rashid] style f2e63ad [Imran Rashid] Merge branch 'master' into SPARK-3454 c22b11f [Imran Rashid] fix compile error 9ea682c [Imran Rashid] go back to good ol' java enums cf86175 [Imran Rashid] style d493b38 [Imran Rashid] Merge branch 'master' into SPARK-3454 f05ae89 [Imran Rashid] add in ExecutorSummaryInfo for MiMa :( 101a698 [Imran Rashid] style d2ef58d [Imran Rashid] revert changes that had HistoryServer refresh the application listing more often b136e39 [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" e031719 [Imran Rashid] fixes from review 1f53a66 [Imran Rashid] style b4a7863 [Imran Rashid] fix compile error 2c8b7ee [Imran Rashid] rats 1578a4a [Imran Rashid] doc 674f8dc [Imran Rashid] more explicit about total numbers of jobs & stages vs. number retained 9922be0 [Imran Rashid] Merge branch 'master' into stage_distributions f5a5196 [Imran Rashid] undo removal of renderJson from MasterPage, since there is no substitute yet db61211 [Imran Rashid] get JobProgressListener directly from UI fdfc181 [Imran Rashid] stage/taskList 63eb4a6 [Imran Rashid] tests for taskSummary ad27de8 [Imran Rashid] error handling on quantile values b2efcaf [Imran Rashid] cleanup, combine stage-related paths into one resource aaba896 [Imran Rashid] wire up task summary a4b1397 [Imran Rashid] stage metric distributions e48ba32 [Imran Rashid] rename eaf3bbb [Imran Rashid] style 25cd894 [Imran Rashid] if only given day, assume GMT 51eaedb [Imran Rashid] more visibility fixes 9f28b7e [Imran Rashid] ack, more cleanup 99764e1 [Imran Rashid] Merge branch 'SPARK-3454_w_jersey' into SPARK-3454 a61a43c [Imran Rashid] oops, remove accidental checkin a066055 [Imran Rashid] set visibility on a lot of classes 1f361c8 [Imran Rashid] update rat-excludes 0be5120 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 2382bef [Imran Rashid] switch to using new "enum" fef6605 [Imran Rashid] some utils for working w/ new "enum" format dbfc7bf [Imran Rashid] style b86bcb0 [Imran Rashid] update test to look at one stage attempt 5f9df24 [Imran Rashid] style 7fd156a [Imran Rashid] refactor jsonDiff to avoid code duplication 73f1378 [Imran Rashid] test json; also add test cases for cleaned stages & jobs 97d411f [Imran Rashid] json endpoint for one job 0c96147 [Imran Rashid] better error msgs for bad stageId vs bad attemptId dddbd29 [Imran Rashid] stages have attempt; jobs are sorted; resource for all attempts for one stage 190c17a [Imran Rashid] StagePage should distinguish no task data, from unknown stage 84cd497 [Imran Rashid] AllJobsPage should still report correct completed & failed job count, even if some have been cleaned, to make it consistent w/ AllStagesPage 36e4062 [Imran Rashid] SparkUI needs to know about startTime, so it can list its own applicationInfo b4c75ed [Imran Rashid] fix merge conflicts; need to widen visibility in a few cases e91750a [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 56d2fc7 [Imran Rashid] jersey needs asm for SPARK_PREPEND_CLASSES to work f7df095 [Imran Rashid] add test for accumulables, and discover that I need update after all 9c0c125 [Imran Rashid] add accumulableInfo 00e9cc5 [Imran Rashid] more style 3377e61 [Imran Rashid] scaladoc d05f7a9 [Imran Rashid] dont use case classes for status api POJOs, since they have binary compatibility issues 654cecf [Imran Rashid] move all the status api POJOs to one file b86e2b0 [Imran Rashid] style 18a8c45 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 5598f19 [Imran Rashid] delete some unnecessary code, more to go 56edce0 [Imran Rashid] style 017c755 [Imran Rashid] add in metrics now available 1b78cb7 [Imran Rashid] fix some import ordering 0dc3ea7 [Imran Rashid] if app isnt found, reload apps from FS before giving up c7d884f [Imran Rashid] fix merge conflicts 0c12b50 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey b6a96a8 [Imran Rashid] compare json by AST, not string cd37845 [Imran Rashid] switch to using java.util.Dates for times a4ab5aa [Imran Rashid] add in explicit dependency on jersey 1.9 -- maven wasn't happy before this 4fdc39f [Imran Rashid] refactor case insensitive enum parsing cba1ef6 [Imran Rashid] add security (maybe?) for metrics json f0264a7 [Imran Rashid] switch to using jersey for metrics json bceb3a9 [Imran Rashid] set http response code on error, some testing e0356b6 [Imran Rashid] put new test expectation files in rat excludes (is this OK?) b252e7a [Imran Rashid] small cleanup of accidental changes d1a8c92 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 4b398d0 [Imran Rashid] expose UI data as json in new endpoints

This was originally added as part of apache#4435, which was reverted.

@Private

Exposes data available in the UI as json over http. Key points: * new endpoints, handled independently of existing XyzPage classes. Root entrypoint is `JsonRootResource` * Uses jersey + jackson for routing & converting POJOs into json * tests against known results in `HistoryServerSuite` * also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages. Author: Imran Rashid <irashid@cloudera.com> Closes apache#4435 from squito/SPARK-3454 and squashes the following commits: da1e35f [Imran Rashid] typos etc. 5e78b4f [Imran Rashid] fix rendering problems 5ae02ad [Imran Rashid] Merge branch 'master' into SPARK-3454 f016182 [Imran Rashid] change all constructors json-pojo class constructors to be private[spark] to protect us from mima-false-positives if we add fields 3347b72 [Imran Rashid] mark EnumUtil as @Private ec140a2 [Imran Rashid] create @Private cc1febf [Imran Rashid] add docs on the metrics-as-json api cbaf287 [Imran Rashid] Merge branch 'master' into SPARK-3454 56db31e [Imran Rashid] update tests for mulit-attempt 7f3bc4e [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" 67008b4 [Imran Rashid] rats 9e51400 [Imran Rashid] style c9bae1c [Imran Rashid] handle multiple attempts per app b87cd63 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 188762c [Imran Rashid] multi-attempt 2af11e5 [Imran Rashid] Merge branch 'master' into SPARK-3454 befff0c [Imran Rashid] review feedback 14ac3ed [Imran Rashid] jersey-core needs to be explicit; move version & scope to parent pom.xml f90680e [Imran Rashid] Merge branch 'master' into SPARK-3454 dc8a7fe [Imran Rashid] style, fix errant comments acb7ef6 [Imran Rashid] fix indentation 7bf1811 [Imran Rashid] move MetricHelper so mima doesnt think its exposed; comments 9d889d6 [Imran Rashid] undo some unnecessary changes f48a7b0 [Imran Rashid] docs 52bbae8 [Imran Rashid] StorageListener & StorageStatusListener needs to synchronize internally to be thread-safe 31c79ce [Imran Rashid] asm no longer needed for SPARK_PREPEND_CLASSES b2f8b91 [Imran Rashid] @DeveloperAPI 2e19be2 [Imran Rashid] lazily convert ApplicationInfo to avoid memory overhead ba3d9d2 [Imran Rashid] upper case enums 39ac29c [Imran Rashid] move EnumUtil d2bde77 [Imran Rashid] update error handling & scoping 4a234d3 [Imran Rashid] avoid jersey-media-json-jackson b/c of potential version conflicts a157a2f [Imran Rashid] style 7bd4d15 [Imran Rashid] delete security test, since it doesnt do anything a325563 [Imran Rashid] style a9c5cf1 [Imran Rashid] undo changes superceeded by master 0c6f968 [Imran Rashid] update deps 1ed0d07 [Imran Rashid] Merge branch 'master' into SPARK-3454 4c92af6 [Imran Rashid] style f2e63ad [Imran Rashid] Merge branch 'master' into SPARK-3454 c22b11f [Imran Rashid] fix compile error 9ea682c [Imran Rashid] go back to good ol' java enums cf86175 [Imran Rashid] style d493b38 [Imran Rashid] Merge branch 'master' into SPARK-3454 f05ae89 [Imran Rashid] add in ExecutorSummaryInfo for MiMa :( 101a698 [Imran Rashid] style d2ef58d [Imran Rashid] revert changes that had HistoryServer refresh the application listing more often b136e39 [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" e031719 [Imran Rashid] fixes from review 1f53a66 [Imran Rashid] style b4a7863 [Imran Rashid] fix compile error 2c8b7ee [Imran Rashid] rats 1578a4a [Imran Rashid] doc 674f8dc [Imran Rashid] more explicit about total numbers of jobs & stages vs. number retained 9922be0 [Imran Rashid] Merge branch 'master' into stage_distributions f5a5196 [Imran Rashid] undo removal of renderJson from MasterPage, since there is no substitute yet db61211 [Imran Rashid] get JobProgressListener directly from UI fdfc181 [Imran Rashid] stage/taskList 63eb4a6 [Imran Rashid] tests for taskSummary ad27de8 [Imran Rashid] error handling on quantile values b2efcaf [Imran Rashid] cleanup, combine stage-related paths into one resource aaba896 [Imran Rashid] wire up task summary a4b1397 [Imran Rashid] stage metric distributions e48ba32 [Imran Rashid] rename eaf3bbb [Imran Rashid] style 25cd894 [Imran Rashid] if only given day, assume GMT 51eaedb [Imran Rashid] more visibility fixes 9f28b7e [Imran Rashid] ack, more cleanup 99764e1 [Imran Rashid] Merge branch 'SPARK-3454_w_jersey' into SPARK-3454 a61a43c [Imran Rashid] oops, remove accidental checkin a066055 [Imran Rashid] set visibility on a lot of classes 1f361c8 [Imran Rashid] update rat-excludes 0be5120 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 2382bef [Imran Rashid] switch to using new "enum" fef6605 [Imran Rashid] some utils for working w/ new "enum" format dbfc7bf [Imran Rashid] style b86bcb0 [Imran Rashid] update test to look at one stage attempt 5f9df24 [Imran Rashid] style 7fd156a [Imran Rashid] refactor jsonDiff to avoid code duplication 73f1378 [Imran Rashid] test json; also add test cases for cleaned stages & jobs 97d411f [Imran Rashid] json endpoint for one job 0c96147 [Imran Rashid] better error msgs for bad stageId vs bad attemptId dddbd29 [Imran Rashid] stages have attempt; jobs are sorted; resource for all attempts for one stage 190c17a [Imran Rashid] StagePage should distinguish no task data, from unknown stage 84cd497 [Imran Rashid] AllJobsPage should still report correct completed & failed job count, even if some have been cleaned, to make it consistent w/ AllStagesPage 36e4062 [Imran Rashid] SparkUI needs to know about startTime, so it can list its own applicationInfo b4c75ed [Imran Rashid] fix merge conflicts; need to widen visibility in a few cases e91750a [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 56d2fc7 [Imran Rashid] jersey needs asm for SPARK_PREPEND_CLASSES to work f7df095 [Imran Rashid] add test for accumulables, and discover that I need update after all 9c0c125 [Imran Rashid] add accumulableInfo 00e9cc5 [Imran Rashid] more style 3377e61 [Imran Rashid] scaladoc d05f7a9 [Imran Rashid] dont use case classes for status api POJOs, since they have binary compatibility issues 654cecf [Imran Rashid] move all the status api POJOs to one file b86e2b0 [Imran Rashid] style 18a8c45 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 5598f19 [Imran Rashid] delete some unnecessary code, more to go 56edce0 [Imran Rashid] style 017c755 [Imran Rashid] add in metrics now available 1b78cb7 [Imran Rashid] fix some import ordering 0dc3ea7 [Imran Rashid] if app isnt found, reload apps from FS before giving up c7d884f [Imran Rashid] fix merge conflicts 0c12b50 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey b6a96a8 [Imran Rashid] compare json by AST, not string cd37845 [Imran Rashid] switch to using java.util.Dates for times a4ab5aa [Imran Rashid] add in explicit dependency on jersey 1.9 -- maven wasn't happy before this 4fdc39f [Imran Rashid] refactor case insensitive enum parsing cba1ef6 [Imran Rashid] add security (maybe?) for metrics json f0264a7 [Imran Rashid] switch to using jersey for metrics json bceb3a9 [Imran Rashid] set http response code on error, some testing e0356b6 [Imran Rashid] put new test expectation files in rat excludes (is this OK?) b252e7a [Imran Rashid] small cleanup of accidental changes d1a8c92 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 4b398d0 [Imran Rashid] expose UI data as json in new endpoints

This was originally added as part of apache#4435, which was reverted.

@Private

Exposes data available in the UI as json over http. Key points: * new endpoints, handled independently of existing XyzPage classes. Root entrypoint is `JsonRootResource` * Uses jersey + jackson for routing & converting POJOs into json * tests against known results in `HistoryServerSuite` * also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages. Author: Imran Rashid <irashid@cloudera.com> Closes apache#4435 from squito/SPARK-3454 and squashes the following commits: da1e35f [Imran Rashid] typos etc. 5e78b4f [Imran Rashid] fix rendering problems 5ae02ad [Imran Rashid] Merge branch 'master' into SPARK-3454 f016182 [Imran Rashid] change all constructors json-pojo class constructors to be private[spark] to protect us from mima-false-positives if we add fields 3347b72 [Imran Rashid] mark EnumUtil as @Private ec140a2 [Imran Rashid] create @Private cc1febf [Imran Rashid] add docs on the metrics-as-json api cbaf287 [Imran Rashid] Merge branch 'master' into SPARK-3454 56db31e [Imran Rashid] update tests for mulit-attempt 7f3bc4e [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" 67008b4 [Imran Rashid] rats 9e51400 [Imran Rashid] style c9bae1c [Imran Rashid] handle multiple attempts per app b87cd63 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 188762c [Imran Rashid] multi-attempt 2af11e5 [Imran Rashid] Merge branch 'master' into SPARK-3454 befff0c [Imran Rashid] review feedback 14ac3ed [Imran Rashid] jersey-core needs to be explicit; move version & scope to parent pom.xml f90680e [Imran Rashid] Merge branch 'master' into SPARK-3454 dc8a7fe [Imran Rashid] style, fix errant comments acb7ef6 [Imran Rashid] fix indentation 7bf1811 [Imran Rashid] move MetricHelper so mima doesnt think its exposed; comments 9d889d6 [Imran Rashid] undo some unnecessary changes f48a7b0 [Imran Rashid] docs 52bbae8 [Imran Rashid] StorageListener & StorageStatusListener needs to synchronize internally to be thread-safe 31c79ce [Imran Rashid] asm no longer needed for SPARK_PREPEND_CLASSES b2f8b91 [Imran Rashid] @DeveloperAPI 2e19be2 [Imran Rashid] lazily convert ApplicationInfo to avoid memory overhead ba3d9d2 [Imran Rashid] upper case enums 39ac29c [Imran Rashid] move EnumUtil d2bde77 [Imran Rashid] update error handling & scoping 4a234d3 [Imran Rashid] avoid jersey-media-json-jackson b/c of potential version conflicts a157a2f [Imran Rashid] style 7bd4d15 [Imran Rashid] delete security test, since it doesnt do anything a325563 [Imran Rashid] style a9c5cf1 [Imran Rashid] undo changes superceeded by master 0c6f968 [Imran Rashid] update deps 1ed0d07 [Imran Rashid] Merge branch 'master' into SPARK-3454 4c92af6 [Imran Rashid] style f2e63ad [Imran Rashid] Merge branch 'master' into SPARK-3454 c22b11f [Imran Rashid] fix compile error 9ea682c [Imran Rashid] go back to good ol' java enums cf86175 [Imran Rashid] style d493b38 [Imran Rashid] Merge branch 'master' into SPARK-3454 f05ae89 [Imran Rashid] add in ExecutorSummaryInfo for MiMa :( 101a698 [Imran Rashid] style d2ef58d [Imran Rashid] revert changes that had HistoryServer refresh the application listing more often b136e39 [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" e031719 [Imran Rashid] fixes from review 1f53a66 [Imran Rashid] style b4a7863 [Imran Rashid] fix compile error 2c8b7ee [Imran Rashid] rats 1578a4a [Imran Rashid] doc 674f8dc [Imran Rashid] more explicit about total numbers of jobs & stages vs. number retained 9922be0 [Imran Rashid] Merge branch 'master' into stage_distributions f5a5196 [Imran Rashid] undo removal of renderJson from MasterPage, since there is no substitute yet db61211 [Imran Rashid] get JobProgressListener directly from UI fdfc181 [Imran Rashid] stage/taskList 63eb4a6 [Imran Rashid] tests for taskSummary ad27de8 [Imran Rashid] error handling on quantile values b2efcaf [Imran Rashid] cleanup, combine stage-related paths into one resource aaba896 [Imran Rashid] wire up task summary a4b1397 [Imran Rashid] stage metric distributions e48ba32 [Imran Rashid] rename eaf3bbb [Imran Rashid] style 25cd894 [Imran Rashid] if only given day, assume GMT 51eaedb [Imran Rashid] more visibility fixes 9f28b7e [Imran Rashid] ack, more cleanup 99764e1 [Imran Rashid] Merge branch 'SPARK-3454_w_jersey' into SPARK-3454 a61a43c [Imran Rashid] oops, remove accidental checkin a066055 [Imran Rashid] set visibility on a lot of classes 1f361c8 [Imran Rashid] update rat-excludes 0be5120 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 2382bef [Imran Rashid] switch to using new "enum" fef6605 [Imran Rashid] some utils for working w/ new "enum" format dbfc7bf [Imran Rashid] style b86bcb0 [Imran Rashid] update test to look at one stage attempt 5f9df24 [Imran Rashid] style 7fd156a [Imran Rashid] refactor jsonDiff to avoid code duplication 73f1378 [Imran Rashid] test json; also add test cases for cleaned stages & jobs 97d411f [Imran Rashid] json endpoint for one job 0c96147 [Imran Rashid] better error msgs for bad stageId vs bad attemptId dddbd29 [Imran Rashid] stages have attempt; jobs are sorted; resource for all attempts for one stage 190c17a [Imran Rashid] StagePage should distinguish no task data, from unknown stage 84cd497 [Imran Rashid] AllJobsPage should still report correct completed & failed job count, even if some have been cleaned, to make it consistent w/ AllStagesPage 36e4062 [Imran Rashid] SparkUI needs to know about startTime, so it can list its own applicationInfo b4c75ed [Imran Rashid] fix merge conflicts; need to widen visibility in a few cases e91750a [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 56d2fc7 [Imran Rashid] jersey needs asm for SPARK_PREPEND_CLASSES to work f7df095 [Imran Rashid] add test for accumulables, and discover that I need update after all 9c0c125 [Imran Rashid] add accumulableInfo 00e9cc5 [Imran Rashid] more style 3377e61 [Imran Rashid] scaladoc d05f7a9 [Imran Rashid] dont use case classes for status api POJOs, since they have binary compatibility issues 654cecf [Imran Rashid] move all the status api POJOs to one file b86e2b0 [Imran Rashid] style 18a8c45 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 5598f19 [Imran Rashid] delete some unnecessary code, more to go 56edce0 [Imran Rashid] style 017c755 [Imran Rashid] add in metrics now available 1b78cb7 [Imran Rashid] fix some import ordering 0dc3ea7 [Imran Rashid] if app isnt found, reload apps from FS before giving up c7d884f [Imran Rashid] fix merge conflicts 0c12b50 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey b6a96a8 [Imran Rashid] compare json by AST, not string cd37845 [Imran Rashid] switch to using java.util.Dates for times a4ab5aa [Imran Rashid] add in explicit dependency on jersey 1.9 -- maven wasn't happy before this 4fdc39f [Imran Rashid] refactor case insensitive enum parsing cba1ef6 [Imran Rashid] add security (maybe?) for metrics json f0264a7 [Imran Rashid] switch to using jersey for metrics json bceb3a9 [Imran Rashid] set http response code on error, some testing e0356b6 [Imran Rashid] put new test expectation files in rat excludes (is this OK?) b252e7a [Imran Rashid] small cleanup of accidental changes d1a8c92 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 4b398d0 [Imran Rashid] expose UI data as json in new endpoints

This was originally added as part of apache#4435, which was reverted.

squito added 3 commits February 6, 2015 15:21

expose UI data as json in new endpoints

4b398d0

add sbt-revolved plugin, to make it easier to start & stop http serve…

d1a8c92

…rs in sbt

small cleanup of accidental changes

b252e7a

squito added 2 commits February 6, 2015 16:38

put new test expectation files in rat excludes (is this OK?)

e0356b6

set http response code on error, some testing

bceb3a9

squito reviewed Feb 6, 2015
View reviewed changes

rxin reviewed Feb 6, 2015
View reviewed changes

squito added 4 commits February 11, 2015 19:32

switch to using jersey for metrics json

f0264a7

add security (maybe?) for metrics json

cba1ef6

refactor case insensitive enum parsing

4fdc39f

add in explicit dependency on jersey 1.9 -- maven wasn't happy before

a4ab5aa

this

andrewor14 mentioned this pull request Feb 19, 2015

[SPARK-3454] Expose JSON representation of data shown in WebU #2333

Closed

andrewor14 mentioned this pull request Feb 19, 2015

[SPARK-3468][WebUI] Timeline-View feature #2342

Closed

squito added 2 commits February 23, 2015 14:07

switch to using java.util.Dates for times

cd37845

compare json by AST, not string

b6a96a8

JoshRosen reviewed Mar 9, 2015
View reviewed changes

squito added 2 commits May 4, 2015 21:16

Merge branch 'master' into SPARK-3454

5ae02ad

Conflicts: core/src/main/scala/org/apache/spark/ui/SparkUI.scala core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala

fix rendering problems

5e78b4f

squito reviewed May 5, 2015
View reviewed changes

typos etc.

da1e35f

pwendell reviewed May 5, 2015
View reviewed changes

squito reviewed May 5, 2015
View reviewed changes

asfgit closed this in d497358 May 5, 2015

squito mentioned this pull request May 6, 2015

[SPARK-3454] separate json endpoints for data in the UI #5940

Closed

asfgit pushed a commit that referenced this pull request May 6, 2015

Add Private annotation.

845d1d4

This was originally added as part of #4435, which was reverted.

asfgit pushed a commit that referenced this pull request May 6, 2015

Add Private annotation.

2163367

This was originally added as part of #4435, which was reverted.

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015

Add Private annotation.

ff9d51b

This was originally added as part of apache#4435, which was reverted.

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015

Add Private annotation.

8c28f93

This was originally added as part of apache#4435, which was reverted.

nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015

Add Private annotation.

9467f11

This was originally added as part of apache#4435, which was reverted.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3454] separate json endpoints for data in the UI #4435

[SPARK-3454] separate json endpoints for data in the UI #4435

squito commented Feb 6, 2015

SparkQA commented Feb 6, 2015

squito Feb 6, 2015

andrewor14 Feb 7, 2015

SparkQA commented Feb 6, 2015

rxin Feb 6, 2015

andrewor14 commented Feb 19, 2015

squito commented Feb 20, 2015

JoshRosen Mar 9, 2015

JoshRosen commented Mar 9, 2015

squito commented Mar 9, 2015

squito commented May 5, 2015

squito May 5, 2015

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

pwendell May 5, 2015

pwendell commented May 5, 2015

squito commented May 5, 2015

squito May 5, 2015

rxin commented May 5, 2015

scwf commented May 6, 2015

rxin commented May 6, 2015

rxin commented May 6, 2015

squito commented May 6, 2015

[SPARK-3454] separate json endpoints for data in the UI #4435

[SPARK-3454] separate json endpoints for data in the UI #4435

Conversation

squito commented Feb 6, 2015

SparkQA commented Feb 6, 2015

squito Feb 6, 2015

Choose a reason for hiding this comment

andrewor14 Feb 7, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 6, 2015

rxin Feb 6, 2015

Choose a reason for hiding this comment

andrewor14 commented Feb 19, 2015

squito commented Feb 20, 2015

JoshRosen Mar 9, 2015

Choose a reason for hiding this comment

JoshRosen commented Mar 9, 2015

squito commented Mar 9, 2015

squito commented May 5, 2015

squito May 5, 2015

Choose a reason for hiding this comment

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

SparkQA commented May 5, 2015

pwendell May 5, 2015

Choose a reason for hiding this comment

pwendell commented May 5, 2015

squito commented May 5, 2015

squito May 5, 2015

Choose a reason for hiding this comment

rxin commented May 5, 2015

scwf commented May 6, 2015

rxin commented May 6, 2015

rxin commented May 6, 2015

squito commented May 6, 2015