SPARK-2150: Provide direct link to finished application UI in yarn resou... #1094

rahulsinghaliitd · 2014-06-16T05:16:47Z

...rce manager UI

Use the event logger directory to provide a direct link to finished
application UI in yarn resourcemanager UI.

AmplabJenkins · 2014-06-16T05:19:43Z

Can one of the admins verify this patch?

vanzin · 2014-06-16T16:31:18Z

Thanks for filing this; this is something I had on my radar but hadn't gotten down to it yet.

While I also want the feature, I think we need a better approach here. For one, I'm not a big fan of using the log directory name as the application's URL in the history server; it's not very discoverable, since it's generated internally by the event logger. Also, slightly unrelated, but I have upcoming changes that make those names even more ugly, so they're not a very user-friendly thing to use as a public-facing URL.

I've actually been playing with this and have some changes that use the actual application ID, when it's available (such as in the case of Yarn), to serve the application from the history server. I think that's a better approach. But if some committer wants to check this in while I work on that, it would not cause (many) conflicts with my change.

While we're here, you'd need to make a similar change in the yarn/alpha project, and also in yarn-client mode (ExecutorLauncher.scala in both projects).

rahulsinghaliitd · 2014-06-16T18:13:55Z

Hi @vanzin,

I don't have any preference as to how the URL is formatted and I think that changing the URL can be a separate activity. I am hoping this can be committed so that we atleast have the functionality.

Thanks for pointing out the changes, I had also realized the same and I am currently in the process of testing them. They should be available soon.

sryza · 2014-06-20T22:10:03Z

This change looks good to me.

rahulsinghaliitd · 2014-06-23T04:04:45Z

@sryza thanks for the thumbs up.

Although I wonder if the approach in #1112 is better for passing the UI address (certainly is much cleaner).

vanzin · 2014-06-23T16:35:18Z

@rahulsinghaliitd that bug is about linking to the running web ui of the driver in yarn-client mode. Slightly different than this one, which is about setting the link after the application is finished.

rahulsinghaliitd · 2014-06-25T06:18:11Z

@vanzin I was only referring to how the UI URL is passed around. I have used the longer way of passing it around using command line arguments whereas the other change uses spark conf by simply setting it as another property.

vanzin · 2014-06-25T16:30:17Z

@rahulsinghaliitd ah, good point. Passing as a SparkConf property should work now that I fixed some things in the yarn-cluster backend.

vanzin · 2014-06-25T16:33:17Z

Latest patch LGTM.

vanzin · 2014-06-25T16:33:36Z

(Aside from rebasing to fix the merge conflicts.)

rahulsinghaliitd · 2014-06-26T03:30:44Z

Am I supposed to resolve the conflicts or will they be resolved by the admin who merges this change?

vanzin · 2014-06-26T16:48:44Z

Generally we ask for the PR submitter to resolve conflicts. You just have to rebase your branch on top of master, resolve conflicts, and force push it into your repo, the PR will be updated automatically.

tsudukim · 2014-07-15T05:03:31Z

Is someone working on this? or facing some problem?

tgravescs · 2014-07-15T14:20:50Z

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala

@@ -132,4 +135,17 @@ object YarnSparkHadoopUtil {
    }
  }

+  def getUIHistoryAddress(sc: SparkContext, conf: SparkConf) : String = {
+    val eventLogDir = sc.eventLogger match {
+      case Some(logger) => logger.logDir.split("/").last


I think it would be better to add a routine to the eventLogger to just give us the name of the directory rather then us splitting it and it possibly breaking in the future.

tgravescs · 2014-07-15T14:26:36Z

This PR conflicts with pr1112. I would like to put that one in first and then upmerge this.

tgravescs · 2014-07-15T14:28:15Z

core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala

@@ -172,6 +172,8 @@ class HistoryServer(
 object HistoryServer {
  private val conf = new SparkConf

+  val UI_PATH_PREFIX = "/history/"


If we are adding this we should also use it in this file to set the path.

tsudukim · 2014-07-15T18:03:59Z

OK, thank you for your reply.

tgravescs · 2014-07-15T19:50:26Z

@rahulsinghaliitd can you please upmerge

rahulsinghaliitd · 2014-07-16T07:48:59Z

@tgravescs updated according to your comments and rebased to current HEAD of master branch. Thanks for following up on this PR.

tgravescs · 2014-07-18T14:05:49Z

core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala

@@ -114,7 +114,7 @@ class HistoryServer(
    attachHandler(createStaticHandler(SparkUI.STATIC_RESOURCE_DIR, "/static"))

    val contextHandler = new ServletContextHandler
-    contextHandler.setContextPath("/history")
+    contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX.stripSuffix("/"))


I think it makes more sense to do this the other way. have the UI_PATH_PREFIX /history and add in the "/" wherever its needed.

rahulsinghaliitd · 2014-07-21T05:13:43Z

@tgravescs updated according to your comments and rebased to current HEAD of master branch.

tgravescs · 2014-07-21T16:01:23Z

yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala

@@ -289,7 +289,7 @@ class ExecutorLauncher(args: ApplicationMasterArguments, conf: Configuration, sp
      .asInstanceOf[FinishApplicationMasterRequest]
    finishReq.setAppAttemptId(appAttemptId)
    finishReq.setFinishApplicationStatus(status)
-    finishReq.setTrackingUrl(sparkConf.get("spark.yarn.historyServer.address", ""))
+    finishReq.setTrackingUrl(sparkConf.get("spark.driver.appUIHistoryAddress", "))


typo, need the other " in the default case.

woah, thanks for catching that!

tgravescs · 2014-07-22T12:44:24Z

Jenkins, test this please

SparkQA · 2014-07-22T12:48:13Z

QA tests have started for PR 1094. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16962/consoleFull

SparkQA · 2014-07-22T12:49:08Z

QA results for PR 1094:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16962/consoleFull

tgravescs · 2014-07-22T13:09:05Z

@rahulsinghaliitd sorry for this dragging on I forgot to kick jenkins. Looks like its complaining about scalastyle. Can you please update.

/home/jenkins/workspace/SparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala message=File line length exceeds 100 characters line=172

tgravescs · 2014-07-22T13:09:06Z

Jenkins, this is okay to test

…source manager UI Use the event logger directory to provide a direct link to finished application UI in yarn resourcemanager UI.

rahulsinghaliitd · 2014-07-23T03:13:42Z

@tgravescs No problem at all. Updated patch.

rahulsinghaliitd · 2014-07-23T03:57:59Z

@tgravescs There does not seem to be a maven goal to check scalastyle, isn't this something we want?

tgravescs · 2014-07-23T13:40:15Z

I've only used the sbt/sbt scalastyle target. I'm not sure if there is a reason it wasn't implemented in maven. you could file a jira for it and get comments from the folks who did it.

tgravescs · 2014-07-23T13:40:27Z

Jenkins, test this please

SparkQA · 2014-07-23T13:43:24Z

QA tests have started for PR 1094. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17034/consoleFull

SparkQA · 2014-07-23T15:26:17Z

QA results for PR 1094:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17034/consoleFull

tgravescs · 2014-07-24T14:32:01Z

+1 thanks @rahulsinghaliitd !

…sou... ...rce manager UI Use the event logger directory to provide a direct link to finished application UI in yarn resourcemanager UI. Author: Rahul Singhal <rahul.singhal@guavus.com> Closes apache#1094 from rahulsinghaliitd/SPARK-2150 and squashes the following commits: 95f230c [Rahul Singhal] SPARK-2150: Provide direct link to finished application UI in yarn resource manager UI

* [CARMEL-6305] Support to show partitions for Delta table * Fix unit tests

vanzin mentioned this pull request Jun 25, 2014

[SPARK-3388] Expose aplication ID in ApplicationStart event, use it in history server. #1218

Closed

tgravescs reviewed Jul 15, 2014
View reviewed changes

tgravescs reviewed Jul 18, 2014
View reviewed changes

tgravescs reviewed Jul 21, 2014
View reviewed changes

SPARK-2150: Provide direct link to finished application UI in yarn re…

95f230c

…source manager UI Use the event logger directory to provide a direct link to finished application UI in yarn resourcemanager UI.

asfgit closed this in 46e224a Jul 24, 2014

rahulsinghaliitd deleted the SPARK-2150 branch July 24, 2014 15:15

wangyum pushed a commit that referenced this pull request May 26, 2023

[CARMEL-6305] Support to show partitions for Delta table (#1094)

e42088d

* [CARMEL-6305] Support to show partitions for Delta table * Fix unit tests

SPARK-2150: Provide direct link to finished application UI in yarn resou... #1094

SPARK-2150: Provide direct link to finished application UI in yarn resou... #1094

Conversation

rahulsinghaliitd commented Jun 16, 2014

AmplabJenkins commented Jun 16, 2014

vanzin commented Jun 16, 2014

rahulsinghaliitd commented Jun 16, 2014

sryza commented Jun 20, 2014

rahulsinghaliitd commented Jun 23, 2014

vanzin commented Jun 23, 2014

rahulsinghaliitd commented Jun 25, 2014

vanzin commented Jun 25, 2014

vanzin commented Jun 25, 2014

vanzin commented Jun 25, 2014

rahulsinghaliitd commented Jun 26, 2014

vanzin commented Jun 26, 2014

tsudukim commented Jul 15, 2014

tgravescs Jul 15, 2014

Choose a reason for hiding this comment

tgravescs commented Jul 15, 2014

tgravescs Jul 15, 2014

Choose a reason for hiding this comment

tsudukim commented Jul 15, 2014

tgravescs commented Jul 15, 2014

rahulsinghaliitd commented Jul 16, 2014

tgravescs Jul 18, 2014

Choose a reason for hiding this comment

rahulsinghaliitd commented Jul 21, 2014

tgravescs Jul 21, 2014

Choose a reason for hiding this comment

rahulsinghaliitd Jul 22, 2014

Choose a reason for hiding this comment

tgravescs commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

tgravescs commented Jul 22, 2014

tgravescs commented Jul 22, 2014

rahulsinghaliitd commented Jul 23, 2014

rahulsinghaliitd commented Jul 23, 2014

tgravescs commented Jul 23, 2014

tgravescs commented Jul 23, 2014

SparkQA commented Jul 23, 2014

SparkQA commented Jul 23, 2014

tgravescs commented Jul 24, 2014