[SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone cluster mode #29644

KevinSmile · 2020-09-04T03:04:10Z

What changes were proposed in this pull request?

Fix [SPARK-32598] (missing driver logs under UI-ApplicationDetails-Executors tab in standalone cluster mode) .

The direct bug is: the original author forgot to implement getDriverLogUrls in StandaloneSchedulerBackend

spark/core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala

Lines 70 to 75 in 1de272f

    
             /** 
        
              * Get the URLs for the driver logs. These URLs are used to display the links in the UI 
        
              * Executors tab for the driver. 
        
              * @return Map containing the log names and their respective URLs 
        
              */ 
        
             def getDriverLogUrls: Option[Map[String, String]] = None

So we set DriverLogUrls as env in DriverRunner, and retrieve it at StandaloneSchedulerBackend.

Why are the changes needed?

Fix bug [SPARK-32598].

Does this PR introduce any user-facing change?

Yes. User will see driver logs (standalone cluster mode) under UI-ApplicationDetails-Executors tab now.

Before:

After:

How was this patch tested?

Re-check the real case in [SPARK-32598] and found this user-facing bug fixed.

KevinSmile · 2020-09-09T11:25:05Z

@srowen Can you also help find someone review this PR? Thanks!

srowen · 2020-09-09T12:38:57Z

In standalone mode aren't these just the logs from the application driver, the stdout output? I don't know this part well (and not enough to review)

KevinSmile · 2020-09-09T13:39:24Z

Yes, it's just stdout & stderr of driver, currently missing in web-ui. Thanks anyway! @srowen

KevinSmile · 2020-09-09T14:30:38Z

@zsxwing @Ngone51 Do you have time to review this? Thanks!

Ngone51

IIUC, this only works for the cluster mode? If so, we could set the driver log URLs through the env variable in DriverRunner like ExecutorRunner does? Then, we can avoid RPC calls between driver and Master.

KevinSmile · 2020-09-15T10:24:04Z

@Ngone51 Thanks for your suggestion! Yes you're right, I have updated this patch. I was thinking of using RPC to get full worker info may help further changes, seems no need.
Also I find it hard to have a standalone cluster mode in suites, and this patch just add some envs, so I didn't add ut here.

core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala

core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala

core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala

Ngone51 · 2020-09-22T14:22:14Z

Can you do some manual tests and paste screenshots of UI to show the fix result in the PR description?

KevinSmile · 2020-09-22T15:17:14Z

Can you do some manual tests and paste screenshots of UI to show the fix result in the PR description?

Done

Ngone51 · 2020-09-23T03:26:29Z

LGTM. I also verified locally. It looks good.

@jiangxb1987 Could you also take a look?

jiangxb1987

LGTM

KevinSmile · 2020-09-27T08:28:39Z

@HyukjinKwon Can you help let Jenkins test this?

HyukjinKwon · 2020-10-12T02:38:28Z

retest this please

SparkQA · 2020-10-12T04:01:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34260/

SparkQA · 2020-10-12T04:20:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34260/

SparkQA · 2020-10-12T05:43:28Z

Test build #129656 has finished for PR 29644 at commit a0bdce9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-01-15T14:50:16Z

Test build #134103 has finished for PR 29644 at commit a0bdce9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…rs tab in standalone cluster mode ### What changes were proposed in this pull request? Fix [SPARK-32598] (missing driver logs under UI-ApplicationDetails-Executors tab in standalone cluster mode) . The direct bug is: the original author forgot to implement `getDriverLogUrls` in `StandaloneSchedulerBackend` https://github.com/apache/spark/blob/1de272f98d0ff22d0dd151797f22b8faf310963a/core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala#L70-L75 So we set DriverLogUrls as env in `DriverRunner`, and retrieve it at `StandaloneSchedulerBackend`. ### Why are the changes needed? Fix bug [SPARK-32598]. ### Does this PR introduce _any_ user-facing change? Yes. User will see driver logs (standalone cluster mode) under UI-ApplicationDetails-Executors tab now. Before: ![image](https://user-images.githubusercontent.com/17903517/93901055-b5de8600-fd28-11ea-879a-d97e6f70cc6e.png) After: ![image](https://user-images.githubusercontent.com/17903517/93901080-baa33a00-fd28-11ea-8895-3787c5efbf88.png) ### How was this patch tested? Re-check the real case in [SPARK-32598] and found this user-facing bug fixed. Closes #29644 from KevinSmile/kw-dev-master. Authored-by: KevinSmile <kevinwang013@hotmail.com> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit c75c29d) Signed-off-by: Sean Owen <srowen@gmail.com>

srowen · 2021-01-15T15:02:34Z

Merged to master/3.1/3.0. It conflicted in 2.4; if we need it there, it needs another PR.

probot-autolabeler bot added the CORE label Sep 4, 2020

KevinSmile force-pushed the kw-dev-master branch from 1c396c7 to 0b4218d Compare September 6, 2020 08:33

KevinSmile changed the title ~~[SPARK-32598][Scheduler] fix missing driver logs in UI Executors tab in standalone mode~~ [SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone mode Sep 7, 2020

KevinSmile force-pushed the kw-dev-master branch 3 times, most recently from 219e16b to 6dcdcdf Compare September 12, 2020 11:41

Ngone51 reviewed Sep 14, 2020

View reviewed changes

KevinSmile force-pushed the kw-dev-master branch 2 times, most recently from 9f85ba3 to 2188497 Compare September 15, 2020 09:47

KevinSmile changed the title ~~[SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone mode~~ [SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone cluster mode Sep 15, 2020

KevinSmile force-pushed the kw-dev-master branch from 2188497 to b6f08a0 Compare September 15, 2020 11:06

Ngone51 reviewed Sep 22, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala Outdated Show resolved Hide resolved

Ngone51 reviewed Sep 22, 2020

View reviewed changes

core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala Outdated Show resolved Hide resolved

KevinSmile force-pushed the kw-dev-master branch from b6f08a0 to 60764a6 Compare September 22, 2020 07:07

Ngone51 reviewed Sep 22, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala Outdated Show resolved Hide resolved

[SPARK-32598][Scheduler] fix missing driver logs in UI Executors tab

a0bdce9

KevinSmile force-pushed the kw-dev-master branch from 60764a6 to a0bdce9 Compare September 22, 2020 14:39

jiangxb1987 approved these changes Sep 23, 2020

View reviewed changes

srowen closed this in c75c29d Jan 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone cluster mode #29644

[SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone cluster mode #29644

KevinSmile commented Sep 4, 2020 •

edited

Loading

KevinSmile commented Sep 9, 2020

srowen commented Sep 9, 2020

KevinSmile commented Sep 9, 2020

KevinSmile commented Sep 9, 2020

Ngone51 left a comment

KevinSmile commented Sep 15, 2020 •

edited

Loading

Ngone51 commented Sep 22, 2020

KevinSmile commented Sep 22, 2020

Ngone51 commented Sep 23, 2020

jiangxb1987 left a comment

KevinSmile commented Sep 27, 2020

HyukjinKwon commented Oct 12, 2020

SparkQA commented Oct 12, 2020

SparkQA commented Oct 12, 2020

SparkQA commented Oct 12, 2020

SparkQA commented Jan 15, 2021

srowen commented Jan 15, 2021

	/**
	* Get the URLs for the driver logs. These URLs are used to display the links in the UI
	* Executors tab for the driver.
	* @return Map containing the log names and their respective URLs
	*/
	def getDriverLogUrls: Option[Map[String, String]] = None

[SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone cluster mode #29644

[SPARK-32598][Scheduler] Fix missing driver logs under UI App-Executors tab in standalone cluster mode #29644

Conversation

KevinSmile commented Sep 4, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

KevinSmile commented Sep 9, 2020

srowen commented Sep 9, 2020

KevinSmile commented Sep 9, 2020

KevinSmile commented Sep 9, 2020

Ngone51 left a comment

Choose a reason for hiding this comment

KevinSmile commented Sep 15, 2020 • edited Loading

Ngone51 commented Sep 22, 2020

KevinSmile commented Sep 22, 2020

Ngone51 commented Sep 23, 2020

jiangxb1987 left a comment

Choose a reason for hiding this comment

KevinSmile commented Sep 27, 2020

HyukjinKwon commented Oct 12, 2020

SparkQA commented Oct 12, 2020

SparkQA commented Oct 12, 2020

SparkQA commented Oct 12, 2020

SparkQA commented Jan 15, 2021

srowen commented Jan 15, 2021

KevinSmile commented Sep 4, 2020 •

edited

Loading

KevinSmile commented Sep 15, 2020 •

edited

Loading