[YARN] SPARK-2668: Add variable of yarn log directory for reference from the log4j configuration

[renozhang]

Assign value of yarn container log directory to java opts "spark.yarn.app.container.log.dir", So user defined log4j.properties can reference this value and write log to YARN container's log directory.
Otherwise, user defined file appender will only write to container's CWD, and log files in CWD will not be displayed on YARN UI，and either cannot be aggregated to HDFS log directory after job finished.

User defined log4j.properties reference example:
log4j.appender.rolling_file.File = ${spark.yarn.app.container.log.dir}/spark.log

[SparkQA]

QA tests have started for PR 1573. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17119/consoleFull

[SparkQA]

QA results for PR 1573:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17119/consoleFull

[tgravescs]

@renozhang sorry for the delay on this, could you upmerge to the latest?

[renozhang]

@tgravescs I've update to the latest, thanks for review.

[tgravescs]

can you rename this to be spark.yarn.app.container.log.dir

Can you also update the documentation in docs/running-on-yarn.md to have this config and a good description.

[renozhang]

Sorry, I missed writing description in PR. I'll fill description in PR later.

As metioned in descritpion of Jira SPARK-2668:

Adding this varialbe is for user to define custom log4j.properties, eg:

log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log

发件人: andrewor14 <notifications@github.commailto:notifications@github.com>
答复: apache/spark <reply@reply.github.commailto:reply@reply.github.com>
日期: Thu, 11 Sep 2014 18:08:37 -0700
至: apache/spark <spark@noreply.github.commailto:spark@noreply.github.com>
抄送: Peng <peng.zhang@xiaomi.commailto:peng.zhang@xiaomi.com>
主题: Re: [spark] [YARN] SPARK-2668: Add variable of yarn log directory for reference from the log4j configuration (#1573)

Where is this config being consumed? Am I missing something obvious?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1573#issuecomment-55350201.

[andrewor14]

Yup, thanks for answering my deleted question. I realized this afterwards after reading the JIRA.

[tgravescs]

@renozhang can you address my comments

[renozhang]

Sorry @tgravescs , these days very busy, I'll address them this weekend.

[SparkQA]

QA tests have started for PR 1573 at commit c56aba6.

• This patch merges cleanly.

[renozhang]

@tgravescs patch updated, thanks for your review.

[SparkQA]

QA tests have finished for PR 1573 at commit c56aba6.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tgravescs]

Since this could be used in more then just streaming applications perhaps we should reword this a little. Perhaps put the information about ${spark.yarn.app.container.log.dir} first and then give the example using RollingFileAppender with streaming.

Something more like: (note feel free to change the exact wording)

If you need a reference to the proper location to put the log files in the YARN so that YARN can properly display and aggregate them, use "${spark.yarn.app.container.log.dir}" in your log4j.properties. For example... (then explain the streaming example).

[tgravescs]

thanks @renozhang, minor request about the documentation, otherwise looks good.

[SparkQA]

QA tests have started for PR 1573 at commit 16c5cb8.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1573 at commit 16c5cb8.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20697/

[tgravescs]

testfailure is unrelated to this pr

[tgravescs]

+1 thanks @renozhang !