MAPREDUCE-7453. Container logs are missing when yarn.app.container.lo… #6042

zhengchenyu · 2023-09-08T13:31:51Z

Description of PR

JIRA: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.

Since HADOOP-18649, in container-log4j.properties, log4j.appender.{APPENDER}.MaxFileSize is set to ${yarn.app.container.log.filesize}, but yarn.app.container.log.filesize is 0 in default. So log is missing. This log is always rolling and only show the latest log.

This is the running log like below:

Log Type: syslog
Log Upload Time: Fri Sep 08 11:36:09 +0800 2023
Log Length: 0

Log Type: syslog.1
Log Upload Time: Fri Sep 08 11:36:09 +0800 2023
Log Length: 179
2023-09-08 11:31:34,494 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Got an error when resolve hostNames. Falling back to /default-rack for all.

When we read the log from web, syslog is alway nothing, syslog.1 only show the latest log.

Note: log4j.appender.{APPENDER}.MaxFileSize is not set before, then use default value 10M, so no problem before HADOOP-18649

How was this patch tested?

test in cluster.

For code changes:

just fix default value to avoid missing logs.

…g.filesize is set to default value 0.

slfan1989 · 2023-09-08T14:01:25Z

LGTM.

hadoop-yetus · 2023-09-08T15:49:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 25s		trunk passed
+1 💚	compile	0m 47s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 42s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	0m 50s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 35s		trunk passed
+1 💚	shadedclient	34m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 35s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 36s	/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 3 new + 499 unchanged - 2 fixed = 502 total (was 501)
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 23s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	1m 23s		the patch passed
+1 💚	shadedclient	34m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	7m 39s		hadoop-mapreduce-client-core in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		135m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6042/1/artifact/out/Dockerfile
GITHUB PR	#6042
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux f9338a5a9366 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2c799b4`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6042/1/testReport/
Max. process+thread count	1376 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6042/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zhengchenyu · 2023-09-13T06:44:29Z

@jojochuang @virajjasani Can you please review this PR?

I think this PR should be merged into next release. In default, for trunk, only show the latest line log. It will confuse the user.

slfan1989 · 2023-09-14T11:39:17Z

@zhengchenyu Thank you for your contribution! We need to fix the checkstyle issue.

slfan1989 · 2023-09-15T12:51:42Z

@zhengchenyu Thanks for your contribution! Merged into trunk.

ayushtkn

I think this broke tests:
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1351/testReport/junit/org.apache.hadoop.mapreduce.v2.app.job.impl/TestMapReduceChildJVM/testCommandLine/

@slfan1989 / @zhengchenyu can you guys check & even link the ticket which broke this on Jira

Mostly we shouldn't change the default values but fix the cause

ayushtkn · 2023-09-19T18:37:44Z

...t/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

  <name>mapreduce.task.userlog.limit.kb</name>
-  <value>0</value>
-  <description>The maximum size of user-logs of each task in KB. 0 disables the cap.
+  <value>10240</value>
+  <description>The maximum size of user-logs of each task in KB.
  </description>
 </property>

 <property>
  <name>yarn.app.mapreduce.am.container.log.limit.kb</name>
-  <value>0</value>
+  <value>10240</value>
  <description>The maximum size of the MRAppMaster attempt container logs in KB.
-    0 disables the cap.


Changing the defaults is an incompatible change. You are changing the behaviour of the application here?

zhengchenyu · 2023-09-19T23:15:12Z

@ayushtkn
Sorry this is my fault. I thought that changing the default value is not a serious behavior. I will fix the UT.
BTW, can we trigger the test4tests when no changed on test?

Mostly we shouldn't change the default values but fix the cause.

I think we should change the default value. I setup a new cluster with the latest package. When I submit a MR application, I can only see only one line, just the latest log.

Then I found that before HADOOP-18649, both log4j.appender.CLA.MaxFileSize and log4j.appender.CLA.maximumFileSize are not set, so we will use the default value 10M. After HADOOP-18649, log4j.appender.CLA.MaxFileSize is set to ${yarn.app.container.log.filesize}, in default for mr is 0, so only see the latest log.

slfan1989 · 2023-09-20T00:01:25Z

@ayushtkn Thank you very much for pointing out the issue! I will revert this PR and I will continue to follow up on this PR.

…ainer.log.filesize is set to default value 0. (#6042) Contributed by Chenyu Zheng." This reverts commit ab2bc90.

…ainer.log.filesize is set to default value 0. (#6042) Contributed by Chenyu Zheng." (#6103) This reverts commit ab2bc90.

…g.filesize is set to default value 0. (apache#6042) Contributed by Chenyu Zheng. Reviewed-by: Shilun Fan <slfan1989@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>

…ainer.log.filesize is set to default value 0. (apache#6042) Contributed by Chenyu Zheng." (apache#6103) This reverts commit ab2bc90.

MAPREDUCE-7453. Container logs are missing when yarn.app.container.lo…

2c799b4

…g.filesize is set to default value 0.

github-actions bot added MapReduce trunk labels Sep 8, 2023

slfan1989 approved these changes Sep 8, 2023

View reviewed changes

slfan1989 merged commit ab2bc90 into apache:trunk Sep 15, 2023
1 of 3 checks passed

ayushtkn reviewed Sep 19, 2023

View reviewed changes

slfan1989 added a commit that referenced this pull request Sep 20, 2023

Revert "MAPREDUCE-7453. Container logs are missing when yarn.app.cont…

c19d5e8

…ainer.log.filesize is set to default value 0. (#6042) Contributed by Chenyu Zheng." This reverts commit ab2bc90.

slfan1989 mentioned this pull request Sep 20, 2023

Revert MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0 (#6042). #6103

Merged

4 tasks

slfan1989 added a commit that referenced this pull request Sep 20, 2023

Revert "MAPREDUCE-7453. Container logs are missing when yarn.app.cont…

512e39e

…ainer.log.filesize is set to default value 0. (#6042) Contributed by Chenyu Zheng." (#6103) This reverts commit ab2bc90.

zhengchenyu mentioned this pull request Sep 20, 2023

MAPREDUCE-7453. Revert HADOOP-18649. #6102

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAPREDUCE-7453. Container logs are missing when yarn.app.container.lo… #6042

MAPREDUCE-7453. Container logs are missing when yarn.app.container.lo… #6042

zhengchenyu commented Sep 8, 2023 •

edited by slfan1989

slfan1989 commented Sep 8, 2023

hadoop-yetus commented Sep 8, 2023

zhengchenyu commented Sep 13, 2023 •

edited

slfan1989 commented Sep 14, 2023

slfan1989 commented Sep 15, 2023

ayushtkn left a comment

ayushtkn Sep 19, 2023

zhengchenyu commented Sep 19, 2023 •

edited

slfan1989 commented Sep 20, 2023

MAPREDUCE-7453. Container logs are missing when yarn.app.container.lo… #6042

MAPREDUCE-7453. Container logs are missing when yarn.app.container.lo… #6042

Conversation

zhengchenyu commented Sep 8, 2023 • edited by slfan1989

Description of PR

How was this patch tested?

For code changes:

slfan1989 commented Sep 8, 2023

hadoop-yetus commented Sep 8, 2023

zhengchenyu commented Sep 13, 2023 • edited

slfan1989 commented Sep 14, 2023

slfan1989 commented Sep 15, 2023

ayushtkn left a comment

Choose a reason for hiding this comment

ayushtkn Sep 19, 2023

Choose a reason for hiding this comment

zhengchenyu commented Sep 19, 2023 • edited

slfan1989 commented Sep 20, 2023

zhengchenyu commented Sep 8, 2023 •

edited by slfan1989

zhengchenyu commented Sep 13, 2023 •

edited

zhengchenyu commented Sep 19, 2023 •

edited