YARN-11670. Add CallerContext in NodeManager #6688

yangjiandan · 2024-03-29T02:24:26Z

Description of PR

Currently, MR and Spark have added caller context, enabling tracing of HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, operators from NodeManagers cannot be identified in the audit log. For example, HDFS operations issued from NodeManagers during resource localization cannot be identified.

How was this patch tested?

current ut
system test in my local env

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

dineshchitlangia

+1 LGTM, pending CI

hadoop-yetus · 2024-03-29T04:11:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 48s		trunk passed
+1 💚	compile	0m 49s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 24s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	0m 54s		trunk passed
+1 💚	shadedclient	19m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 22s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 47s		the patch passed
+1 💚	compile	0m 44s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 15s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18)
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	0m 51s		the patch passed
+1 💚	shadedclient	20m 8s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	22m 18s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		106m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/1/artifact/out/Dockerfile
GITHUB PR	#6688
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 52f0adffb427 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `38d1079`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/1/testReport/
Max. process+thread count	552 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

yangjiandan · 2024-03-29T05:55:56Z

@dineshchitlangia Thank you very much for your review. In my test environment, when running MR jobs, some of the HDFS-audit logs are as follows:

2024-03-28 22:31:09,716 INFO FSNamesystem.audit: allowed=true	ugi=hadoop (auth:SIMPLE)	ip=/172.16.70.128	cmd=getfileinfo	src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.splitmetainfo	dst=null	perm=null	proto=rpc	callerContext=nodemanager_hadoop01
2024-03-28 22:31:09,717 INFO FSNamesystem.audit: allowed=true	ugi=hadoop (auth:SIMPLE)	ip=/172.16.70.128	cmd=getfileinfo	src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.splitmetainfo	dst=null	perm=null	proto=rpc	callerContext=nodemanager_hadoop01
2024-03-28 22:31:09,717 INFO FSNamesystem.audit: allowed=true	ugi=hadoop (auth:SIMPLE)	ip=/172.16.70.128	cmd=open	src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.splitmetainfo	dst=null	perm=null	proto=rpc	callerContext=nodemanager_hadoop01
2024-03-28 22:31:09,726 INFO FSNamesystem.audit: allowed=true	ugi=hadoop (auth:SIMPLE)	ip=/172.16.70.128	cmd=getfileinfo	src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.jar	dst=null	perm=null	proto=rpc	callerContext=nodemanager_hadoop01
2024-03-28 22:31:09,727 INFO FSNamesystem.audit: allowed=true	ugi=hadoop (auth:SIMPLE)	ip=/172.16.70.128	cmd=getfileinfo	src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.jar	dst=null	perm=null	proto=rpc	callerContext=nodemanager_hadoop01
2024-03-28 22:31:09,728 INFO FSNamesystem.audit: allowed=true	ugi=hadoop (auth:SIMPLE)	ip=/172.16.70.128	cmd=open	src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.jar	dst=null	perm=null	proto=rpc	callerContext=nodemanager_hadoop01

slfan1989 · 2024-04-02T05:21:54Z

@dineshchitlangia Can you help review this pr again? Thank you very much!

dineshchitlangia · 2024-04-02T05:25:03Z

@yangjiandan can you please address the new check style violation?

hadoop-yetus · 2024-04-02T10:48:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	55m 18s		trunk passed
+1 💚	compile	1m 42s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	1m 42s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 49s		trunk passed
+1 💚	mvnsite	0m 51s		trunk passed
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 43s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 34s		trunk passed
+1 💚	shadedclient	44m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	compile	1m 29s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	1m 29s		the patch passed
+1 💚	compile	1m 35s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	1m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 31s		the patch passed
+1 💚	mvnsite	0m 51s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 37s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 48s		the patch passed
+1 💚	shadedclient	45m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	28m 41s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		197m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/2/artifact/out/Dockerfile
GITHUB PR	#6688
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 83331f45e46b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `aac21ea`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/2/testReport/
Max. process+thread count	532 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

yangjiandan · 2024-04-02T11:56:51Z

@dineshchitlangia @slfan1989 Thank you for your review. The style issue has been addressed and corrected. Could you please take another look?

slfan1989 · 2024-04-02T14:42:38Z

LGTM.

yangjiandan · 2024-04-03T07:59:11Z

Thank you, @slfan1989 and @dineshchitlangia, for reviewing my merge request and for your approval. I appreciate your time and feedback.

dineshchitlangia

+1 LGTM

dineshchitlangia · 2024-04-09T02:51:17Z

Thanks @yangjiandan for contribution and @whbing & @slfan1989 for your reviews.

YARN-11670. Add CallerContext in NodeManager

38d1079

github-actions bot added YARN trunk labels Mar 29, 2024

dineshchitlangia reviewed Mar 29, 2024

View reviewed changes

slfan1989 approved these changes Mar 29, 2024

View reviewed changes

fix checkstyle issue

aac21ea

whbing approved these changes Apr 8, 2024

View reviewed changes

dineshchitlangia approved these changes Apr 9, 2024

View reviewed changes

dineshchitlangia merged commit 3f8af73 into apache:trunk Apr 9, 2024
1 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11670. Add CallerContext in NodeManager #6688

YARN-11670. Add CallerContext in NodeManager #6688

yangjiandan commented Mar 29, 2024 •

edited

dineshchitlangia left a comment

hadoop-yetus commented Mar 29, 2024

yangjiandan commented Mar 29, 2024 •

edited

slfan1989 commented Apr 2, 2024

dineshchitlangia commented Apr 2, 2024

hadoop-yetus commented Apr 2, 2024

yangjiandan commented Apr 2, 2024

slfan1989 commented Apr 2, 2024

yangjiandan commented Apr 3, 2024

dineshchitlangia left a comment

dineshchitlangia commented Apr 9, 2024

YARN-11670. Add CallerContext in NodeManager #6688

YARN-11670. Add CallerContext in NodeManager #6688

Conversation

yangjiandan commented Mar 29, 2024 • edited

Description of PR

How was this patch tested?

For code changes:

dineshchitlangia left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Mar 29, 2024

yangjiandan commented Mar 29, 2024 • edited

slfan1989 commented Apr 2, 2024

dineshchitlangia commented Apr 2, 2024

hadoop-yetus commented Apr 2, 2024

yangjiandan commented Apr 2, 2024

slfan1989 commented Apr 2, 2024

yangjiandan commented Apr 3, 2024

dineshchitlangia left a comment

Choose a reason for hiding this comment

dineshchitlangia commented Apr 9, 2024

yangjiandan commented Mar 29, 2024 •

edited

yangjiandan commented Mar 29, 2024 •

edited