YARN-10835. Pass user defined variables of yarn.nodemanager.env-whitelist along jobs

[symious]

What changes were proposed in this pull request?

The configuration of "yarn.nodemanager.env-whitelist" has been helpful in running multi version Hadoop on Yarn. But the pain point is that users need to use different configurations to run jobs of different versions of Hadoop.

The first idea is to have SRE prepared different versions of configurations and let users specify different environment variables of "HADOOP_CONF_DIR", this shifts the burden from users to SRE that they have to maintain different versions of configurations with only one or two configurations changed.

This ticket proposes to export the value of the environment variable in "yarn.nodemanager.env-whitelist" via the environment variable or Java properties of the client submitting jobs. So that SRE only has to maintain one version of configuration and users can easily define Hadoop version with one export in an environment variable or Java property.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/YARN-10835

How was this patch tested?

Add unit test to pass Java properties to ContainerLaunchContext.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 46s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 7s		trunk passed
+1 💚	compile	20m 54s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	18m 11s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 53s		trunk passed
+1 💚	mvnsite	2m 46s		trunk passed
+1 💚	javadoc	2m 26s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 18s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 28s		trunk passed
+1 💚	shadedclient	14m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 38s		the patch passed
+1 💚	compile	20m 14s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	20m 14s		the patch passed
+1 💚	compile	18m 8s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	18m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 51s		the patch passed
+1 💚	mvnsite	2m 43s		the patch passed
+1 💚	javadoc	2m 25s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 17s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	5m 1s		the patch passed
+1 💚	shadedclient	14m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 11s		hadoop-yarn-api in the patch passed.
+1 💚	unit	27m 34s		hadoop-yarn-client in the patch passed.
+1 💚	unit	8m 38s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		219m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/artifact/out/Dockerfile
GITHUB PR	#3154
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux af1ca7a11478 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 1f43d95
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/testReport/
Max. process+thread count	700 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 14s		#3154 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#3154
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3154/1/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[github-actions]

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.