YARN-8990. Fix fair scheduler race condition in app submit and queue …

[aajisaka]

…cleanup. (Contributed by Wilfred Spiegelenburg)

(cherry picked from commit 524a752)

Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java

Backport YARN-8990 to branch-3.2.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.2 Compile Tests _
+1 💚	mvninstall	32m 28s		branch-3.2 passed
+1 💚	compile	0m 50s		branch-3.2 passed
+1 💚	checkstyle	0m 39s		branch-3.2 passed
+1 💚	mvnsite	0m 54s		branch-3.2 passed
+1 💚	javadoc	0m 37s		branch-3.2 passed
+1 💚	spotbugs	1m 59s		branch-3.2 passed
+1 💚	shadedclient	17m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 59s		the patch passed
+1 💚	compile	0m 45s		the patch passed
+1 💚	javac	0m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	mvnsite	0m 55s		the patch passed
+1 💚	javadoc	0m 35s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 353 unchanged - 6 fixed = 353 total (was 359)
+1 💚	spotbugs	2m 13s		the patch passed
+1 💚	shadedclient	17m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	87m 1s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		179m 21s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2
	hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher
	hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestFSSchedulerConfigurationStore

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3254/1/artifact/out/Dockerfile
GITHUB PR	#3254
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 08e081a1e9a3 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / ee52048
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3254/1/testReport/
Max. process+thread count	799 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3254/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[aajisaka]

I ran the following three failed tests in branch-3.2 and all of them failed without modification.

• TestSystemMetricsPublisherForV2
• TestCombinedSystemMetricsPublisher
• TestFSSchedulerConfigurationStore

[aajisaka]

Note: Backported YARN-9338 to branch-3.2 to fix TestSystemMetricsPublisherForV2