YARN-11660. Fix huge performance regression for SingleConstraintAppPlacementAllocator #6623

zuston · 2024-03-12T08:19:27Z

Description of PR

JIRA: YARN-11660. Fix huge performance regression for SingleConstraintAppPlacementAllocator.

When using the SingleConstraintAppPlacementAllocator with scheduling request in our internal cluster, I found the huge performance regression from the metric of allocateAvgTime.

After digging this bug, I found this dangerous bug will always check the non-pending resource apps due to missing of desc pending resource for one app for the async scheduling threads.

How was this patch tested?

Has been applied in our internal cluster

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-03-12T11:12:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 49s		trunk passed
+1 💚	compile	0m 35s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	javadoc	0m 37s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 29s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 10s		trunk passed
+1 💚	shadedclient	19m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 28s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 26s		the patch passed
+1 💚	javadoc	0m 24s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 28s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 9s		the patch passed
+1 💚	shadedclient	19m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	88m 49s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		171m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/1/artifact/out/Dockerfile
GITHUB PR	#6623
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5a9aac54ba70 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4eba785`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/1/testReport/
Max. process+thread count	926 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2024-03-12T14:45:35Z

...p/yarn/server/resourcemanager/scheduler/placement/SingleConstraintAppPlacementAllocator.java

@@ -309,6 +309,10 @@ private void decreasePendingNumAllocation() {
    // Deduct pending #allocations by 1
    ResourceSizing sizing = schedulingRequest.getResourceSizing();
    sizing.setNumAllocations(sizing.getNumAllocations() - 1);
+
+    appSchedulingInfo.decPendingResource(


one line may be better.

…ntAllocator

hadoop-yetus · 2024-03-13T05:11:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 51s		trunk passed
+1 💚	compile	0m 34s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	javadoc	0m 35s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 31s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 8s		trunk passed
+1 💚	shadedclient	19m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 29s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 10s		the patch passed
+1 💚	shadedclient	19m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	86m 39s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		170m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/2/artifact/out/Dockerfile
GITHUB PR	#6623
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5dcf70804e69 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3cd69ab`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/2/testReport/
Max. process+thread count	939 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

dineshchitlangia

+1 LGTM, thanks @zuston for fixing this regression.

slfan1989 · 2024-03-15T00:15:10Z

@zuston Thanks for the contribution! @dineshchitlangia Thanks for the review!

github-actions bot added YARN trunk labels Mar 12, 2024

slfan1989 reviewed Mar 12, 2024

View reviewed changes

YARN-11660. Fix performance regression for SingleConstraintAppPlaceme…

3cd69ab

…ntAllocator

zuston force-pushed the yarn-11660 branch from 4eba785 to 3cd69ab Compare March 13, 2024 02:19

slfan1989 approved these changes Mar 14, 2024

View reviewed changes

dineshchitlangia approved these changes Mar 14, 2024

View reviewed changes

slfan1989 merged commit 783cc3e into apache:trunk Mar 15, 2024
1 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11660. Fix huge performance regression for SingleConstraintAppPlacementAllocator #6623

YARN-11660. Fix huge performance regression for SingleConstraintAppPlacementAllocator #6623

zuston commented Mar 12, 2024 •

edited by slfan1989

hadoop-yetus commented Mar 12, 2024

slfan1989 Mar 12, 2024

zuston Mar 13, 2024

hadoop-yetus commented Mar 13, 2024

dineshchitlangia left a comment

slfan1989 commented Mar 15, 2024

YARN-11660. Fix huge performance regression for SingleConstraintAppPlacementAllocator #6623

YARN-11660. Fix huge performance regression for SingleConstraintAppPlacementAllocator #6623

Conversation

zuston commented Mar 12, 2024 • edited by slfan1989

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Mar 12, 2024

slfan1989 Mar 12, 2024

Choose a reason for hiding this comment

zuston Mar 13, 2024

Choose a reason for hiding this comment

hadoop-yetus commented Mar 13, 2024

dineshchitlangia left a comment

Choose a reason for hiding this comment

slfan1989 commented Mar 15, 2024

zuston commented Mar 12, 2024 •

edited by slfan1989