YARN-11000. Replace queue resource calculation logic in updateClusterResource #5783

tomicooler · 2023-06-27T08:48:22Z

Description of PR

The new queue resource calculation logic is capable of calculating the queue resources based on mixed resource types. The queues under one parent can also mix their modes (e.g.: root.queue1=percentage root.queue2=weight root.queue3=absolute).

Precedence is: absolute -> percentage -> weight.

Note that the calculation is done by a single resource type at once.

Design doc: YARN-10888

Example

yarn.scheduler.capacity.root.default.capacity         = [memory=1w, vcores=4]
yarn.scheduler.capacity.root.test_1.capacity          = [memory=16384, vcores=100%]
yarn.scheduler.capacity.root.test_2.capacity          = [memory=3w, vcores=12]
yarn.scheduler.capacity.root.test_1.test_1_1.capacity = [memory=1w, vcores=1w]
yarn.scheduler.capacity.root.test_1.test_1_2.capacity = [memory=50%, vcores=2]
yarn.scheduler.capacity.root.test_1.test_1_3.capacity = [memory=12288, vcores=86%]

Further examples: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySchedulerMixedMode.java

The feature is behind a feature flag, it is disabled by default.

yarn.scheduler.capacity.legacy-queue-mode.enabled = true/false

Note that the mixed mode is not yet supported for AQCv2 (or v1) templates. YARN-11520

Key changes

The capacity and absoluteCapacity (and maxApplications and other derived properties) of a queue is calculated from the effectiveMinResource (= the current available resource for a given queue based on the queue hierarchy configuration and the available cluster resource).
When there is no cluster resources available yet, the maxApplications are set to the configured values, so applications can be submitted.
Changes in the scheduler-response: the capacity and absoluteCapacity resembles the effective values based on the queue configs and cluster resources. The normalizedWeight is not filled in the scheduler-response when queues are in weight mode.

How was this patch tested?

I'll run all the unit test in both Legacy and the "Uniform" queue calculation mode in this PR.
Follow-up ticket for some additional tests: YARN-11521

Acknowledges

The current state of this PR was a team effort, thanks to:

@9uapaw
@brumi1024
@K0K0V0K

Follow up items

YARN-11514 Extend SchedulerResponse with capacityVector
YARN-11520 Support capacity vector for AQCv2 dynamic templates
YARN-11521 Create a test set that runs with both Legacy/Uniform queue calculation
YARN-11522 Update documentation for the New capacity modes in CS

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'YARN-11000. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…Resource Change-Id: If92b4fb22e66d7f9318bf10419990672ee988c6a

hadoop-yetus · 2023-06-27T12:22:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 37 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 14s		trunk passed
+1 💚	compile	1m 5s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 52s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 1s		trunk passed
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 46s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 10s		trunk passed
+1 💚	shadedclient	25m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	0m 56s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 56s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 46s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 48s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 34 new + 747 unchanged - 23 fixed = 781 total (was 770)
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 39s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 0s		the patch passed
+1 💚	shadedclient	25m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	101m 58s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		212m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/1/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint
uname	Linux 93af61232d45 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a29af42`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/1/testReport/
Max. process+thread count	904 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Change-Id: I7cb6cb73293c4368c69b9344923e60a3297b9a94

Change-Id: Ibb6673638faeaad6f0f9b677d9f75e47132967be

… THIS) Change-Id: I944fda71addcbe13d1665d440317b71daf11697b

hadoop-yetus · 2023-06-27T17:29:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 41 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	53m 32s		trunk passed
+1 💚	compile	1m 3s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 9s		trunk passed
+1 💚	shadedclient	39m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 49s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 779 unchanged - 23 fixed = 787 total (was 802)
+1 💚	mvnsite	0m 49s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 40s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 59s		the patch passed
+1 💚	shadedclient	39m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	104m 14s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		256m 20s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreation
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig
	hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
	hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/2/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint
uname	Linux f439441a3326 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `da47ec6`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/2/testReport/
Max. process+thread count	908 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

… (REVERT THIS)" This reverts commit da47ec6.

Change-Id: I0a4f9c2460facda9dacf7d2845f58c2d9de589f5

Change-Id: Ie55a648ddde96cad76068b251fcfa8ad86ee4947

Change-Id: Ie7a14bb2bc5a5a928bcb6db5360bdcf94177e96a

This fixes the test issue: testDynamicAutoCreatedQueueRecoveryWithDefaultQueue (clusterResource was zero during the failover) Change-Id: I8341ef2d2364fe6b3a6b5c36624fd4053ac6ac16

…ue between the new and legacy queue mode Change-Id: I4d06b10d016c3a4386fe7261a1e322a57e41420f

Change-Id: I96dc7b9bb4c6695b943a979a979d75936f091b6a

Change-Id: I014fdc581ccfbc21ece7d20886d22c232c7b0a91

hadoop-yetus · 2023-06-29T17:52:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 49 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	50m 3s		trunk passed
+1 💚	compile	1m 4s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 4s		trunk passed
+1 💚	shadedclient	39m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 49s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 12 new + 856 unchanged - 24 fixed = 868 total (was 880)
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 45s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 40s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 57s		the patch passed
+1 💚	shadedclient	40m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	102m 40s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		251m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/3/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux c52bda787775 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3fc5216`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/3/testReport/
Max. process+thread count	863 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Change-Id: Ib93930e95e009d81122fdf640a01389bb88fba20

This reverts commit d9c9945.

tomicooler · 2023-06-29T18:02:56Z

Note: +1 Overall with the new queue calculation mode enabled. 💚

hadoop-yetus · 2023-06-29T22:46:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	jsonlint	0m 0s		jsonlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 49 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	50m 18s		trunk passed
+1 💚	compile	1m 2s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 1s		trunk passed
+1 💚	shadedclient	38m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 49s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 857 unchanged - 23 fixed = 865 total (was 880)
+1 💚	mvnsite	0m 49s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 58s		the patch passed
+1 💚	shadedclient	39m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	102m 27s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		250m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/4/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux 7e92986c05c8 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `84d1bb3`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/4/testReport/
Max. process+thread count	909 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tomicooler · 2023-06-30T05:24:03Z

Green with Legacy mode as well. 👍

Change-Id: Id1711c95808d4d84b63cd3073203733c819bdcb6

Change-Id: I0d95a3f14bf92a73a6c8fcb1111a6c5670663f07

hadoop-yetus · 2023-06-30T15:23:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 49 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 56s		trunk passed
+1 💚	compile	1m 2s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 59s		trunk passed
+1 💚	mvnsite	0m 57s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 59s		trunk passed
+1 💚	shadedclient	39m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	0m 55s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 55s		the patch passed
+1 💚	compile	0m 48s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 51s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 857 unchanged - 23 fixed = 865 total (was 880)
+1 💚	mvnsite	0m 49s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 40s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 59s		the patch passed
+1 💚	shadedclient	39m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	102m 35s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		250m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/5/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux 3c436f503ef3 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ed55c3e`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/5/testReport/
Max. process+thread count	920 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

...java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractLeafQueue.java

...va/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java

...a/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java

.../org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java

...a/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservationQueue.java

...adoop/yarn/server/resourcemanager/scheduler/capacity/AbsoluteResourceCapacityCalculator.java

…egacy Change-Id: I084176714fa1de3d67e5cfca2799d899fc5ddb57

Change-Id: I4aca138d1856761af165479063c98c32f6b20753

Change-Id: I9f8b763d1a51d6773db1b68e713ad02559f5fc3e

Change-Id: Iedbb9b891d7dc9ce78b43fdd337b9b922616032b

hadoop-yetus · 2023-07-03T20:21:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 49 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	50m 25s		trunk passed
+1 💚	compile	1m 4s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 1s		trunk passed
+1 💚	shadedclient	39m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s		the patch passed
+1 💚	compile	0m 56s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 56s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 49s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 857 unchanged - 23 fixed = 865 total (was 880)
+1 💚	mvnsite	0m 52s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 42s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 0s		the patch passed
+1 💚	shadedclient	39m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	102m 24s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		249m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/6/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux 6aecf7461d50 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `04fca6b`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/6/testReport/
Max. process+thread count	926 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Change-Id: I7f375f86f2ef7a927cc885d5b71ba2291ea075af

hadoop-yetus · 2023-07-04T01:00:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 49 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	53m 48s		trunk passed
+1 💚	compile	1m 2s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 55s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 46s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 3s		trunk passed
+1 💚	shadedclient	39m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 50s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 857 unchanged - 23 fixed = 865 total (was 880)
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 57s		the patch passed
+1 💚	shadedclient	39m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	102m 52s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		254m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/7/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux 92f965168a8c 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `02b60d3`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/7/testReport/
Max. process+thread count	927 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

This reverts commit 02b60d3.

2800 -> 1600 Change-Id: I5790df9a293fc5cfbfa8b4209139ad879d1e07a8

hadoop-yetus · 2023-07-04T09:42:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	25m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 49 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 28s		trunk passed
+1 💚	compile	1m 3s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	0m 59s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 1s		trunk passed
+1 💚	shadedclient	41m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 51s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 50s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 857 unchanged - 23 fixed = 865 total (was 880)
+1 💚	mvnsite	0m 49s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 57s		the patch passed
+1 💚	shadedclient	39m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	119m 23s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		292m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/8/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux 369307a92aee 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `09210de`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/8/testReport/
Max. process+thread count	920 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-07-04T12:33:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	23m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	jsonlint	0m 0s		jsonlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 60 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	51m 12s		trunk passed
+1 💚	compile	1m 2s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 2s		trunk passed
+1 💚	shadedclient	39m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 56s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 56s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 50s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 992 unchanged - 23 fixed = 1001 total (was 1015)
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 58s		the patch passed
+1 💚	shadedclient	39m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	103m 28s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		274m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/9/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux 4a68590b2e5a 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ce98e11`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/9/testReport/
Max. process+thread count	938 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

...adoop/yarn/server/resourcemanager/scheduler/capacity/AbsoluteResourceCapacityCalculator.java

...n/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java

szilard-nemeth · 2023-07-04T17:39:38Z

...g/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java

+    nm1 = new MockNM("127.0.0.1:1234", 2 * 1024,
+        rm.getResourceTrackerService());
+    MockNM nm2 = new MockNM("127.0.0.2:1234", 2 * 1024,
+        rm.getResourceTrackerService());


Nit: Can fit in previous line.

szilard-nemeth · 2023-07-04T17:39:56Z

...g/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java

+        .of(NodeId.newInstance("127.0.0.2", 0), Sets.newHashSet(LABEL_LY)));
+
+    MockNM nm3 = new MockNM("127.0.0.2:1234", 128 * 1024,
+        rm.getResourceTrackerService());


Nit: Can fit in previous line.

szilard-nemeth · 2023-07-04T17:40:03Z

...g/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java

+
+    // Default partition
+    MockNM nm4 = new MockNM("127.0.0.3:1234", 128 * 1024,
+        rm.getResourceTrackerService());


Nit: Can fit in previous line.

...g/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java

szilard-nemeth · 2023-07-04T17:43:34Z

...est/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java

+
+      csConf.setCapacity(ROOT + DOT + B, 90);
+      csConf.setMaximumCapacity(ROOT + DOT + B,
+          100);


Nit: Can fit in previous line.

szilard-nemeth

Left some minor comments.

Change-Id: I3dac094e9f1023df4941c7597020554787da4ba9

Change-Id: I9e8eac5f45296cc1fbec88122c6ea9d3d577c154

Change-Id: I231aa662d5a9ce606c9d53a8ca4c55525272e675

Change-Id: Ia6d8390411f632d076f85fe8dd1a3f78448da432

Change-Id: Iaf0036abb67b1242fec2507ac32c2054ee16a27a

Change-Id: I0425f0e2edc04588e787e213ed1d293ccb7fb2de

Change-Id: I984d61e447b4437d22b187147bba4e04b64ba184

Change-Id: I32962351dfe5394bf6bf6f4c0fc2dca9f82bdad6

tomicooler · 2023-07-05T08:41:24Z

@szilard-nemeth thanks for the review. I added the fixes, there was no big change, so I don't think we need to run the tests 2x times (legacy/non-legacy).

hadoop-yetus · 2023-07-05T12:50:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 60 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 50s		trunk passed
+1 💚	compile	1m 4s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	2m 0s		trunk passed
+1 💚	shadedclient	39m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 51s		the patch passed
+1 💚	compile	0m 55s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 55s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 50s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 993 unchanged - 23 fixed = 1002 total (was 1016)
+1 💚	mvnsite	0m 49s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 42s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 58s		the patch passed
+1 💚	shadedclient	39m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	102m 40s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		249m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/10/artifact/out/Dockerfile
GITHUB PR	#5783
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint xmllint
uname	Linux e6be65ee3f59 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b48fbfb`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/10/testReport/
Max. process+thread count	912 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

szilard-nemeth · 2023-07-05T23:45:58Z

@tomicooler ,
Thanks for the fixes.
Checked the following commits separately, ofc I haven't checked the whole patch again:
4529eba (#5783)
77fc689 (#5783)
414e043 (#5783)
b33e1bf (#5783)
442e93d (#5783)
ed814e9 (#5783)
c3d0a63 (#5783)
b48fbfb (#5783)

Latest patch LGTM, +1.

brumi1024 · 2023-07-06T08:16:39Z

Thanks @9uapaw for the original patch, @tomicooler for the updates and test fixes, @K0K0V0K for the help and additional test cases and @szilard-nemeth for the review. Merging this to trunk.

…Resource (apache#5783) Contributed by: Andras Gyori, Tamas Domok, Bence Kosztolnik and Benjamin Teke --------- Co-authored-by: Benjamin Teke <benjamin.teke@gmail.com>

YARN-11000. Replace queue resource calculation logic in updateCluster…

a29af42

…Resource Change-Id: If92b4fb22e66d7f9318bf10419990672ee988c6a

brumi1024 mentioned this pull request Jun 27, 2023

YARN-11000. Replace queue resource calculation logic in updateClusterResource #3618

Closed

4 tasks

tomicooler added 3 commits June 27, 2023 14:49

fix blank lines

487cb71

Change-Id: I7cb6cb73293c4368c69b9344923e60a3297b9a94

checkstyle fixes

55a8fc3

Change-Id: Ibb6673638faeaad6f0f9b677d9f75e47132967be

TEMPORARY CHANGE: ENABLE NEW QUEUE CALCULATION FOR A TEST RUN (REVERT…

da47ec6

… THIS) Change-Id: I944fda71addcbe13d1665d440317b71daf11697b

tomicooler and others added 9 commits June 28, 2023 07:57

Revert "TEMPORARY CHANGE: ENABLE NEW QUEUE CALCULATION FOR A TEST RUN…

15b153a

… (REVERT THIS)" This reverts commit da47ec6.

REVERT THIS LATER

d9c9945

Change-Id: I0a4f9c2460facda9dacf7d2845f58c2d9de589f5

Fix reservations.

31c4964

Change-Id: Ie55a648ddde96cad76068b251fcfa8ad86ee4947

fix TestRMWebServicesForCSWithPartitions

4b7337c

Change-Id: Ie7a14bb2bc5a5a928bcb6db5360bdcf94177e96a

Fix Legacy AQC: the capacitVector must be parsed from the template.

4591b0a

This fixes the test issue: testDynamicAutoCreatedQueueRecoveryWithDefaultQueue (clusterResource was zero during the failover) Change-Id: I8341ef2d2364fe6b3a6b5c36624fd4053ac6ac16

Add AQCv2 recovery test.

fb423e9

TestRMWebServicesCapacitySched better capacities no more rounding iss…

655321a

…ue between the new and legacy queue mode Change-Id: I4d06b10d016c3a4386fe7261a1e322a57e41420f

TestRMWebServicesCapacitySchedLegacyQueueCreation add cluster resource

b2fb700

Change-Id: I96dc7b9bb4c6695b943a979a979d75936f091b6a

TestRMWebServicesCapacitySchedDynamicConfig make it green for both mode

3fc5216

Change-Id: I014fdc581ccfbc21ece7d20886d22c232c7b0a91

tomicooler added 2 commits June 29, 2023 20:01

checkstyle fixes

6aba692

Change-Id: Ib93930e95e009d81122fdf640a01389bb88fba20

Revert "REVERT THIS LATER"

84d1bb3

This reverts commit d9c9945.

tomicooler added 2 commits June 30, 2023 13:06

remove unused leaf queue parameter

65f1ae2

Change-Id: Id1711c95808d4d84b63cd3073203733c819bdcb6

deprecated -> legacymode

ed55c3e

Change-Id: I0d95a3f14bf92a73a6c8fcb1111a6c5670663f07

tomicooler commented Jul 3, 2023

View reviewed changes

tomicooler added 4 commits July 3, 2023 16:48

review fix: refreshAfterResourceCalculation is only called from non-l…

e59d0a8

…egacy Change-Id: I084176714fa1de3d67e5cfca2799d899fc5ddb57

review fix: remove unnecessary code change

589e09d

Change-Id: I4aca138d1856761af165479063c98c32f6b20753

review fix: TestReservationQueue

d730374

Change-Id: I9f8b763d1a51d6773db1b68e713ad02559f5fc3e

review fix: absolute resorcce calc

04fca6b

Change-Id: Iedbb9b891d7dc9ce78b43fdd337b9b922616032b

LEGACY OFF (REVERT LATER)

02b60d3

Change-Id: I7f375f86f2ef7a927cc885d5b71ba2291ea075af

tomicooler added 2 commits July 4, 2023 06:47

Revert "LEGACY OFF (REVERT LATER)"

09210de

This reverts commit 02b60d3.

capacity unit tests fix some MockRM thread leakeage

ce98e11

2800 -> 1600 Change-Id: I5790df9a293fc5cfbfa8b4209139ad879d1e07a8

szilard-nemeth reviewed Jul 4, 2023

View reviewed changes

szilard-nemeth requested changes Jul 4, 2023

View reviewed changes

tomicooler added 8 commits July 5, 2023 09:28

review fix: 1e-1f -> epsilon

4529eba

Change-Id: I3dac094e9f1023df4941c7597020554787da4ba9

reviewfix: skip test

77fc689

Change-Id: I9e8eac5f45296cc1fbec88122c6ea9d3d577c154

review fix: io exception is not thrown

414e043

Change-Id: I231aa662d5a9ce606c9d53a8ca4c55525272e675

reviewfix: queue hierarchy comment

b33e1bf

Change-Id: Ia6d8390411f632d076f85fe8dd1a3f78448da432

review fix: better explanation

442e93d

Change-Id: Iaf0036abb67b1242fec2507ac32c2054ee16a27a

review fix: better explanation

ed814e9

Change-Id: I0425f0e2edc04588e787e213ed1d293ccb7fb2de

explain Assert.fails

c3d0a63

Change-Id: I984d61e447b4437d22b187147bba4e04b64ba184

explain it better

b48fbfb

Change-Id: I32962351dfe5394bf6bf6f4c0fc2dca9f82bdad6

brumi1024 requested a review from szilard-nemeth July 5, 2023 21:26

brumi1024 merged commit a36d8ad into apache:trunk Jul 6, 2023

YARN-11000. Replace queue resource calculation logic in updateClusterResource #5783

YARN-11000. Replace queue resource calculation logic in updateClusterResource #5783

Uh oh!

Conversation

tomicooler commented Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Key changes

How was this patch tested?

Acknowledges

Follow up items

For code changes:

Uh oh!

hadoop-yetus commented Jun 27, 2023

Uh oh!

hadoop-yetus commented Jun 27, 2023

Uh oh!

hadoop-yetus commented Jun 29, 2023

Uh oh!

tomicooler commented Jun 29, 2023

Uh oh!

hadoop-yetus commented Jun 29, 2023

Uh oh!

tomicooler commented Jun 30, 2023

Uh oh!

hadoop-yetus commented Jun 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jul 3, 2023

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

szilard-nemeth Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

szilard-nemeth Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

szilard-nemeth Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

szilard-nemeth Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

szilard-nemeth left a comment

Choose a reason for hiding this comment

Uh oh!

tomicooler commented Jul 5, 2023

Uh oh!

hadoop-yetus commented Jul 5, 2023

Uh oh!

szilard-nemeth commented Jul 5, 2023

Uh oh!

brumi1024 commented Jul 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tomicooler commented Jun 27, 2023 •

edited

Loading