YARN-11088. Introduce the config to control the AM allocated to non-e… #4105

zuston · 2022-03-24T14:35:06Z

…xclusive nodes

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

zuston · 2022-03-24T14:38:08Z

@9uapaw @szilard-nemeth If u have time, could u help review this ticket? Thanks

hadoop-yetus · 2022-03-24T19:13:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 48s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 23s		trunk passed
+1 💚	compile	9m 45s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	8m 30s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 43s		trunk passed
+1 💚	mvnsite	3m 18s		trunk passed
+1 💚	javadoc	2m 57s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 48s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 2s		trunk passed
+1 💚	shadedclient	22m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 9s		the patch passed
+1 💚	compile	9m 4s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	9m 4s		the patch passed
+1 💚	compile	8m 24s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	8m 24s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 36s		the patch passed
+1 💚	mvnsite	3m 1s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 42s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 32s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 23s		the patch passed
+1 💚	shadedclient	22m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	1m 10s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch passed.
+1 💚	unit	4m 59s		hadoop-yarn-common in the patch passed.
+1 💚	unit	116m 13s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		277m 6s

Reason	Tests
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/1/artifact/out/Dockerfile
GITHUB PR	#4105
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 340326807f2a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 5fa6d52b293e8d3eacfa798e3f8642fd470d5a93
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/1/testReport/
Max. process+thread count	929 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

brumi1024

Thanks @zuston for working on this! I had a few minor comments.

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

...hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java

...doop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java

zuston · 2022-03-28T13:55:59Z

Thanks for your review. The change will be uploaded as soon as possible. @brumi1024

zuston · 2022-03-29T15:43:01Z

Updated @brumi1024. If u have time, could you help review again.

brumi1024 · 2022-03-29T16:19:59Z

Thanks @zuston, the latest patch LGTM. Will commit when it gets a +1.

hadoop-yetus · 2022-03-29T20:01:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 59s		trunk passed
+1 💚	compile	9m 35s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	8m 30s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 42s		trunk passed
+1 💚	mvnsite	3m 17s		trunk passed
+1 💚	javadoc	3m 0s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 49s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 1s		trunk passed
+1 💚	shadedclient	22m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 9s		the patch passed
+1 💚	compile	9m 1s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	9m 1s		the patch passed
+1 💚	compile	8m 21s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	8m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 38s		the patch passed
+1 💚	mvnsite	3m 3s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 40s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 34s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 20s		the patch passed
+1 💚	shadedclient	22m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	1m 11s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch passed.
+1 💚	unit	4m 59s		hadoop-yarn-common in the patch passed.
+1 💚	unit	100m 48s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		261m 58s

Reason	Tests
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/2/artifact/out/Dockerfile
GITHUB PR	#4105
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 05158f4b465b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 6aae6de29110212863f216608144ce53d7c70c66
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/2/testReport/
Max. process+thread count	961 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

brumi1024 · 2022-03-29T20:39:00Z

@zuston can you please take a look at the unit test failure?

zuston · 2022-03-30T01:56:14Z

@brumi1024 Thanks for your quick reply.
The failure is due to the lack of yarn.resourcemanager.node-labels.am.default-node-label-expression in yarn-default.xml, which is introduced in #4060 by me.

#4060 (comment)

However it's removed when merging to trunk. Maybe i need to fix it with another PR and then merge this PR after that.
Do u think so? @brumi1024

brumi1024 · 2022-03-30T10:15:22Z

@zuston thanks, I see now. Yes, please open a different PR to fix it :) then a rebase should give this a green build as well.

cc: @9uapaw

9uapaw · 2022-03-30T10:15:35Z

@zuston It was my fault sorry, I thought it is not needed in case the default is empty.

…xclusive nodes

hadoop-yetus · 2022-03-31T06:56:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 12s		trunk passed
+1 💚	compile	9m 50s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	8m 29s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 42s		trunk passed
+1 💚	mvnsite	3m 21s		trunk passed
+1 💚	javadoc	2m 58s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 48s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 0s		trunk passed
+1 💚	shadedclient	22m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 9s		the patch passed
+1 💚	compile	9m 9s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	9m 9s		the patch passed
+1 💚	compile	8m 27s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	8m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 39s		the patch passed
+1 💚	mvnsite	3m 6s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 39s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 23s		the patch passed
+1 💚	shadedclient	25m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 12s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 0s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	115m 24s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+0 🆗	asflicense	0m 39s		ASF License check generated no output?
		283m 48s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/3/artifact/out/Dockerfile
GITHUB PR	#4105
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux af4d03027833 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `663cc3a`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/3/testReport/
Max. process+thread count	863 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4105/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

zuston · 2022-03-31T10:05:34Z

The test case failure looks unrelated with this PR. @brumi1024

brumi1024 · 2022-03-31T10:40:51Z

Thanks @zuston, committed to trunk.

zuston · 2022-03-31T11:55:59Z

Thanks @brumi1024

brumi1024 requested changes Mar 28, 2022

View reviewed changes

zuston requested a review from brumi1024 March 29, 2022 15:43

brumi1024 approved these changes Mar 29, 2022

View reviewed changes

zuston requested a review from brumi1024 March 30, 2022 01:56

zuston added 3 commits March 31, 2022 10:10

YARN-11088. Introduce the config to control the AM allocated to non-e…

c809f63

…xclusive nodes

Rename config

f79b3c0

Add test case

663cc3a

zuston force-pushed the YARN-11088 branch from 6aae6de to 663cc3a Compare March 31, 2022 02:11

brumi1024 closed this Mar 31, 2022

zuston deleted the YARN-11088 branch March 31, 2022 11:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11088. Introduce the config to control the AM allocated to non-e… #4105

YARN-11088. Introduce the config to control the AM allocated to non-e… #4105

zuston commented Mar 24, 2022

zuston commented Mar 24, 2022

hadoop-yetus commented Mar 24, 2022

brumi1024 left a comment

zuston commented Mar 28, 2022

zuston commented Mar 29, 2022

brumi1024 commented Mar 29, 2022

hadoop-yetus commented Mar 29, 2022

brumi1024 commented Mar 29, 2022

zuston commented Mar 30, 2022

brumi1024 commented Mar 30, 2022

9uapaw commented Mar 30, 2022

hadoop-yetus commented Mar 31, 2022

zuston commented Mar 31, 2022

brumi1024 commented Mar 31, 2022

zuston commented Mar 31, 2022

YARN-11088. Introduce the config to control the AM allocated to non-e… #4105

YARN-11088. Introduce the config to control the AM allocated to non-e… #4105

Conversation

zuston commented Mar 24, 2022

Description of PR

How was this patch tested?

For code changes:

zuston commented Mar 24, 2022

hadoop-yetus commented Mar 24, 2022

brumi1024 left a comment

Choose a reason for hiding this comment

zuston commented Mar 28, 2022

zuston commented Mar 29, 2022

brumi1024 commented Mar 29, 2022

hadoop-yetus commented Mar 29, 2022

brumi1024 commented Mar 29, 2022

zuston commented Mar 30, 2022

brumi1024 commented Mar 30, 2022

9uapaw commented Mar 30, 2022

hadoop-yetus commented Mar 31, 2022

zuston commented Mar 31, 2022

brumi1024 commented Mar 31, 2022

zuston commented Mar 31, 2022