YARN-11391. Support using DNS to resolve yarn services to IP addresses. #5297

lnbest0707 · 2023-01-12T23:05:39Z

Description of PR

The patch reuses the resolver introduced from the HDFS side and apply similar logic to YARN RM service addresses resolving.
To utilize the YARN DNS support and use DNS as endpoint, simply upgrade the hadoop binary and revise configs from, for example:

<property>
  <name>yarn.resourcemanager.address.rm1</name>
  <value>rm1_address:8032</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address.rm1</name>
  <value>rm1_address:8030</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
  <value>rm1_address:8031</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address.rm1</name>
  <value>rm1_address:8033</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>rm1_address:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.https.address.rm1</name>
  <value>rm1_address:8090</value>
</property>
<property>
  <name>yarn.resourcemanager.address.rm2</name>
  <value>rm2_address:8032</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address.rm2</name>
  <value>rm2_address:8030</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
  <value>rm2_address:8031</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address.rm2</name>
  <value>rm2_address:8033</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>rm2_address:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.https.address.rm2</name>
  <value>rm2_address:8090</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>

to:

<property>
  <name>yarn.resourcemanager.address.rm1</name>
  <value>rm_multi_a_dns:8032</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address.rm1</name>
  <value>rm_multi_a_dns:8030</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
  <value>rm_multi_a_dns:8031</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address.rm1</name>
  <value>rm_multi_a_dns:8033</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>rm_multi_a_dns:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.https.address.rm1</name>
  <value>rm_multi_a_dns:8090</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.resolve-needed</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.resolver.useFQDN</name>
  <value>true</value> # required in secure mode
</property>
<property>
  <name>yarn.resourcemanager.ha.refresh-period-ms</name>
  <value>180000</value> # 3 min
</property>

where rm_multi_a_dns is a multi-A DNS record for rm1_address and rm2_address. This means the following output on the terminal.

$ dig +short <rm_multi_a_dns> | xargs -n +1 dig +short -x | sort
<rm1_address>
<rm2_address>

For the newly introduced flags, please refer to yarn-default.xml.

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

slfan1989 · 2023-01-13T03:20:46Z

@lnbest0707 Thanks for your contribution, we are waiting for the compilation result of yetus.

hadoop-yetus · 2023-01-13T03:25:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 42s		Maven dependency ordering for branch
+1 💚	mvninstall	31m 12s		trunk passed
+1 💚	compile	23m 8s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 38s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	4m 52s		trunk passed
+1 💚	mvnsite	3m 57s		trunk passed
-1 ❌	javadoc	1m 14s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 39s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	6m 39s		trunk passed
+1 💚	shadedclient	24m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 39s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 34s		the patch passed
+1 💚	compile	22m 25s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 25s		the patch passed
+1 💚	compile	20m 30s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 37s	/results-checkstyle-root.txt	root: The patch generated 2 new + 184 unchanged - 0 fixed = 186 total (was 184)
+1 💚	mvnsite	3m 46s		the patch passed
-1 ❌	javadoc	1m 4s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 42s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
-1 ❌	spotbugs	2m 4s	/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	25m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 28s		hadoop-common in the patch passed.
+1 💚	unit	1m 19s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 42s		hadoop-yarn-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		258m 25s

Reason	Tests
SpotBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
	Inconsistent synchronization of org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.rmServiceIds; locked 90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:[line 147]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5297/1/artifact/out/Dockerfile
GITHUB PR	#5297
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux d9480f7f53d9 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d8c801d`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5297/1/testReport/
Max. process+thread count	1278 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5297/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2023-01-13T04:00:31Z

@lnbest0707 We need to fix the checkstyle issue.

hadoop-yetus · 2023-01-17T22:29:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 6s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 40s		trunk passed
+1 💚	compile	24m 31s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	21m 27s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	3m 47s		trunk passed
+1 💚	mvnsite	3m 36s		trunk passed
-1 ❌	javadoc	1m 15s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 41s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	6m 38s		trunk passed
+1 💚	shadedclient	24m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 33s		the patch passed
+1 💚	compile	22m 27s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 27s		the patch passed
+1 💚	compile	20m 26s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 37s		the patch passed
+1 💚	mvnsite	3m 41s		the patch passed
-1 ❌	javadoc	1m 5s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 40s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
-1 ❌	spotbugs	2m 5s	/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	24m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 30s		hadoop-common in the patch passed.
+1 💚	unit	1m 18s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 43s		hadoop-yarn-common in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		259m 50s

Reason	Tests
SpotBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
	Inconsistent synchronization of org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.rmServiceIds; locked 90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:90% of time Unsynchronized access at RequestHedgingRMFailoverProxyProvider.java:[line 147]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5297/2/artifact/out/Dockerfile
GITHUB PR	#5297
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux e1a91b710522 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2b60df3`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5297/2/testReport/
Max. process+thread count	1278 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5297/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

lnbest0707 · 2023-01-30T20:06:19Z

@slfan1989 style issue already fixed, others issues should not be related and not able to be resolved by rebasing. Could you please review and how can I merge the change? Thanks

lnbest0707 · 2023-02-15T20:35:11Z

@slfan1989 friendly remind to merge the PR if appropriate.

github-actions · 2025-10-28T00:22:09Z

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

Xin Gao added 3 commits January 12, 2023 14:55

YARN-11391 Add yarn RM DNS support

1d1624e

YARN-11391 Fix Jenkins reported issue

609df5d

YARN-11391 Improve logging format

d8c801d

slfan1989 changed the title ~~Yarn 11391~~ YARN-11391. Support using DNS to resolve yarn services to IP addresses. Jan 13, 2023

slfan1989 approved these changes Jan 13, 2023

View reviewed changes

Fix code style issues

2b60df3

github-actions bot added the Stale label Oct 28, 2025

github-actions bot closed this Oct 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11391. Support using DNS to resolve yarn services to IP addresses. #5297

YARN-11391. Support using DNS to resolve yarn services to IP addresses. #5297

Uh oh!

lnbest0707 commented Jan 12, 2023

Uh oh!

slfan1989 commented Jan 13, 2023

Uh oh!

hadoop-yetus commented Jan 13, 2023

Uh oh!

slfan1989 commented Jan 13, 2023

Uh oh!

hadoop-yetus commented Jan 17, 2023

Uh oh!

lnbest0707 commented Jan 30, 2023

Uh oh!

lnbest0707 commented Feb 15, 2023

Uh oh!

github-actions bot commented Oct 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YARN-11391. Support using DNS to resolve yarn services to IP addresses. #5297

YARN-11391. Support using DNS to resolve yarn services to IP addresses. #5297

Uh oh!

Conversation

lnbest0707 commented Jan 12, 2023

Description of PR

How was this patch tested?

For code changes:

Uh oh!

slfan1989 commented Jan 13, 2023

Uh oh!

hadoop-yetus commented Jan 13, 2023

Uh oh!

slfan1989 commented Jan 13, 2023

Uh oh!

hadoop-yetus commented Jan 17, 2023

Uh oh!

lnbest0707 commented Jan 30, 2023

Uh oh!

lnbest0707 commented Feb 15, 2023

Uh oh!

github-actions bot commented Oct 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants