YARN-11463. Node Labels root directory creation doesn't have a retry logic #5562

ashutoshcipher · 2023-04-17T14:42:23Z

Description of PR

Node Labels root directory creation doesn't have a retry logic

JIRA - YARN-11463

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…logic

brumi1024

@ashutoshcipher thanks for the patch! I had a comment.

brumi1024 · 2023-04-17T15:17:53Z

...p-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/AbstractFSNodeStore.java

-    // mkdir of root dir path
-    fs.mkdirs(fsWorkingPath);
+    // mkdir of root dir path with retry logic
+    int maxRetries = 3;


Can you please create a configuration entry for the retry count and the interval? Something like the
yarn.resourcemanager.zk-num-retries and yarn.resourcemanager.zk-retry-interval-ms parameters defined for ZK connections. That way the user could configure a longer period in case HDFS in safe mode for a few minutes when the RM tries to start.

Thanks @brumi1024 for your review. I hve made this change in last commit

hadoop-yetus · 2023-04-17T16:27:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 0s		trunk passed
+1 💚	compile	0m 51s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	0m 46s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 49s		trunk passed
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 43s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	1m 54s		trunk passed
+1 💚	shadedclient	21m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	0m 47s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 25s		the patch passed
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	javadoc	0m 38s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	0m 39s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	1m 45s		the patch passed
+1 💚	shadedclient	21m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	5m 29s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt	hadoop-yarn-common in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		103m 53s

Reason	Tests
Failed junit tests	hadoop.yarn.nodelabels.TestFileSystemNodeLabelsStore

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5562/1/artifact/out/Dockerfile
GITHUB PR	#5562
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8aba91f002da 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9b677dd`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5562/1/testReport/
Max. process+thread count	666 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5562/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

brumi1024

Thanks @ashutoshcipher for the update. Apart from two nits it looks good to me. Can you please check the test failure?

brumi1024 · 2023-04-18T10:14:10Z

...hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java

@@ -137,6 +137,16 @@ private static void addDeprecatedKeys() {
  // Resource types configs
  ////////////////////////////

+  public static final String NODE_STORE_ROOT_DIR_NUM_RETRIES =
+      YARN_PREFIX + "nodestore-rootdir.num-retries";


Nit: Since this is an RM config (capacity scheduler feature) I think we could use the yarn.resourcemanager prefix.

Makes sense. I will do that

brumi1024 · 2023-04-18T10:14:40Z

...hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java

+  public static final String NODE_STORE_ROOT_DIR_NUM_RETRIES =
+      YARN_PREFIX + "nodestore-rootdir.num-retries";
+
+  public static final int NODE_STORE_ROOT_DIR_NUM_DEFAULT_RETRIES = 3;


Nit: To be inline with the other retries I think 500/1000 would be a more appropriate default.

I will change it to 500

ashutoshcipher · 2023-04-18T11:14:08Z

Thanks @ashutoshcipher for the update. Apart from two nits it looks good to me. Can you please check the test failure?

Thanks @brumi1024 for review. I will look at it.

ashutoshcipher · 2023-04-20T19:05:25Z

Hi @brumi1024. Sorry for getting a bit late. I have addressed your comments. Please review. Thanks

hadoop-yetus · 2023-04-20T21:47:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 46s		trunk passed
+1 💚	compile	10m 36s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	9m 13s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	checkstyle	1m 45s		trunk passed
+1 💚	mvnsite	1m 58s		trunk passed
+1 💚	javadoc	1m 54s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 52s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	4m 13s		trunk passed
+1 💚	shadedclient	22m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 20s		the patch passed
+1 💚	compile	9m 49s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	9m 49s		the patch passed
+1 💚	compile	8m 56s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	8m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 42s		the patch passed
+1 💚	mvnsite	1m 47s		the patch passed
+1 💚	javadoc	1m 42s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 35s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	4m 28s		the patch passed
+1 💚	shadedclient	22m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 9s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 38s		hadoop-yarn-common in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		162m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5562/3/artifact/out/Dockerfile
GITHUB PR	#5562
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 6f4971f4a893 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `596a9fc`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5562/3/testReport/
Max. process+thread count	557 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5562/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

brumi1024 · 2023-04-21T12:03:44Z

Thanks @ashutoshcipher for the patch, merged to trunk.

ashutoshcipher · 2023-04-21T12:16:01Z

Thanks @brumi1024 for review and merge.

YARN-11463. Node Labels root directory creation doesn't have a retry …

9b677dd

…logic

brumi1024 requested changes Apr 17, 2023

View reviewed changes

Adding config

2b28097

brumi1024 reviewed Apr 18, 2023

View reviewed changes

Addressing comments

596a9fc

brumi1024 merged commit 964c190 into apache:trunk Apr 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11463. Node Labels root directory creation doesn't have a retry logic #5562

YARN-11463. Node Labels root directory creation doesn't have a retry logic #5562

ashutoshcipher commented Apr 17, 2023

brumi1024 left a comment

brumi1024 Apr 17, 2023 •

edited

Loading

ashutoshcipher Apr 18, 2023

hadoop-yetus commented Apr 17, 2023

brumi1024 left a comment

brumi1024 Apr 18, 2023

ashutoshcipher Apr 18, 2023

brumi1024 Apr 18, 2023

ashutoshcipher Apr 18, 2023

ashutoshcipher commented Apr 18, 2023

ashutoshcipher commented Apr 20, 2023

hadoop-yetus commented Apr 20, 2023

brumi1024 commented Apr 21, 2023

ashutoshcipher commented Apr 21, 2023

YARN-11463. Node Labels root directory creation doesn't have a retry logic #5562

YARN-11463. Node Labels root directory creation doesn't have a retry logic #5562

Conversation

ashutoshcipher commented Apr 17, 2023

Description of PR

For code changes:

brumi1024 left a comment

Choose a reason for hiding this comment

brumi1024 Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

ashutoshcipher Apr 18, 2023

Choose a reason for hiding this comment

hadoop-yetus commented Apr 17, 2023

brumi1024 left a comment

Choose a reason for hiding this comment

brumi1024 Apr 18, 2023

Choose a reason for hiding this comment

ashutoshcipher Apr 18, 2023

Choose a reason for hiding this comment

brumi1024 Apr 18, 2023

Choose a reason for hiding this comment

ashutoshcipher Apr 18, 2023

Choose a reason for hiding this comment

ashutoshcipher commented Apr 18, 2023

ashutoshcipher commented Apr 20, 2023

hadoop-yetus commented Apr 20, 2023

brumi1024 commented Apr 21, 2023

ashutoshcipher commented Apr 21, 2023

brumi1024 Apr 17, 2023 •

edited

Loading