HDFS-16397. Reconfig slow disk parameters for datanode #3828

tomscut · 2021-12-27T01:42:06Z

In large clusters, rolling restart datanodes takes long time. We can make slow peers parameters and slow disks parameters in datanode reconfigurable to facilitate cluster operation and maintenance.

hadoop-yetus · 2021-12-27T09:28:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 14s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 20s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 59s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	1m 3s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 32s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 26s		trunk passed
+1 💚	shadedclient	25m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 26s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 26s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 52s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 133 unchanged - 2 fixed = 136 total (was 135)
+1 💚	mvnsite	1m 20s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 25s		the patch passed
+1 💚	shadedclient	25m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	339m 35s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		464m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/1/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 4d5e49aab07e 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / f8794fea0c6f0d54e121fc8df35f734ac75af553
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/1/testReport/
Max. process+thread count	1975 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-27T09:45:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 43s		trunk passed
+1 💚	compile	1m 29s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 28s		trunk passed
+1 💚	javadoc	1m 3s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 12s		trunk passed
+1 💚	shadedclient	22m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 19s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 52s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 134 unchanged - 2 fixed = 137 total (was 136)
+1 💚	mvnsite	1m 20s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 12s		the patch passed
+1 💚	shadedclient	22m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	231m 0s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		330m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/2/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 9d4a4c4f909f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 8a6cff984d78ac01152c5ed4043083643044921b
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/2/testReport/
Max. process+thread count	2979 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-27T12:49:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 18s		trunk passed
+1 💚	compile	1m 25s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 23s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 3s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 10s		trunk passed
+1 💚	shadedclient	23m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	1m 31s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 31s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 0s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 133 unchanged - 2 fixed = 136 total (was 135)
+1 💚	mvnsite	1m 22s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 34s		the patch passed
+1 💚	shadedclient	22m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	224m 56s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		325m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/4/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 4ce0d8933a10 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 39f001f9081a953251c31171212860af62b87a3d
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/4/testReport/
Max. process+thread count	3617 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-27T13:09:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 58s		trunk passed
+1 💚	compile	1m 36s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 29s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 6s		trunk passed
+1 💚	mvnsite	1m 32s		trunk passed
+1 💚	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 28s		trunk passed
+1 💚	shadedclient	25m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 22s		the patch passed
+1 💚	compile	1m 26s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 26s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 55s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 133 unchanged - 2 fixed = 136 total (was 135)
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	0m 57s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 29s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 39s		the patch passed
+1 💚	shadedclient	25m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	235m 21s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		346m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/3/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux b121ed2a6bb3 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 39f001f9081a953251c31171212860af62b87a3d
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/3/testReport/
Max. process+thread count	3620 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-02-15T19:34:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 18s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 20s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 30s		trunk passed
+1 💚	javadoc	1m 4s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 19s		trunk passed
+1 💚	shadedclient	22m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 16s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 21s		the patch passed
+1 💚	compile	1m 11s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 11s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 51s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 136 unchanged - 2 fixed = 139 total (was 138)
+1 💚	mvnsite	1m 17s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 24s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 13s		the patch passed
+1 💚	shadedclient	22m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	226m 8s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		325m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/5/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux dcca2c281b61 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1053659`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/5/testReport/
Max. process+thread count	3263 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-02-18T06:04:01Z

Hi @tasanuma @ayushtkn @jojochuang @Hexiaoqiao , could you please review this PR? Thanks.

tomscut · 2022-02-18T07:42:13Z

This feature is similar to HDFS-16396. Reconfig slow peer parameters for datanode.

tasanuma

@tomscut Sorry for being late, and thanks for working on this. It mostly looks good to me. I left some minor comments.

...project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java

tasanuma · 2022-02-24T01:19:22Z

.../hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java

+    return isEnabled;
+  }
+
+  public int getSampleRangeMax() {


Please add VisibleForTesting.

.../hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java

tomscut · 2022-02-24T01:58:33Z

@tomscut Sorry for being late, and thanks for working on this. It mostly looks good to me. I left some minor comments.

Thank you @tasanuma very much for your review and comments. I fixed them.

tasanuma · 2022-02-24T02:16:38Z

@tomscut Thanks for updating the PR. +1, pending Jenkins.

hadoop-yetus · 2022-02-24T07:33:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 9s		trunk passed
+1 💚	compile	1m 27s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 19s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 3s		trunk passed
+1 💚	mvnsite	1m 28s		trunk passed
+1 💚	javadoc	1m 4s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 19s		trunk passed
+1 💚	shadedclient	22m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 18s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 52s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 136 unchanged - 2 fixed = 138 total (was 138)
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	0m 51s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 23s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 16s		the patch passed
-1 ❌	shadedclient	19m 11s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	239m 46s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		335m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/6/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 9aa8b04b73af 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d07e0be`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/6/testReport/
Max. process+thread count	4099 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-02-24T08:57:15Z

It looks like there are some errors. Let me retrigger it.

hadoop-yetus · 2022-02-24T11:06:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 43s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 18s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 29s		trunk passed
+1 💚	javadoc	1m 5s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 30s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	25m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 31s		the patch passed
+1 💚	compile	1m 40s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 40s		the patch passed
+1 💚	compile	1m 28s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 1s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 136 unchanged - 2 fixed = 138 total (was 138)
+1 💚	mvnsite	1m 33s		the patch passed
+1 💚	javadoc	1m 3s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 15s		the patch passed
+1 💚	shadedclient	27m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	17m 2s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+0 🆗	asflicense	0m 29s		ASF License check generated no output?
		125m 57s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestReadStripedFileWithMissingBlocks
	hadoop.hdfs.client.impl.TestBlockReaderFactory

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/7/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 8f8069a8fa46 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `08411f9`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/7/testReport/
Max. process+thread count	931 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-02-24T11:30:40Z

Hi @tasanuma those failed unit tests seem related to OOM. Do I need to trigger the builder again? Thanks.

tasanuma

Thanks for updating PR. I think the result is enough. The failed tests seem not to be related. +1.

tasanuma · 2022-02-24T16:23:40Z

Merged. Thanks for your contribution, @tomscut!

hadoop-yetus · 2022-02-24T17:36:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 2s		trunk passed
+1 💚	compile	1m 30s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 19s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 2s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 31s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	23m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 27s		the patch passed
+1 💚	compile	1m 35s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 35s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 59s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 136 unchanged - 2 fixed = 138 total (was 138)
+1 💚	mvnsite	1m 30s		the patch passed
+1 💚	javadoc	1m 1s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 44s		the patch passed
+1 💚	shadedclient	25m 43s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	35m 43s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+0 🆗	asflicense	0m 28s		ASF License check generated no output?
		140m 22s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStorageStateRecovery
	hadoop.hdfs.TestReadStripedFileWithMissingBlocks
	hadoop.hdfs.TestGetBlocks
	hadoop.hdfs.TestReadStripedFileWithDNFailure
	hadoop.hdfs.client.impl.TestBlockReaderLocal
	hadoop.hdfs.web.TestWebHDFSForHA

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/8/artifact/out/Dockerfile
GITHUB PR	#3828
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux c937a0d6c41b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9acc63a`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/8/testReport/
Max. process+thread count	1456 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3828/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2022-02-25T00:28:03Z

Thanks @tasanuma for reviewing and merging it.

(cherry picked from commit 6b07c85)

tomscut force-pushed the HDFS-16397 branch from 0d7246a to 39f001f Compare December 27, 2021 07:22

HDFS-16397. Reconfig slow disk parameters for datanode

1053659

tomscut force-pushed the HDFS-16397 branch from 39f001f to 1053659 Compare February 15, 2022 14:08

tasanuma self-requested a review February 19, 2022 02:07

tasanuma reviewed Feb 24, 2022

View reviewed changes

add @VisibleForTesting, refine unit test, fix checkstyles

d07e0be

empty commit

08411f9

empty commit

9acc63a

tasanuma approved these changes Feb 24, 2022

View reviewed changes

tasanuma merged commit 6b07c85 into apache:trunk Feb 24, 2022

tasanuma pushed a commit that referenced this pull request Feb 25, 2022

HDFS-16397. Reconfig slow disk parameters for datanode (#3828)

74f5f90

(cherry picked from commit 6b07c85)

cbaenziger pushed a commit to cbaenziger/hadoop that referenced this pull request Mar 8, 2022

HDFS-16397. Reconfig slow disk parameters for datanode (apache#3828)

7c59629

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16397. Reconfig slow disk parameters for datanode (apache#3828)

7bb81df

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16397. Reconfig slow disk parameters for datanode #3828

HDFS-16397. Reconfig slow disk parameters for datanode #3828

tomscut commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Feb 15, 2022

tomscut commented Feb 18, 2022

tomscut commented Feb 18, 2022

tasanuma left a comment

tasanuma Feb 24, 2022

tomscut commented Feb 24, 2022

tasanuma commented Feb 24, 2022

hadoop-yetus commented Feb 24, 2022

tomscut commented Feb 24, 2022

hadoop-yetus commented Feb 24, 2022

tomscut commented Feb 24, 2022

tasanuma left a comment

tasanuma commented Feb 24, 2022

hadoop-yetus commented Feb 24, 2022

tomscut commented Feb 25, 2022

HDFS-16397. Reconfig slow disk parameters for datanode #3828

HDFS-16397. Reconfig slow disk parameters for datanode #3828

Conversation

tomscut commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Dec 27, 2021

hadoop-yetus commented Feb 15, 2022

tomscut commented Feb 18, 2022

tomscut commented Feb 18, 2022

tasanuma left a comment

Choose a reason for hiding this comment

tasanuma Feb 24, 2022

Choose a reason for hiding this comment

tomscut commented Feb 24, 2022

tasanuma commented Feb 24, 2022

hadoop-yetus commented Feb 24, 2022

tomscut commented Feb 24, 2022

hadoop-yetus commented Feb 24, 2022

tomscut commented Feb 24, 2022

tasanuma left a comment

Choose a reason for hiding this comment

tasanuma commented Feb 24, 2022

hadoop-yetus commented Feb 24, 2022

tomscut commented Feb 25, 2022