HDFS-15879. Exclude slow nodes when choose targets for blocks #2748

tomscut · 2021-03-06T01:44:55Z

Previously, we have monitored the slow nodes, related to HDFS-11194

We can use a thread to periodically collect these slow nodes into a set. Then use the set to filter out slow nodes when choose targets for blocks.

This feature can be configured to be turned on when needed.

tomscut · 2021-03-08T03:15:30Z

Failed junit tests:
hadoop.hdfs.server.namenode.TestFsck

This failed unit tests was unrelated to the change, and it worked fine locally.

tomscut · 2021-03-08T06:17:15Z

Hi @Hexiaoqiao , could you please help review the code? Thank you.

tomscut · 2021-03-09T14:57:32Z

Hi @dineshchitlangia @tasanuma @ayushtkn @qizhu-lucas , could you please help review the code? Thank you.

dineshchitlangia

@tomscut Thank you for contributing this improvement. Requested minor changes inline.

...hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java

hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

...hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SlowPeerTracker.java

tomscut · 2021-03-11T06:06:13Z

Thanks @dineshchitlangia for your review and advice. And I will fix it soon.

dineshchitlangia

+1 LGTM, thank you for making the changes.
I will leave it open for a few other folks to review this.

...ava/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java

tomscut · 2021-03-19T11:29:41Z

+1 LGTM, thank you for making the changes.
I will leave it open for a few other folks to review this.

Thanks @dineshchitlangia for your careful review.

hadoop-yetus · 2021-03-22T16:45:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 19s		#2748 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#2748
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-03-23T00:17:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 17s		#2748 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#2748
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/6/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-03-23T09:36:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 15s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 5s		trunk passed
+1 💚	mvnsite	1m 17s		trunk passed
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 23s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 13s		trunk passed
+1 💚	shadedclient	16m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 10s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 10s		the patch passed
+1 💚	compile	1m 3s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 52s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 535 unchanged - 0 fixed = 544 total (was 535)
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 13s		the patch passed
+1 💚	shadedclient	16m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	435m 44s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		521m 45s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestDecommissioningStatus
	hadoop.hdfs.TestViewDistributedFileSystemContract
	hadoop.hdfs.TestSnapshotCommands
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeWithHdfsScheme
	hadoop.hdfs.TestStateAlignmentContextWithHA
	hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
	hadoop.hdfs.server.namenode.TestNamenodeStorageDirectives
	hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS
	hadoop.hdfs.server.namenode.TestFileTruncate
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.server.datanode.TestIncrementalBrVariations
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.TestHDFSFileSystemContract
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.web.TestWebHdfsFileSystemContract

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/7/artifact/out/Dockerfile
GITHUB PR	#2748
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux eb754777450b 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3227d6d5086da84074281d780c4eb4127e154486
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/7/testReport/
Max. process+thread count	2677 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-03-23T19:29:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 1s		trunk passed
+1 💚	compile	1m 21s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 12s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 25s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 15s		trunk passed
+1 💚	shadedclient	18m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 57s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 535 unchanged - 0 fixed = 544 total (was 535)
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 17s		the patch passed
+1 💚	shadedclient	18m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	342m 8s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		434m 53s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.server.datanode.TestBlockRecovery
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/8/artifact/out/Dockerfile
GITHUB PR	#2748
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux dee14ef2a8cd 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5e95a30`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/8/testReport/
Max. process+thread count	1986 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tasanuma

Thanks for updating the PR, @tomscut.
I have one more request and commented on it.

...ava/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java

hadoop-yetus · 2021-03-26T09:59:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	2m 17s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 42s		trunk passed
+1 💚	compile	1m 26s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 13s		trunk passed
+1 💚	mvnsite	1m 43s		trunk passed
+1 💚	javadoc	1m 2s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 28s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	4m 2s		trunk passed
+1 💚	shadedclient	20m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 57s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 12 new + 535 unchanged - 0 fixed = 547 total (was 535)
+1 💚	mvnsite	1m 15s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 18s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 24s		the patch passed
+1 💚	shadedclient	19m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	395m 50s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		496m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestStateAlignmentContextWithHA
	hadoop.hdfs.web.TestWebHdfsFileSystemContract
	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
	hadoop.hdfs.server.namenode.TestFileTruncate
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.TestViewDistributedFileSystemContract
	hadoop.hdfs.server.datanode.TestBlockRecovery
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.datanode.TestIncrementalBrVariations
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/9/artifact/out/Dockerfile
GITHUB PR	#2748
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 52194d15325c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bd27b7d`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/9/testReport/
Max. process+thread count	1894 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tasanuma · 2021-03-26T10:27:40Z

@tomscut Looks good to me, except for the checkstyle issues.
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/9/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

Could you fix them? (I'm sorry that I should have mentioned it before.)
If you use IntelliJ, I recommend CheckStyle-IDEA plugin. The configuration file is hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml.

...hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java

hadoop-yetus · 2021-03-26T19:41:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 19s		trunk passed
+1 💚	compile	1m 29s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 7s		trunk passed
+1 💚	mvnsite	1m 34s		trunk passed
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 22s		trunk passed
+1 💚	shadedclient	19m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	1m 26s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 26s		the patch passed
+1 💚	compile	1m 20s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 58s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 535 unchanged - 0 fixed = 536 total (was 535)
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 34s		the patch passed
+1 💚	shadedclient	19m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	383m 11s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		478m 50s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestStateAlignmentContextWithHA
	hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.TestLeaseRecovery
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.datanode.TestBlockRecovery
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/10/artifact/out/Dockerfile
GITHUB PR	#2748
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 67cbb135696e 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `983d3df`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/10/testReport/
Max. process+thread count	1994 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-03-26T20:06:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 14s		trunk passed
+1 💚	compile	1m 27s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 40s		trunk passed
+1 💚	shadedclient	19m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	1m 21s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 21s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 3s		the patch passed
+1 💚	mvnsite	1m 32s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 59s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 49s		the patch passed
+1 💚	shadedclient	21m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	378m 1s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		477m 54s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestBlockScanner
	hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks
	hadoop.hdfs.TestViewDistributedFileSystem
	hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
	hadoop.hdfs.TestPersistBlocks
	hadoop.hdfs.server.datanode.TestBlockRecovery
	hadoop.hdfs.server.datanode.TestDataNodeUUID
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
	hadoop.hdfs.TestSnapshotCommands
	hadoop.hdfs.server.datanode.TestIncrementalBrVariations
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/11/artifact/out/Dockerfile
GITHUB PR	#2748
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 379440357f60 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c1400c9`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/11/testReport/
Max. process+thread count	1714 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2748/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2021-03-27T07:02:06Z

Hi @tasanuma , those failed unit tests are unrelated to the change, and they work fine locally. Please take a look at the new commit, thank you.

tasanuma

LGTM. +1.

tasanuma · 2021-03-27T08:57:51Z

Merged it. Thanks for your contribution, @tomscut!
Thanks for your review @dineshchitlangia!

tomscut · 2021-03-27T11:06:57Z

Thanks @tasanuma @dineshchitlangia !

…#2748) Reviewed-by: Dinesh Chitlangia <dineshc@apache.org> Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>

dineshchitlangia requested changes Mar 11, 2021

View reviewed changes

dineshchitlangia approved these changes Mar 18, 2021

View reviewed changes

tasanuma reviewed Mar 19, 2021

View reviewed changes

...ava/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java Outdated Show resolved Hide resolved

tomscut force-pushed the HDFS-15879 branch from 07c6aa7 to 3227d6d Compare March 23, 2021 00:53

HDFS-15879. Exclude slow nodes when choose targets for blocks

5e95a30

tomscut force-pushed the HDFS-15879 branch from 3227d6d to 5e95a30 Compare March 23, 2021 12:13

tasanuma reviewed Mar 24, 2021

View reviewed changes

...ava/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java Show resolved Hide resolved

...ava/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyExcludeSlowNodes.java Outdated Show resolved Hide resolved

Improve unit tests

bd27b7d

tasanuma reviewed Mar 26, 2021

View reviewed changes

...hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java Outdated Show resolved Hide resolved

tomscut added 2 commits March 26, 2021 19:41

fix checkstyle

983d3df

fix checkstyle

c1400c9

tasanuma approved these changes Mar 27, 2021

View reviewed changes

tasanuma merged commit 72037a6 into apache:trunk Mar 27, 2021

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15879. Exclude slow nodes when choose targets for blocks (apache…

789b939

…#2748) Reviewed-by: Dinesh Chitlangia <dineshc@apache.org> Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15879. Exclude slow nodes when choose targets for blocks #2748

HDFS-15879. Exclude slow nodes when choose targets for blocks #2748

tomscut commented Mar 6, 2021

tomscut commented Mar 8, 2021

tomscut commented Mar 8, 2021

tomscut commented Mar 9, 2021

dineshchitlangia left a comment

tomscut commented Mar 11, 2021

dineshchitlangia left a comment

tomscut commented Mar 19, 2021

hadoop-yetus commented Mar 22, 2021

hadoop-yetus commented Mar 23, 2021

hadoop-yetus commented Mar 23, 2021

hadoop-yetus commented Mar 23, 2021

tasanuma left a comment

hadoop-yetus commented Mar 26, 2021

tasanuma commented Mar 26, 2021

hadoop-yetus commented Mar 26, 2021

hadoop-yetus commented Mar 26, 2021

tomscut commented Mar 27, 2021

tasanuma left a comment

tasanuma commented Mar 27, 2021

tomscut commented Mar 27, 2021

HDFS-15879. Exclude slow nodes when choose targets for blocks #2748

HDFS-15879. Exclude slow nodes when choose targets for blocks #2748

Conversation

tomscut commented Mar 6, 2021

tomscut commented Mar 8, 2021

tomscut commented Mar 8, 2021

tomscut commented Mar 9, 2021

dineshchitlangia left a comment

Choose a reason for hiding this comment

tomscut commented Mar 11, 2021

dineshchitlangia left a comment

Choose a reason for hiding this comment

tomscut commented Mar 19, 2021

hadoop-yetus commented Mar 22, 2021

hadoop-yetus commented Mar 23, 2021

hadoop-yetus commented Mar 23, 2021

hadoop-yetus commented Mar 23, 2021

tasanuma left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Mar 26, 2021

tasanuma commented Mar 26, 2021

hadoop-yetus commented Mar 26, 2021

hadoop-yetus commented Mar 26, 2021

tomscut commented Mar 27, 2021

tasanuma left a comment

Choose a reason for hiding this comment

tasanuma commented Mar 27, 2021

tomscut commented Mar 27, 2021