HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement #2626

aajisaka · 2021-01-19T10:10:49Z

JIRA: https://issues.apache.org/jira/browse/HDFS-15783

Remove the usage of TreeSet#add, its time complexity is O(logN).
~~This fix is similar to https://issues.apache.org/jira/browse/HDFS-14102~~

…kPlacement.

hadoop-yetus · 2021-01-19T13:32:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	37m 28s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	1m 40s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	1m 22s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 55s		trunk passed
+1 💚	mvnsite	1m 34s		trunk passed
+1 💚	shadedclient	21m 4s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 6s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	4m 0s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 57s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 20s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 10s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	1m 10s		the patch passed
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	mvnsite	1m 11s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	14m 48s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 31s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 29s		the patch passed
			_ Other Tests _
-1 ❌	unit	99m 57s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		200m 34s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestMaintenanceState
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestDFSUpgradeFromImage

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2626/1/artifact/out/Dockerfile
GITHUB PR	#2626
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b0a758fd0c48 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a326f22`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2626/1/testReport/
Max. process+thread count	4321 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2626/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-01-19T18:35:19Z

...ava/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java

+    // Count locations on different racks.
+    final long rackCount = Arrays.stream(locs)
+        .map(DatanodeInfo::getNetworkLocation).distinct().count();
+    return new BlockPlacementStatusDefault(Math.toIntExact(rackCount),


You are following the same as in HDFS-14102 but I'd prefer extracting the int casting instead of doing it in the call.
A little more verbose but I would extract it.

Removed the casting. Thank you @goiri.

hadoop-yetus · 2021-01-20T08:20:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	2m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 15s		trunk passed
+1 💚	compile	1m 26s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	1m 29s		trunk passed
+1 💚	shadedclient	17m 55s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 0s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 32s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	3m 15s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 13s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	1m 6s		the patch passed
+1 💚	checkstyle	0m 42s		the patch passed
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 8s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	1m 21s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	3m 10s		the patch passed
			_ Other Tests _
-1 ❌	unit	135m 29s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		231m 20s

Reason	Tests
Failed junit tests	hadoop.hdfs.qjournal.server.TestJournalNodeSync
	hadoop.hdfs.TestViewDistributedFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2626/2/artifact/out/Dockerfile
GITHUB PR	#2626
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 3651290512cc 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `68bc721`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2626/2/testReport/
Max. process+thread count	3553 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2626/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka · 2021-01-21T01:08:05Z

Merged. Thanks @goiri.

…kPlacement (#2626) (cherry picked from commit 45b3a84)

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBloc…

180ac79

…kPlacement.

goiri reviewed Jan 19, 2021

View reviewed changes

Removed lambda to extract casting

881075f

goiri approved these changes Jan 20, 2021

View reviewed changes

aajisaka merged commit 45b3a84 into apache:trunk Jan 21, 2021

aajisaka deleted the HDFS-15783 branch January 21, 2021 01:08

aajisaka added a commit that referenced this pull request Jan 21, 2021

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBloc…

03c84fb

…kPlacement (#2626) (cherry picked from commit 45b3a84)

aajisaka added a commit that referenced this pull request Jan 21, 2021

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBloc…

f1dff46

…kPlacement (#2626) (cherry picked from commit 45b3a84)

aajisaka added a commit that referenced this pull request Jan 21, 2021

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBloc…

33a71c8

…kPlacement (#2626) (cherry picked from commit 45b3a84)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement #2626

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement #2626

aajisaka commented Jan 19, 2021 •

edited

hadoop-yetus commented Jan 19, 2021

goiri Jan 19, 2021

aajisaka Jan 20, 2021

hadoop-yetus commented Jan 20, 2021

aajisaka commented Jan 21, 2021

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement #2626

HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement #2626

Conversation

aajisaka commented Jan 19, 2021 • edited

hadoop-yetus commented Jan 19, 2021

goiri Jan 19, 2021

Choose a reason for hiding this comment

aajisaka Jan 20, 2021

Choose a reason for hiding this comment

hadoop-yetus commented Jan 20, 2021

aajisaka commented Jan 21, 2021

aajisaka commented Jan 19, 2021 •

edited