HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics #2676

tomscut · 2021-02-03T01:10:54Z

JIRA: https://issues.apache.org/jira/browse/HDFS-15814

For ease of use, especially for small clusters, we can change some parameters(MIN_OUTLIER_DETECTION_DISKS, SLOW_DISK_LOW_THRESHOLD_MS) configurable.

hadoop-yetus · 2021-02-03T06:20:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 49s		trunk passed
+1 💚	compile	1m 21s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚	checkstyle	1m 12s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	shadedclient	17m 34s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 28s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+0 🆗	spotbugs	3m 13s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 11s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚	javac	1m 8s		the patch passed
-0 ⚠️	checkstyle	1m 4s	/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 611 unchanged - 2 fixed = 616 total (was 613)
+1 💚	mvnsite	1m 15s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 27s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 23s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
+1 💚	findbugs	3m 17s		the patch passed
			_ Other Tests _
-1 ❌	unit	215m 33s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		308m 54s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2676/1/artifact/out/Dockerfile
GITHUB PR	#2676
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 16c46c9927cc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f37bf65`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2676/1/testReport/
Max. process+thread count	2252 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2676/1/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2021-02-04T02:25:13Z

Failed junit tests:
hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

Sorry. I didn't update those four unit tests.

tomscut · 2021-02-09T02:11:28Z

@hanishakoneru
Hi Hanisha Koneru, could you please review the code for me? Thanks!

arp7 · 2021-02-09T15:42:34Z

The change looks pretty good to me. Please file an Apache jira.

The failed tests look unrelated to your change.

tomscut · 2021-02-10T02:44:06Z

The change looks pretty good to me. Please file an Apache jira.

The failed tests look unrelated to your change.

Thank you for your review!

This is the Apache jira: https://issues.apache.org/jira/browse/HDFS-15814

...p-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeDiskMetrics.java

arp7 · 2021-02-12T16:08:00Z

Can you look at the 5 checkstyle warnings flagged above whether they were introduced by the patch?

hadoop-yetus · 2021-02-13T18:12:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	24m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 2s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 9s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	shadedclient	18m 23s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 23s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	3m 15s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 13s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 14s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 9s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 9s		the patch passed
+1 💚	checkstyle	1m 5s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 612 unchanged - 2 fixed = 612 total (was 614)
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	shadedclient	14m 54s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	3m 12s		the patch passed
			_ Other Tests _
-1 ❌	unit	213m 49s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		330m 25s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2676/2/artifact/out/Dockerfile
GITHUB PR	#2676
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux df5455cfa77e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f9a073c`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2676/2/testReport/
Max. process+thread count	2456 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2676/2/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tomscut · 2021-02-14T04:02:12Z

Can you look at the 5 checkstyle warnings flagged above whether they were introduced by the patch?

Hi, Arpit. Thank you for your advice. I fixed the checkstyle warnings. And the failed test (hadoop.hdfs.TestRollingUpgrade ) is unrelated to the change.

dineshchitlangia

+1 LGTM
Thanks @tomscut for the contribution and @arp7 for the reviews.

tomscut · 2021-02-14T12:43:08Z

Thanks @arp7 and @dineshchitlangia .

Make some parameters configurable for DataNodeDiskMetrics

f0844cf

arp7 self-requested a review February 9, 2021 15:42

arp7 changed the title ~~Make some parameters configurable for DataNodeDiskMetrics~~ HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics Feb 10, 2021

arp7 reviewed Feb 10, 2021

View reviewed changes

...p-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeDiskMetrics.java Show resolved Hide resolved

adjust for checkstyle

825ec20

dineshchitlangia approved these changes Feb 14, 2021

View reviewed changes

dineshchitlangia merged commit c3134ab into apache:trunk Feb 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics #2676

HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics #2676

tomscut commented Feb 3, 2021

hadoop-yetus commented Feb 3, 2021

tomscut commented Feb 4, 2021

tomscut commented Feb 9, 2021

arp7 commented Feb 9, 2021

tomscut commented Feb 10, 2021

arp7 commented Feb 12, 2021

hadoop-yetus commented Feb 13, 2021

tomscut commented Feb 14, 2021

dineshchitlangia left a comment

tomscut commented Feb 14, 2021

HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics #2676

HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics #2676

Conversation

tomscut commented Feb 3, 2021

hadoop-yetus commented Feb 3, 2021

tomscut commented Feb 4, 2021

tomscut commented Feb 9, 2021

arp7 commented Feb 9, 2021

tomscut commented Feb 10, 2021

arp7 commented Feb 12, 2021

hadoop-yetus commented Feb 13, 2021

tomscut commented Feb 14, 2021

dineshchitlangia left a comment

Choose a reason for hiding this comment

tomscut commented Feb 14, 2021