HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. #5708

hfutatzhanghb · 2023-06-02T05:53:58Z

Description of PR

Currently, method getActualNonDfsUsed may return negative value, which can cause getRemainingReserved computed inaccurately.

The reason is FsVolumeImpl#getDfsUsed use GetSpaceUsed to compute the usage periodically, not in time.

How was this patch tested?

TestFsVolumeList
TestRoundRobinVolumeChoosingPolicy
TestAvailableSpaceVolumeChoosingPolicy
TestDatanodeReport
TestStorageBlockPoolUsageStdDev

hfutatzhanghb · 2023-06-02T05:58:26Z

hadoop-yetus · 2023-06-02T11:32:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 20s		trunk passed
+1 💚	compile	1m 15s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 8s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 7s		trunk passed
+1 💚	mvnsite	1m 17s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 15s		trunk passed
+1 💚	shadedclient	22m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 3s		the patch passed
+1 💚	compile	1m 9s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 9s		the patch passed
+1 💚	compile	1m 3s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	1m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 51s		the patch passed
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 4s		the patch passed
+1 💚	shadedclient	22m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	236m 29s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		337m 33s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/1/artifact/out/Dockerfile
GITHUB PR	#5708
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6a34cbf02637 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2648b81`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/1/testReport/
Max. process+thread count	3181 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hfutatzhanghb · 2023-06-02T12:34:37Z

The failed unit test was passed in my local.

hfutatzhanghb · 2023-06-07T02:20:43Z

@ayushtkn @Hexiaoqiao @zhangshuyan0 . Hi, sir. could you please take a look at this minor issue when you are not busy ? Thanks a lot.

Hexiaoqiao

LGTM. +1 from my side. Let's wait if other folks to feedback util two workdays.

hfutatzhanghb · 2023-06-07T03:59:07Z

LGTM. +1 from my side. Let's wait if other folks to feedback util two workdays.

Sir, thanks a lot for your reviewing.

ayushtkn

Why don't we fix getActualNonDfsUsed()? It should not return -ve, we can do that Math.max there?
Else this problem can be with getRemainingReserved() and getNonDfsUsed()?

hfutatzhanghb · 2023-06-07T10:07:40Z

Sir, thanks a lot for reviewing. Agree with you here, have modified code and run some UT in local.

hadoop-yetus · 2023-06-07T15:32:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 25s		trunk passed
+1 💚	compile	1m 15s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 12s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 32s		trunk passed
+1 💚	shadedclient	22m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 6s		the patch passed
+1 💚	compile	1m 2s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	1m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 51s		the patch passed
+1 💚	mvnsite	1m 6s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 2s		the patch passed
+1 💚	shadedclient	22m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	210m 58s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		321m 25s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/3/artifact/out/Dockerfile
GITHUB PR	#5708
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b3dc36d2642d 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c4cb2b8`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/3/testReport/
Max. process+thread count	2835 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-06-07T16:11:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 43s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 20s		trunk passed
+1 💚	mvnsite	1m 24s		trunk passed
+1 💚	javadoc	1m 21s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 40s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 54s		trunk passed
+1 💚	shadedclient	27m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 57s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 16s		the patch passed
+1 💚	shadedclient	23m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	239m 30s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		361m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/2/artifact/out/Dockerfile
GITHUB PR	#5708
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e7c321129ef3 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c4cb2b8`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/2/testReport/
Max. process+thread count	3269 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5708/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

LGTM

HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value.

2648b81

Hexiaoqiao approved these changes Jun 7, 2023

View reviewed changes

Hexiaoqiao mentioned this pull request Jun 7, 2023

HDFS-17037. Consider nonDfsUsed when running balancer. #5715

Merged

ayushtkn reviewed Jun 7, 2023

View reviewed changes

hfutatzhanghb added 2 commits June 7, 2023 18:08

modify getActualNonDfsUsed.

156243a

modify getActualNonDfsUsed.

c4cb2b8

zhangshuyan0 approved these changes Jun 8, 2023

View reviewed changes

ayushtkn approved these changes Jun 8, 2023

View reviewed changes

ayushtkn merged commit ddae78b into apache:trunk Jun 8, 2023

hfutatzhanghb deleted the HDFS-17035 branch June 8, 2023 09:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. #5708

HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. #5708

hfutatzhanghb commented Jun 2, 2023 •

edited

Loading

hfutatzhanghb commented Jun 2, 2023

hadoop-yetus commented Jun 2, 2023

hfutatzhanghb commented Jun 2, 2023

hfutatzhanghb commented Jun 7, 2023

Hexiaoqiao left a comment

hfutatzhanghb commented Jun 7, 2023

ayushtkn left a comment

hfutatzhanghb commented Jun 7, 2023

hadoop-yetus commented Jun 7, 2023

hadoop-yetus commented Jun 7, 2023

ayushtkn left a comment

HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. #5708

HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. #5708

Conversation

hfutatzhanghb commented Jun 2, 2023 • edited Loading

Description of PR

How was this patch tested?

hfutatzhanghb commented Jun 2, 2023

hadoop-yetus commented Jun 2, 2023

hfutatzhanghb commented Jun 2, 2023

hfutatzhanghb commented Jun 7, 2023

Hexiaoqiao left a comment

Choose a reason for hiding this comment

hfutatzhanghb commented Jun 7, 2023

ayushtkn left a comment

Choose a reason for hiding this comment

hfutatzhanghb commented Jun 7, 2023

hadoop-yetus commented Jun 7, 2023

hadoop-yetus commented Jun 7, 2023

ayushtkn left a comment

Choose a reason for hiding this comment

hfutatzhanghb commented Jun 2, 2023 •

edited

Loading