HDFS-16324. Fix error log in BlockManagerSafeMode #3661

GuoPhilipse · 2021-11-15T13:05:44Z

Description of PR

if recheckInterval was set as invalid value, there will be warning log output, but the message seems not that proper ,we can improve it.

How was this patch tested?

No need test cases, just update warning log.

sync

GuoPhilipse · 2021-11-16T01:58:56Z

@tomscut Could you kindly help verify, the test error seems not related with the patch.
[ERROR] testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks) Time elapsed: 120.023 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 120000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.shell.SetReplication.waitForReplication(SetReplication.java:137) at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:78) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121) at org.apache.hadoop.fs.shell.Command.run(Command.java:179) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(TestUnderReplicatedBlocks.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)

tomscut · 2021-11-16T02:05:27Z

@tomscut Could you kindly help verify, the test error seems not related with the patch. [ERROR] testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks) Time elapsed: 120.023 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 120000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.shell.SetReplication.waitForReplication(SetReplication.java:137) at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:78) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121) at org.apache.hadoop.fs.shell.Command.run(Command.java:179) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(TestUnderReplicatedBlocks.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)

Can you commit an empty commit to trigger the builder again?

GuoPhilipse · 2021-11-16T02:23:19Z

@tomscut Could you kindly help verify, the test error seems not related with the patch. [ERROR] testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks) Time elapsed: 120.023 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 120000 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.shell.SetReplication.waitForReplication(SetReplication.java:137) at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:78) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121) at org.apache.hadoop.fs.shell.Command.run(Command.java:179) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(TestUnderReplicatedBlocks.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)

Can you commit an empty commit to trigger the builder again?

sure, have just triggered

GuoPhilipse · 2021-11-16T10:45:17Z

@tomscut the tests rerun ok 💯 :)

GuoPhilipse · 2021-11-18T00:56:14Z

cc @aajisaka @ayushtkn @goiri

… into fixwrongdecisionvalue

GuoPhilipse · 2021-11-25T11:44:09Z

ping @aajisaka @ayushtkn any new ideas?

ayushtkn

LGTM.
In general I think we can survive without a test case also for log level changes…
Will hold it for Akira to have a final look.

...fs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java

ayushtkn

Thanx @GuoPhilipse for the fix. have dropped a nit.
Will commit once addressed.

...fs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java

… into fixwrongdecisionvalue

GuoPhilipse · 2021-12-04T09:59:06Z

Thanx @GuoPhilipse for the fix. have dropped a nit. Will commit once addressed.

Thanks @ayushtkn for your careful review, have updated the code

hadoop-yetus · 2021-12-06T14:46:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 46s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 56s		trunk passed
+1 💚	compile	5m 43s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 17s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 23s		trunk passed
+1 💚	mvnsite	2m 35s		trunk passed
+1 💚	javadoc	1m 50s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 22s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 24s		trunk passed
+1 💚	shadedclient	24m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 20s		the patch passed
+1 💚	compile	5m 48s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 48s		the patch passed
+1 💚	compile	5m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 18s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 133 new + 270 unchanged - 0 fixed = 403 total (was 270)
+1 💚	mvnsite	2m 22s		the patch passed
+1 💚	javadoc	1m 33s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 9s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	6m 10s		the patch passed
+1 💚	shadedclient	24m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	227m 8s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		369m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/11/artifact/out/Dockerfile
GITHUB PR	#3661
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 5d2e06c05ecf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b92b7d6`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/11/testReport/
Max. process+thread count	3354 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

...fs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java

aajisaka · 2021-12-08T12:22:06Z

Merged. Thank you @GuoPhilipse for your contribution and thanks @tomscut @ayushtkn for your reviews.

Reviewed-by: tomscut <litao@bigo.sg> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

GuoPhilipse and others added 8 commits November 29, 2020 05:02

Merge pull request #1 from apache/trunk

3c51579

sync

Merge pull request #2 from apache/trunk

a449d04

sync

Merge pull request #3 from apache/trunk

d4f968b

sync

Merge pull request #4 from apache/trunk

d522abf

sync

Merge pull request #5 from apache/trunk

de633b8

sync

Merge pull request #6 from apache/trunk

66ce06b

sync

'fixwrongkey'

2fc6fa1

'addblockinfo'

93e21da

This comment has been minimized.

Sign in to view

tomscut approved these changes Nov 16, 2021

View reviewed changes

'triggerrebuild'

1ad22de

This comment has been minimized.

Sign in to view

ayushtkn approved these changes Nov 18, 2021

View reviewed changes

aajisaka approved these changes Nov 18, 2021

View reviewed changes

GuoPhilipse added 2 commits November 21, 2021 15:05

addtestcase

58e089e

removeunnecessarychange

5cd7060

This comment has been minimized.

Sign in to view

tclxgf13871 added 2 commits November 22, 2021 13:47

'triggerrebuild'

a911292

Merge branch 'fixwrongdecisionvalue' of github.com:GuoPhilipse/hadoop…

3f90c90

… into fixwrongdecisionvalue

This comment has been minimized.

Sign in to view

tclxgf13871 added 4 commits November 23, 2021 21:09

'improvecode'

84fe1c0

'improvecode1'

f634108

'modifyvariable'

e4fb5ce

'addtestcase'

84386f9

ayushtkn approved these changes Nov 25, 2021

View reviewed changes

tclxgf13871 added 2 commits November 26, 2021 10:13

'fixcheckstyle'

85c5224

'triggerrebuild'

561b370

aajisaka reviewed Nov 29, 2021

View reviewed changes

...fs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java Outdated Show resolved Hide resolved

tclxgf13871 added 2 commits November 29, 2021 16:37

'improveassert'

632afbe

'improveassert1'

df966ea

This comment has been minimized.

Sign in to view

'improveassert2'

ad6bebf

This comment has been minimized.

Sign in to view

ayushtkn reviewed Dec 4, 2021

View reviewed changes

...fs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java Outdated Show resolved Hide resolved

研发中心-郭飞-philipse and others added 7 commits December 4, 2021 16:08

fixcomment

fef812b

removeuselesscode

ea3260e

removeuselesscode1

ddcee71

fixcomments

bc99edc

Merge branch 'fixwrongdecisionvalue' of github.com:GuoPhilipse/hadoop…

d6ab918

… into fixwrongdecisionvalue

addspace

0dfb811

correctcode

06c21c8

This comment has been minimized.

Sign in to view

triggerrebuild

b92b7d6

This comment has been minimized.

Sign in to view

aajisaka reviewed Dec 8, 2021

View reviewed changes

...fs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java Outdated Show resolved Hide resolved

aajisaka approved these changes Dec 8, 2021

View reviewed changes

aajisaka changed the title ~~HDFS-16324. fix error log in BlockManagerSafeMode~~ HDFS-16324. Fix error log in BlockManagerSafeMode Dec 8, 2021

aajisaka merged commit 00d483d into apache:trunk Dec 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16324. Fix error log in BlockManagerSafeMode #3661

HDFS-16324. Fix error log in BlockManagerSafeMode #3661

GuoPhilipse commented Nov 15, 2021

This comment has been minimized.

GuoPhilipse commented Nov 16, 2021

tomscut commented Nov 16, 2021

GuoPhilipse commented Nov 16, 2021

This comment has been minimized.

GuoPhilipse commented Nov 16, 2021

GuoPhilipse commented Nov 18, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

GuoPhilipse commented Nov 25, 2021

ayushtkn left a comment

This comment has been minimized.

This comment has been minimized.

ayushtkn left a comment

GuoPhilipse commented Dec 4, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hadoop-yetus commented Dec 6, 2021

aajisaka commented Dec 8, 2021

HDFS-16324. Fix error log in BlockManagerSafeMode #3661

HDFS-16324. Fix error log in BlockManagerSafeMode #3661

Conversation

GuoPhilipse commented Nov 15, 2021

Description of PR

How was this patch tested?

This comment has been minimized.

GuoPhilipse commented Nov 16, 2021

tomscut commented Nov 16, 2021

GuoPhilipse commented Nov 16, 2021

This comment has been minimized.

GuoPhilipse commented Nov 16, 2021

GuoPhilipse commented Nov 18, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

GuoPhilipse commented Nov 25, 2021

ayushtkn left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

ayushtkn left a comment

Choose a reason for hiding this comment

GuoPhilipse commented Dec 4, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hadoop-yetus commented Dec 6, 2021

aajisaka commented Dec 8, 2021