HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed #5283

haiyang1987 · 2023-01-07T04:56:18Z

Description of PR

A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" was introduced in HDFS-16881.
However, this parameter was not added to hdfs-default.xml, cause run TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed.

…instXml failed

hadoop-yetus · 2023-01-07T11:44:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 45s		trunk passed
+1 💚	compile	1m 27s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	1m 30s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 30s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	65m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	1m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	1m 20s		the patch passed
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	23m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	297m 29s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 51s		The patch does not generate ASF License warnings.
		406m 39s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.namenode.TestCheckpoint

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/1/artifact/out/Dockerfile
GITHUB PR	#5283
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 79911b2d84da 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `de0da0a`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/1/testReport/
Max. process+thread count	3126 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-01-07T12:52:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 52s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	71m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	1m 20s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	1m 20s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	1m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	1m 19s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 24s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	26m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	364m 42s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		470m 33s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.balancer.TestBalancerService

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/2/artifact/out/Dockerfile
GITHUB PR	#5283
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 18ab3cc48ac8 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `de0da0a`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/2/testReport/
Max. process+thread count	1940 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

szetszwo

@haiyang1987 , thanks a lot for fixing the bug! Please see the two comments inlined.

szetszwo · 2023-01-07T13:32:50Z

hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

+  <value>1000</value>
+  <description>
+    If an external AccessControlEnforcer runs for a long time to check permission with the FSnamesystem lock,
+    we will print a WARN message when it happens. This sets how long must be run for logging to occur.


Change we will print a WARN message when it happens to print a WARN log message.

szetszwo · 2023-01-07T13:33:27Z

hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

@@ -1240,6 +1240,15 @@
  </description>
 </property>

+<property>
+  <name>dfs.namenode.access-control-enforcer-reporting-threshold-ms</name>


Please move this right below dfs.namenode.read-lock-reporting-threshold-ms since they are related.

cnauroth

+1, pending resolution of the other review feedback.

@haiyang1987 , thank you for the patch. Sorry we missed this in HDFS-16881.

haiyang1987 · 2023-01-09T07:53:42Z

Update PR.
Hi @szetszwo @cnauroth please help me review this pr again when you are available, Thanks.

szetszwo

+1 the change looks good.

hadoop-yetus · 2023-01-09T14:25:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 51s		trunk passed
+1 💚	compile	1m 31s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	1m 26s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	1m 33s		trunk passed
+1 💚	javadoc	1m 7s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	68m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	1m 27s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	1m 27s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	1m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	1m 21s		the patch passed
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javadoc	1m 28s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	23m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	293m 5s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		392m 53s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.server.balancer.TestBalancerService

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/3/artifact/out/Dockerfile
GITHUB PR	#5283
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 1318d51b6f04 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `803ae82`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/3/testReport/
Max. process+thread count	3137 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5283/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

szetszwo · 2023-01-10T01:02:44Z

There are some test failures but they are not related to this change.

szetszwo · 2023-01-10T01:03:56Z

Thanks @haiyang1987 for working on this!

Thanks also @cnauroth and @tomscut for reviewing this.

haiyang1987 · 2023-01-11T02:33:04Z

Thanks @szetszwo @cnauroth @tomscut help review and merge it.

haiyang1987 added 2 commits January 7, 2023 12:55

HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAga…

39e26e8

…instXml failed

Update Description

de0da0a

szetszwo reviewed Jan 7, 2023

View reviewed changes

cnauroth approved these changes Jan 7, 2023

View reviewed changes

Update Description

803ae82

tomscut approved these changes Jan 9, 2023

View reviewed changes

szetszwo approved these changes Jan 9, 2023

View reviewed changes

szetszwo merged commit f3cff03 into apache:trunk Jan 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed #5283

HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed #5283

haiyang1987 commented Jan 7, 2023

hadoop-yetus commented Jan 7, 2023

hadoop-yetus commented Jan 7, 2023

szetszwo left a comment

szetszwo Jan 7, 2023

szetszwo Jan 7, 2023

cnauroth left a comment

haiyang1987 commented Jan 9, 2023

szetszwo left a comment

hadoop-yetus commented Jan 9, 2023

szetszwo commented Jan 10, 2023

szetszwo commented Jan 10, 2023

haiyang1987 commented Jan 11, 2023

HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed #5283

HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed #5283

Conversation

haiyang1987 commented Jan 7, 2023

Description of PR

hadoop-yetus commented Jan 7, 2023

hadoop-yetus commented Jan 7, 2023

szetszwo left a comment

Choose a reason for hiding this comment

szetszwo Jan 7, 2023

Choose a reason for hiding this comment

szetszwo Jan 7, 2023

Choose a reason for hiding this comment

cnauroth left a comment

Choose a reason for hiding this comment

haiyang1987 commented Jan 9, 2023

szetszwo left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Jan 9, 2023

szetszwo commented Jan 10, 2023

szetszwo commented Jan 10, 2023

haiyang1987 commented Jan 11, 2023