HDFS-15520 Use visitor pattern to visit namespace tree #2203

szetszwo · 2020-08-08T01:29:25Z

https://issues.apache.org/jira/browse/HDFS-15520

hadoop-yetus · 2020-08-08T05:11:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 9s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	29m 20s	trunk passed
+1 💚	compile	1m 16s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	1m 9s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 48s	trunk passed
+1 💚	mvnsite	1m 20s	trunk passed
+1 💚	shadedclient	15m 40s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	1m 20s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	3m 3s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 0s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s	the patch passed
+1 💚	compile	1m 8s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	1m 8s	the patch passed
+1 💚	compile	1m 4s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	1m 4s	the patch passed
-0 ⚠️	checkstyle	0m 42s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 137 unchanged - 0 fixed = 143 total (was 137)
+1 💚	mvnsite	1m 8s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 37s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 20s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	1m 17s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	3m 2s	the patch passed
		_ Other Tests _
-1 ❌	unit	123m 2s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s	The patch does not generate ASF License warnings.
		204m 38s

Reason	Tests
Failed junit tests	hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestGetFileChecksum

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/1/artifact/out/Dockerfile
GITHUB PR	#2203
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7b4a3596aee9 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `64753ad`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/1/testReport/
Max. process+thread count	3789 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hemanthboyina · 2020-08-10T13:43:20Z

...hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/visitor/NamespacePrintVisitor.java

+ *          \- z_file4   (INodeFile@45848712)
+ */
+public class NamespacePrintVisitor implements NamespaceVisitor {
+  static final String NON_LAST_ITEM = "+-";


we have these constants also in InodeDirectory , can we move these constants into a common place ?

We will remove the constants in INodeDirectory with HDFS-15521.

bshashikant · 2020-08-10T15:52:07Z

@szetszwo , thanks for working on this. The patch looks good. +1
Plz take care of the checkstyle issues if possible before the merge.

szetszwo · 2020-08-11T05:12:07Z

@bshashikant , thanks for the review. Just have pushed commits for fixing the checkstype warnings, adding javadoc and some minor changes. Could you take a look again?

hadoop-yetus · 2020-08-11T06:45:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	33m 22s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	31m 40s	trunk passed
+1 💚	compile	1m 13s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	1m 8s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	0m 49s	trunk passed
+1 💚	mvnsite	1m 16s	trunk passed
+1 💚	shadedclient	15m 54s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 50s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	1m 25s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	3m 6s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 3s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 9s	the patch passed
+1 💚	compile	1m 10s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	1m 10s	the patch passed
+1 💚	compile	1m 2s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	1m 2s	the patch passed
-0 ⚠️	checkstyle	0m 41s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 137 unchanged - 0 fixed = 143 total (was 137)
+1 💚	mvnsite	1m 6s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 43s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 58s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	1m 35s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	3m 45s	the patch passed
		_ Other Tests _
-1 ❌	unit	141m 28s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s	The patch does not generate ASF License warnings.
		260m 0s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSAddressConfig
	hadoop.hdfs.TestBlocksScheduledCounter
	hadoop.hdfs.TestFileChecksumCompositeCrc
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.TestStripedFileAppend
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.TestHAAuxiliaryPort
	hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader
	hadoop.hdfs.TestErasureCodingPolicies
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestErasureCodingMultipleRacks
	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.TestDFSOutputStream
	hadoop.hdfs.TestDistributedFileSystemWithECFile
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/2/artifact/out/Dockerfile
GITHUB PR	#2203
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 781bf2887cab 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `7938ebf`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/2/testReport/
Max. process+thread count	3461 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

bshashikant · 2020-08-11T07:00:55Z

Thanks @szetszwo for the contribution. I have committed this.

hadoop-yetus · 2020-08-11T09:37:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 17s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	33m 14s	trunk passed
+1 💚	compile	1m 28s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	1m 17s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	0m 53s	trunk passed
+1 💚	mvnsite	1m 30s	trunk passed
+1 💚	shadedclient	18m 59s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 57s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	1m 26s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	3m 50s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 47s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s	the patch passed
+1 💚	compile	1m 25s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	1m 25s	the patch passed
+1 💚	compile	1m 13s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	1m 13s	the patch passed
-0 ⚠️	checkstyle	0m 50s	hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 137 unchanged - 0 fixed = 140 total (was 137)
+1 💚	mvnsite	1m 24s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 44s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 53s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	1m 18s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
-1 ❌	findbugs	3m 45s	hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
		_ Other Tests _
-1 ❌	unit	121m 32s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s	The patch does not generate ASF License warnings.
		217m 32s

Reason	Tests
FindBugs	module:hadoop-hdfs-project/hadoop-hdfs
	Found reliance on default encoding in org.apache.hadoop.hdfs.server.namenode.visitor.NamespacePrintVisitor.print(FSNamesystem, PrintStream):in org.apache.hadoop.hdfs.server.namenode.visitor.NamespacePrintVisitor.print(FSNamesystem, PrintStream): new java.io.PrintWriter(OutputStream) At NamespacePrintVisitor.java:[line 81]
	Found reliance on default encoding in org.apache.hadoop.hdfs.server.namenode.visitor.NamespacePrintVisitor.print2File(INode, File):in org.apache.hadoop.hdfs.server.namenode.visitor.NamespacePrintVisitor.print2File(INode, File): new java.io.FileWriter(File) At NamespacePrintVisitor.java:[line 59]
Failed junit tests	hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestGetFileChecksum
	hadoop.hdfs.server.datanode.TestDataNodeLifeline
	hadoop.hdfs.TestStripedFileAppend
	hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
	hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/3/artifact/out/Dockerfile
GITHUB PR	#2203
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1c511fc37b9d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `32895f4`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
checkstyle	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/3/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
findbugs	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/3/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/3/testReport/
Max. process+thread count	2880 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2203/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

sodonnel · 2020-08-11T21:53:05Z

Looks like this PR introduced two new find bugs warnings. Could you check and fix them please?

szetszwo · 2020-08-11T22:09:26Z

Thanks for pointing out. Will fix it.

Change-Id: I68c712c884032b31476470912f33f07e55729413 (cherry picked from commit b6e7a13)

szetszwo requested a review from bshashikant August 8, 2020 01:29

HDFS-15520

5874f32

szetszwo force-pushed the HDFS-15520 branch from 4f05a26 to 5874f32 Compare August 8, 2020 01:35

hemanthboyina reviewed Aug 10, 2020

View reviewed changes

bshashikant approved these changes Aug 10, 2020

View reviewed changes

szetszwo added 2 commits August 10, 2020 19:14

Revise NamespaceVisitor interface.

f99873d

Add static methods to NamespacePrintVisitor and fix checkstyle warnings.

5036fb8

bshashikant approved these changes Aug 11, 2020

View reviewed changes

bshashikant merged commit 592127b into apache:trunk Aug 11, 2020

jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023

HDFS-15520 Use visitor pattern to visit namespace tree (apache#2203)

e1c3372

Change-Id: I68c712c884032b31476470912f33f07e55729413 (cherry picked from commit b6e7a13)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15520 Use visitor pattern to visit namespace tree #2203

HDFS-15520 Use visitor pattern to visit namespace tree #2203

szetszwo commented Aug 8, 2020

hadoop-yetus commented Aug 8, 2020

hemanthboyina Aug 10, 2020

szetszwo Aug 10, 2020

bshashikant commented Aug 10, 2020

szetszwo commented Aug 11, 2020

hadoop-yetus commented Aug 11, 2020

bshashikant commented Aug 11, 2020

hadoop-yetus commented Aug 11, 2020

sodonnel commented Aug 11, 2020

szetszwo commented Aug 11, 2020

HDFS-15520 Use visitor pattern to visit namespace tree #2203

HDFS-15520 Use visitor pattern to visit namespace tree #2203

Conversation

szetszwo commented Aug 8, 2020

hadoop-yetus commented Aug 8, 2020

hemanthboyina Aug 10, 2020

Choose a reason for hiding this comment

szetszwo Aug 10, 2020

Choose a reason for hiding this comment

bshashikant commented Aug 10, 2020

szetszwo commented Aug 11, 2020

hadoop-yetus commented Aug 11, 2020

bshashikant commented Aug 11, 2020

hadoop-yetus commented Aug 11, 2020

sodonnel commented Aug 11, 2020

szetszwo commented Aug 11, 2020