HDFS-15450. Fix NN trash emptier to work in HA mode if ViewFSOveroadScheme enabled by umamaheswararao · Pull Request #2120 · apache/hadoop

umamaheswararao · 2020-07-02T23:38:50Z

https://issues.apache.org/jira/browse/HDFS-15450

…cheme enabled

hadoop-yetus · 2020-07-03T02:24:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 55s	trunk passed
+1 💚	compile	1m 22s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	1m 11s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 52s	trunk passed
+1 💚	mvnsite	1m 21s	trunk passed
+1 💚	shadedclient	16m 2s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 36s	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 47s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	2m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 50s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 8s	the patch passed
+1 💚	compile	1m 7s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	1m 7s	the patch passed
+1 💚	compile	1m 2s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	1m 2s	the patch passed
+1 💚	checkstyle	0m 40s	the patch passed
+1 💚	mvnsite	1m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 48s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 31s	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 42s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	2m 56s	the patch passed
		_ Other Tests _
-1 ❌	unit	94m 25s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 40s	The patch does not generate ASF License warnings.
		164m 25s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/1/artifact/out/Dockerfile
GITHUB PR	#2120
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ab5575fe3a47 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `04abd0e`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/1/testReport/
Max. process+thread count	4919 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

surendralilhore · 2020-07-04T14:25:13Z

Changes looks good to me.

One more option is to override below property with DistributedFileSystem class in NameNode.

<property>
  <name>fs.hdfs.impl</name>
  <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
</property>

umamaheswararao · 2020-07-04T17:20:16Z

Thank you @surendralilhore , this is a good suggestion. Initially I thought, your suggestion may have a problem in the case, when users genuinely extends DFS. But on rethinking, I thought why NN should be impacted by client changes?
NN tested well with DFS class today(trashEmptier uses it and works fine for it's functionality). So, NN should continue to work well with that. I tried to set fs.hdfs.impl now. Please take a look if that works.

surendralilhore · 2020-07-04T17:40:13Z

+1, pending Jenkins..

hadoop-yetus · 2020-07-04T20:29:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 1s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 2s	trunk passed
+1 💚	compile	1m 15s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	1m 9s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 49s	trunk passed
+1 💚	mvnsite	1m 16s	trunk passed
+1 💚	shadedclient	16m 9s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 36s	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 47s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	3m 46s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 43s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 20s	the patch passed
+1 💚	compile	1m 19s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	1m 19s	the patch passed
+1 💚	compile	1m 12s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	1m 12s	the patch passed
+1 💚	checkstyle	0m 48s	the patch passed
+1 💚	mvnsite	1m 12s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 44s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 30s	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 44s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	2m 57s	the patch passed
		_ Other Tests _
-1 ❌	unit	120m 36s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s	The patch does not generate ASF License warnings.
		192m 21s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStorageStateRecovery
	hadoop.hdfs.server.namenode.TestFSEditLogLoader
	hadoop.hdfs.server.namenode.TestSecureNameNode
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
	hadoop.hdfs.server.namenode.TestEditLogRace
	hadoop.hdfs.server.namenode.TestQuotaByStorageType
	hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
	hadoop.hdfs.TestErasureCodeBenchmarkThroughput
	hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestEditLogJournalFailures
	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/2/artifact/out/Dockerfile
GITHUB PR	#2120
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 57a940d22772 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f86f15c`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/2/testReport/
Max. process+thread count	3764 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2120/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

umamaheswararao · 2020-07-04T20:37:16Z

Test failures are unrelated. They seems to be failed with OOM.

java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)

HDFS-15450. Fix NN trash emptier to work in HA mode if ViewFSOveroadS…

5912bd2

…cheme enabled

umamaheswararao requested a review from surendralilhore July 4, 2020 05:32

Fix review comment.

93f55c4

umamaheswararao merged commit 55a2ae8 into apache:trunk Jul 4, 2020

umamaheswararao deleted the HDFS-15450 branch July 4, 2020 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15450. Fix NN trash emptier to work in HA mode if ViewFSOveroadScheme enabled#2120

HDFS-15450. Fix NN trash emptier to work in HA mode if ViewFSOveroadScheme enabled#2120
umamaheswararao merged 2 commits intoapache:trunkfrom
umamaheswararao:HDFS-15450

umamaheswararao commented Jul 2, 2020

Uh oh!

hadoop-yetus commented Jul 3, 2020

Uh oh!

surendralilhore commented Jul 4, 2020 •

edited

Loading

Uh oh!

umamaheswararao commented Jul 4, 2020

Uh oh!

surendralilhore commented Jul 4, 2020

Uh oh!

hadoop-yetus commented Jul 4, 2020

Uh oh!

umamaheswararao commented Jul 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

umamaheswararao commented Jul 2, 2020

Uh oh!

hadoop-yetus commented Jul 3, 2020

Uh oh!

surendralilhore commented Jul 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

umamaheswararao commented Jul 4, 2020

Uh oh!

surendralilhore commented Jul 4, 2020

Uh oh!

hadoop-yetus commented Jul 4, 2020

Uh oh!

umamaheswararao commented Jul 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

surendralilhore commented Jul 4, 2020 •

edited

Loading